Jordan Spieth is the only player from the top four who has not said whether he is going to Brazil. A decision could come as early as this weekend ahead of The Open Championship. Rickie Fowler also is undecided.
Johnson, No. 2 in the world after his U.S. Open title and another World Golf Championship victory last week, is the 13th golfer to pull out of the Rio Games. Eight of them, including world No. 1 Jason Day and No. 4 Rory McIlroy, specifically have cited Zika as the reason.
Patrick Reed, who said earlier this week at the Scottish Open that he wants to play in the Olympics, would replace Johnson. The U.S. is allowed four players, the most of any country in golf, because they are in the top 15 in the world.
"This was not an easy decision for me," Johnson said in a statement released by his management company. "But my concerns about the Zika virus cannot be ignored."
The Zika virus can cause severe birth defects, including microcephaly, in which babies are born with abnormally small heads.
Johnson said he and his fiancée, Paulina Gretzky, plan to have more children. They have an 18-month-old son, Tatum.
"I feel it would be irresponsible to put myself, her or our family at risk," Johnson said. "I believe I am making the right decision for me and, most importantly, my family. While I am sure some will be critical of my decision, my hope is that most will understand and support it."
The decision follows a meeting at the Bridgestone Invitational with USA Golf in which the leading four Americans were briefed on Zika and security concerns. Johnson had called it a productive meeting and said he was planning to play, although he still wanted a few more answers.
Of the top four Americans, only two-time Masters champion Bubba Watson said he is going.
Golf was last an Olympic sport in 1904 in St. Louis.
Nine of the 13 players who have withdrawn are among the top 30 in the world: Day, Johnson, McIlroy, Adam Scott, Branden Grace, Louis Oosthuizen, Hideki Matsuyama, Charl Schwartzel and Shane Lowry.
The Associated Press contributed to this report.