Spieth, 22, pulled out because of fears concerning the Zika virus in Brazil.
The American is the latest high-profile player to pull out of the event in Rio, where the sport is making its return to the Games after a 112-year absence.
World number one Jason Day, number two Dustin Johnson and number four Rory McIlroy have all already pulled out.
The Zika virus has been linked to defects in newborn babies.
Two-time major winner Spieth, who is replaced by Matt Kuchar, is expected to give details about his withdrawal at a news conference before the Open Championship on Tuesday.
IGF president Peter Dawson said the number of withdrawals by top players "hasn't shed golf in the best light".
He added: "We do understand why these individual decisions have been made.
"Personally, I think there's been something of an overreaction to the Zika situation, but that's for individuals to determine, and there's certainly a great deal of concern about this issue inside the game of golf."
Meanwhile, defending Open champion Zach Johnson has questioned the inclusion of golf, football and basketball in the Olympics.
Johnson, 40, said the three sports would instead count majors, the World Cup and the NBA as their pinnacles.
"I don't know if golf has its place," Johnson said. "Basketball and soccer, do they really need to be in there either? My guess is they want a World Cup, an NBA Championship, before a gold medal.
"No offence to the Olympics but I'd rather be on the Ryder Cup team. As an American golfer I have that opportunity and that's what I'd rather do."
American Bubba Watson, the world number five, is now the highest-ranked player on the Olympic list, while Masters champion Danny Willett (nine in the world) and fellow Englishman Justin Rose (11th) are the highest-ranked Britons due to compete.
Australian Adam Scott and South African Louis Oosthuizen are among the other major winners who will not take part but USA Ryder Cup team-mates Rickie Fowler and Patrick Reed will compete.
The Olympics take place from 5 to 21 August, with the men's golf tournament from 11 to 14 August.
World number three Jordan Spieth will not take part in this summer's Olympic Games, the International Golf Federation says.