Murray and Brazilian partner Bruno Soares, who were seeded fourth, lost 7-6 (7-5) 7-6 (7-4) against 16th seeds Marcin Matkowski and Leander Paes.
The pair won the Australian Open earlier this year - their first Slam playing together.
It means Murray, 30, must wait for another chance to reclaim the world number one ranking he held in March.
Martina Hingis and Sania Mirza's hopes of holding all four women's doubles Grand Slam titles were ended with a 6-3 6-2 defeat by Czech duo Barbora Krejcikova and Katerina Siniakova in the third round.
The top seeds were hoping to win four majors in a row after following up last year's Wimbledon and US Open triumphs with victory in January's Australian Open.
Venus and Serena Williams, the last women's team to hold all four majors in 2010, also went out in the third round, losing 6-3 6-3 to the Dutch-Swedish pairing of Kiki Bertens and Johanna Larsson.
Subscribe to the BBC Sport newsletter to get our pick of news, features and video sent to your inbox.
Jamie Murray's eight-match winning run in Grand Slam men's doubles has ended in the French Open third round.