Murray, 31, had reached the last eight for the first time but he and Soares went down 3-6 7-6 (7-3) 7-6 (7-4) to Santiago Gonzalez of Mexico and Donald Young of the United States.
"We had our chance, didn't take it and it came back to haunt us," Murray said.
"We should have won, we should have found a way to get through it."
He added: "It was a big opportunity, a lot of guys had lost and it was pretty open."
In 2016, Murray and Soares won the Australian and US Open doubles titles.
Find out how to get into tennis in our special guide.
Earlier on Monday Andy Murray made it through to the singles quarter-finals.
The younger Murray came through to beat Karen Khachanov 6-3 6-4 6-4 to claim his 650th Tour-level victory in two hours and four minutes.
Jamie Murray's French Open is over after he and doubles partner Bruno Soares lost a third-set tie-break at Roland Garros.