The performance earned the Ireland team their first medal of the Rio Games.
The Skibbereen pair produced a brilliant performance to finish only 0.53 seconds behind French duo Pierre Houin and Jeremie Azou.
The O'Donovans closed on the French duo in the last 500 metres as bronze winners Norway also challenged.
The Irish pair clocked six minutes, 31.23 seconds which left them only 0.16 seconds ahead of Norwegian pair Kristoffer Brun and Are Strandli.
The younger O'Donovan brother, Paul, said that he was almost "disappointed" that they hadn't been able to chase down the French boat in the closing stages.
"We put it up to the French as best we could," the 22-year-old told RTE.
"At the end, I'd say we were going all over the lane."
Media playback is not supported on this device
Paul also joked that their Irish Olympic team-mate, Belfast boxer Michael Conlan, had warned them not to return from Friday's final without the gold medal.
"I'm already dreading going home because Mick Conlan said he'd box the head off us if we didn't get the gold."
Twenty-three-old Gary, the elder by 16 months, said that they had targeted an Olympic medal after finishing 11th at last year's World Championships.
"Our goal was to win the Olympics and we knew we'd have to beat one of the best crews in the world in France.
"But to come away with a silver medal, we have to be happy."
The jovial Cork duo have been gaining a big following in Rio and indeed worldwide after giving a series of highly entertaining interviews to the Irish national broadcaster.
Cork brothers Gary and Paul O'Donovan have won Ireland's first ever Olympic rowing medal after they took silver in the lightweight double sculls in Rio.