Relatives believe the Bugaled Breizh trawler was dragged underwater when a submarine became snagged in its nets.
But an appeal court confirmed the case was closed, saying it was unclear why the fishing boat sank.
All five crew members died when the vessel went down off Cornwall in 2004.
The circumstances of the sinking, which happened in broad daylight in relatively calm seas off Britain's Lizard Point, have never been fully explained.
Relatives and shipping officials said Wednesday's ruling in Rennes was an "injustice".
"Eleven years of inquiry for nothing. What a waste," said Michel Douce, owner of the Bugaled Breizh, who complained of a contempt for those who disappeared and their families.
Thierry Lemetayer, whose father died in the sinking, said it was an "insult to the fishing and maritime world", according to France's 20minutes news website.
"It's as though the justice system has sent the Bugaled Breizh down for a second time and buried the men on board with it," lawyer Christian Bergot was quoted as saying.
Mr Bergot had asked the the appeal court in Rennes to re-open a previous inquiry, based on allegations that a US submarine had been in the area at the time.
But the court ruled on Wednesday there was no evidence that a submarine was involved or that it was a fishing accident.
In 2007, it emerged that traces of titanium had been found on the Bugaled Breizh's cables, a substance sometimes found on the hulls of submarines.
The following year, an inquiry in the Breton town of Quimper ruled that the submarine theory was the "most serious", as a day before the boat sank Nato military exercises had taken place in the area.
Allegations that a Royal Navy submarine, HMS Turbulent, was involved were discounted and French maritime investigators said the Bugaled Breizh's cables had probably snagged on a sand bank.
After years of legal wrangling, the case was thrown out by a court in Nantes in 2014.
Families of victims killed in a mysterious shipwreck 11 years ago have expressed outrage after a French court dismissed their last-ditch bid to find out what happened.