In 2015, Goncalo Amaral was ordered to pay 500,000 euros (Â£425,000) to Kate and Gerry McCann, of Rothley, Leicestershire, after he implicated them in their daughter's disappearance.
The decision was overturned following an appeal last year.
The McCanns took the case to Portugal's Supreme Court, but it was dismissed.
Madeleine went missing days before her fourth birthday, while on a family holiday in the Algarve in 2007, and has not been seen since.
Mr Amaral had suggested the couple faked her abduction.
While the judges' official ruling is yet to be published, lawyers for Mr and Mrs McCann have been informed of the decision.
According to the Associated Press news agency, a court official said the court had ruled the allegations were protected by freedom of expression laws.
"What we have been told by our lawyers is obviously extremely disappointing," the McCanns said in a statement.
"It is eight years since we brought the action and in that time the landscape has dramatically changed, namely there is now a joint Metropolitan Police-Policia Judiciaria investigation, which is what we've always wanted.
"The police in both countries continue to work on the basis that there is no evidence that Madeleine has come to physical harm."
Madeleine went missing from her family's holiday apartment in Praia da Luz, on May 3 2007, as her parents dined at a nearby tapas restaurant with friends.
Mr Amaral, who led the initial investigation into Madeleine's disappearance but was taken off the case in October 2007, released his book - In The Truth Of The Lie - three days after Portuguese police closed their investigation the following year.
He later took part in a documentary for Portuguese television in which he claimed Madeleine was dead, there had been no abduction and that the McCanns had hidden her body.
The McCanns have said previously the claims had exacerbated their anguish and discouraged people from coming forward with information.
Scotland Yard started a review of the case in May 2011, after then-Prime Minister David Cameron responded to a plea from the McCanns.
Madeleine McCann's parents have told of their disappointment after Portugal's highest court threw out their libel case against an ex-detective.