Four times runner-up in the Formula 1 drivers' championship, Moss effectively retired in 1962 but continued to race in historic cars and legends events.
But after qualifying for the Le Mans Legends he said it was time to stop.
"I have always said if I felt I was not up to it or I was getting in the way of competitors I would retire," he said.
Moss, who suffered paralysis to his left side and spent a month in a coma after a crash in the Glover Trophy at Goodwood in 1962, revealed that the decision to finally hang up his gloves came while racing his Porsche RS61 in practice.
Moss's 16 grands prix wins is one more than Lewis Hamilton
He had been set to partner Ian Nuthall in the Legends event - which comes before the main 24-hour Le Mans race and features classic cars from 1949-65.
"This afternoon I scared myself," he told stirlingmoss.com. "I love racing, but now it is time to stop."
During a career that spanned 63 years from his debut as an 18-year-old, Moss won 16 grands prix, including the British Grand Prix on two occasions and Monaco on three.
In 1955 he finished runner-up to Juan Manuel Fangio in the drivers' standings for the first of three times, while in 1958 he finished just a point behind champion Mike Hawthorn - Moss even helping Hawthorn avoid a potentially decisive points penalty in Portugal.
Commenting on the retirement, Martin Brundle - who raced alongside Moss in the British Touring Car Championship, tweeted: "I'm hearing that Sir Stirling Moss, my 1981 team-mate and good friend, has declared he is retiring from race driving at 81 years old. Legend."
Sir Stirling Moss has retired from racing at the age of 81 after a remarkable career that spanned eight decades and took in 212 victories.