The vandals painted the words "We broke your hearts" in thick black letters on the "lenses" of the glasses.
The stainless steel installation, which is titled Perceiving Freedom, has caused controversy in South Africa.
Critics accuse the artist of seeking to benefit from the anti-apartheid icon's legacy and say the project should not have received public money.
Some have also said the sculpture, in the Sea Point area of the city, is a marketing stunt for the eyewear company which sponsored it and whose glasses were used as a design.
The artist, Michael Elion, has said he stands by his work. Before it was vandalised, he tweeted that the Nelson Mandela Foundation had inspected the sculpture and plaque and was satisfied that it was not commercially exploitative.
The foundation said in a Facebook post that it had no comment on the sculpture as a work of art but its guidelines regarding intellectual property and commercial exploitation had been respected.
The City of Cape Town has defended the artwork, saying it was part of a wider project that gave opportunities for creative expression to diverse communities.
A group calling itself Tokolos Stencil Collective has said it defaced the "atrocious" work of art, which it also called "white supremacist corporatist art".
Other slogans painted on the artwork included "myopic art" and "remember Marikana" - a reference to events in 2012 when South African police killed 34 striking workers at a platinum mine.
Mr Mandela, South Africa's first black president, died almost a year ago at the age of 95.
He spent 27 years in prison for fighting white minority rule in South Africa.
A sculpture of a pair of spectacles, inspired by Nelson Mandela, has been defaced by vandals in Cape Town.