Historic England said the scheme, which includes two skyscrapers, a five-star hotel, flats and restaurants, threatens the city's "precious heritage".
The Sir Ralph Abercromby pub and Bootle Street's former police station would go as part of the Jackson's Row project.
The Manchester United legends said the development would transform the area.
No-one from the developers was available for comment.
Neville and Giggs recently submitted the plans for the St Michael's development, near Manchester Town Hall, to Manchester City Council.
The scheme includes a 200-bed five-star hotel, 153 apartments, 135,000 sq ft of Grade A offices and a synagogue.
Retail and leisure space, including two sky bars/restaurants, is planned for the 31-storey Number One St Michael's skyscraper and a 21-storey office tower would be housed in Number Two St Michael's.
Catherine Dewar, Historic England's planning director in the North West, said she was "deeply concerned" about the project.
"It would have an impact on people's appreciation and experience of the stunning town hall and library but it would also erase different layers of this area's history, irreparably damaging the special character of the surrounding conservation area," she said.
"A dynamic city like ours needs to fully embrace development but this scheme is not good enough to justify the damage it would cause to the streets around the site and to the setting of the city's most important buildings and spaces."
She added it "threatened Manchester" with the loss of buildings "that have soul and tell important stories about our city's past".
Last week Neville said the project would be "one of the biggest statements in architecture in modern times in Manchester".
The former Manchester United stars said the scheme would create 1,000 jobs.
Meanwhile an online petition to save the Abercromby pub - said to be the inspiration for the pub in BBC's Life On Mars - has attracted more than 4,500 supporters.
Neville and Giggs were involved in the development of Hotel Football, near their former club's Old Trafford stadium.
They are also turning the former Manchester Stock Exchange into a luxury hotel.
Before renovation work started earlier this year, they allowed homeless people to stay in the building.
Ex-footballers Gary Neville and Ryan Giggs's Â£200m plans to redevelop part of central Manchester would "erase" the area's history, campaigners warned.