The 24m (80ft) sculpture on Place Vendome in Paris was brought down when supporting cords were cut.
Earlier, US artist Paul McCarthy told a French newspaper that he was attacked by a man who said the sculpture had no place being on the street.
The sculpture is part of the week-long International Contemporary Art Fair in the French capital.
The art fair said it would restore the deflated sculpture as soon as it could.
McCarthy told French newspaper Le Monde that his work, entitled Tree, was an "abstract work" rooted in a joke about a sex toy and was also inspired by a Christmas tree.
Police said the sculpture had been attacked overnight.
"An unidentified group of people cut the cables which were holding the artwork, which caused it to collapse," police said.
Paris Mayor Anne Hidalgo said the attack was unacceptable, and also denounced the attack on McCarthy.
"Paris will not succumb to the threats of those who, by attacking an artist or a work, are attacking artistic freedom," she said.
"Art has its place in our streets and nobody will be able to chase it away."
A huge inflatable sculpture inspired by a sex toy has been sabotaged days after it was installed in Paris.