The sites in question are part of the Cornwall-based Morleigh Group, the Care Quality Commission (CQC) said.
Clinton House in St Austell was closed by the owner earlier this month due to safeguarding concerns.
The Morleigh Group said staff had been removed and a "thorough review" had been carried out.
The council apologised to residents and said what had been uncovered was "shocking and utterly unacceptable".
The other homes involved are St Theresa's in Callington, Elmsleigh in Par and Collamere in Lostwithiel.
The CQC said all four would be downgraded in inspection reports due to be made public next week following a "significant deterioration in the standards of care".
The concerns relate to bad management, staffing levels, the poor management of medication and people not being treated with dignity and respect. The allegations are not thought to involve physical abuse.
The CQC has investigated Morleigh Group care homes 22 times in the last two years and said owners had "squandered" the opportunity to put things right.
Cornwall Council said further action would be taken once it had official confirmation from the CQC.
Trevor Doughty, from the council, said: "We would never make placements in care homes rated 'inadequate'."
He said concerns raised about Clinton House were mostly upheld. All 32 residents will be moved by November 25.
Council investigations into the other three homes should be completed within days.
Mr Doughty added: "I am sorry that the standard of care provided by Morleigh Group has fallen far short of what residents, their relatives and the general public have a right to expect."
Christine Stewart, whose 85-year-old mother Sylvia lived at Clinton House, first raised concerns about the home in 2013.
She said: "It really is a scandal that things have been allowed to get to this stage."
Devon and Cornwall Police confirmed it was involved in a multi-agency investigation and that no arrests had been made.
Four private care homes will be downgraded to "inadequate" amid serious concerns for residents' safety, the BBC can reveal.