Patients queued in corridors because of a lack of cubicles in the department and there were not enough staff to care for them, a report said.
The Care Quality Commission said its layout made it difficult to keep an eye on patients, which "posed unacceptable risks to patient and staff safety".
The trust said improvements were being made.
In October 2015, the CQC investigated Great Western Hospitals NHS Foundation Trust, which runs the main hospital and four community hospitals - Chippenham, Savernake, Trowbridge and Warminster.
All community hospital services were rated "good", while children and young people's services were rated "outstanding".
But Chief Inspector of Hospitals Sir Mike Richards, said there was a "marked variation in the quality of services" between those hospitals and the Great Western Hospital.
Its "observation unit" was isolated, meaning children waiting to be seen could not be properly watched, the report said.
There were not always enough staff to look after patients left queuing in corridors and not all staff adhered to "good hand hygiene practices or using protective personal clothing".
Overall the trust's services "require improvement", the inspectors found.
Sir Mike said: "Although these issues were recognised and known, we found that the necessary improvements had not been made or sustained."
He said the trust had been under financial pressure and leadership was "open" about challenges and "must now work hard to meet the demands required".
The trust's chief executive Nerissa Vaughan said safety was "our number one priority".
"We know we need to do more to build on the improvements we have already made," she said.
She added measures had been introduced to improve the situation including new initial nurse assessments, meaning "patients are likely to be seen sooner", and more training for staff.
Safety in the A&E department at Swindon's Great Western Hospital has been rated "inadequate" by inspectors.