The Independent Monitoring Board said three prisoners were found dead in their cells during the reporting year.
It blamed "inadequate" physical and mental health services for inmates and said a shortage of staff had resulted in more time spent in cells.
However, it praised staff for their "considerable efforts" in engaging with prisoners on a day-to-day basis.
A Prison Service spokesman said Chelmsford was getting extra funding to pay for more officers.
LIVE: Read more on this and other stories from Essex
The report covered the 12 months from September 2015 and said the board "fears for general prison safety".
It also acknowledged the prison had long-term issues which were "beyond the control of the governor".
It said the Victorian buildings were dilapidated and in need of major refurbishment, for which money was not available.
In a statement, a Prison Service spokesman said: "We are committed to transforming prisons into places of safety and reform and we've announced a major overhaul of the prison system including 2,500 extra frontline officers.
"We take the mental health of those in our custody extremely seriously, but recognise that more can be done to support prisoners.
"That is why we've invested in specialist mental health training for prison officers, increased funding for prison safety, and launched a suicide and self-harm reduction project to address the increase in self-inflicted deaths and self-harm in our prisons.
"We have also invested Â£14 million to provide more than 400 extra staff in 10 of the most challenging prisons, with HMP Chelmsford using the additional funding to boost the number of frontline officers."
Bullying, violence and self-harm among inmates at Chelmsford Prison has "increased sharply", a report has said.