The volunteers belong to the Independent Monitoring Board (IMB) and were attacked in separate incidents over the past 12 months.
It is thought this is the first time prison volunteers have been attacked.
The Prison Service said it would push for the "strongest possible punishment" for those involved.
The volunteers' role includes ensuring standards of care and decency for prisoners are maintained.
In two of the incidents, volunteers were spat at, one of the IMB members was pushed, and another was punched.
No-one required hospital treatment.
There are about 1,500 IMB volunteers across England and Wales who monitor the day-to-day life of prisons and immigration removal centres.
The volunteers have unrestricted access to the jail they are visiting, and can talk to prisoners out of the sight or hearing of Prison Service staff.
It is thought to be the first time that members of the IMB have been attacked.
The two spitting incidents took place in prison segregation areas - the IMB members were spat at through the cell door hatch.
The punch incident was also in a segregation area.
Segregation areas are staffed so prison officers would have been present.  The push took place on a prison wing.
No-one needed hospital treatment but the visitor who was punched has taken time off their IMB duties to recover.
A Prison Service spokesman said: "We do not tolerate any violence against staff and the dedicated volunteers who work in our prisons.
"Where incidents like this occur we will always work closely with the police to push for the strongest possible punishment."
Four people have been attacked by prisoners while doing unpaid work inspecting conditions in jails in England and Wales, the BBC has learned.