The Independent Monitoring Board (IMB) said about 170 men are on open-ended sentences at the jail near Devizes.
A lack of funding has led to the withdrawal of mandatory courses those prisoners are required to undertake in order to qualify for a parole hearing.
The IMB said the matter is causing "a destabilising influence in the prison".
Its report claimed the cost of keeping each prisoner who is over tariff at the Category C jail amounts to more than Â£20,500 per annum.
Inspectors advised the government to implement a "finite action plan" to address the needs of prisoners affected which must include funding of courses to "reduce the wait times, and increase the number of parole boards".
The report also recommended a national database to identify prisons which deliver interventions and match prisoners to appropriate courses.
However, the Ministry of Justice said those who remain in custody do so because the independent parole board "has assessed them as posing an unacceptable risk to the public".
The Criminal Justice Act 2003 introduced indeterminate sentencing for public protection for serious sexual and violent offenders.
The Prison Reform Trust said: "A lack of resources means that prisoners are unable to prove their reduced risk and are waiting longer and longer for a parole decision, with no means of knowing when, or if, they will be released.
"To right this wrong would not only save over Â£40m a year but should also be done on grounds of simple humanity."
Erlestoke Prison did not wish to comment on the report.
HMP Erlestoke houses about 500 men and specialises in rehabilitating long-term prisoners and preparing them for release.
Action is needed to reduce the amount of prisoners detained at Wiltshire's Erlestoke Prison beyond their minimum tariff, inspectors have said.