The Department of Home Affairs has applied for planning permission to pull down Victoria Road Prison because the building is falling into disrepair.
The prison was decommissioned in September 2008 and the prisoners were transferred to a new jail in Jurby.
The site has since been used for various projects including the filming of a horror film.
Home Affairs minister Adrian Earnshaw said: "A recent survey indicates parts of the gatehouse and cell blocks have deteriorated to a degree where the risk associated with allowing public access is something I am no longer comfortable with.
"Therefore, the department is reluctantly having to refuse all new requests for access.
"The building occupies a prime two acre site in Douglas and I now want its fate to be decided as soon as possible.
"There are surprisingly significant costs in maintaining this Victorian building without repairs to its fabric.
"Those costs will only increase and that is why I want to see the future of the former prison resolved."
A former prison in the Isle of Man could be demolished if a planning application is approved.