The centre in east Belfast has been closed since February.
It had been under the ownership of Castlereagh Council. It transferred to Belfast City Council when the super councils were created in April.
The recommendation will only come into play if Belfast loses a legal challenge over the transfer of ownership.
A restricted document before the committee said the cost of refurbishment would be in the region of Â£5m.
An investigation found asbestos in the squash courts, pool areas, changing rooms, weights room and the viewing gallery, while legionella was found in the pool area and the associated changing rooms.
The document said asbestos would have to be removed and that would cost around Â£400,000.
The recommendation for the demolition of the centre was agreed at the policy and resources committee and it will now go before full council in June.
The Robinson leisure centre had been part of Castlereagh Borough Council, which under the council changes at the start of April, merged with Lisburn to form Lisburn and Castlereagh Council.
Boundary changes meant the centre is now located in the Belfast City Council area.
Belfast City Council's policy and resources committee has recommended the demolition of the Robinson Centre after asbestos and legionella were found.