Barry Island Pleasure Park owners added the extension to the Treasure Island and Funfair arcade without proper authority, council planners said.
Officers have recommended the planning authority orders an enforcement notice for the building to be to taken down at a meeting to be held on Thursday.
But park owner Henry Danter said: "I think we're within our rights."
If this order is ignored, the council could take legal action.
The extension is about 540 sq m (5,800 sq ft) and described as having "a run-down appearance" while the "quality of workmanship and materials is also questionable, with evidence of some inappropriate repairs".
But Mr Danter, who took over the park in 2015, said: "We're not a prison, we're an amusement park - people can't see through walls. This is our shop front.
"I don't think we need planning permission. If they think this is detrimental to Barry, they must be joking."
The report said officers had visited the site "on several occasions" over the course of more than 12 months.
"No planning application has been made to date despite recent reminders," it added.
It is recommended the enforcement notice is issued, ordering the removal of the building, frame and all associated materials and "restore the land and perimeter wall to their former condition".
Part of a Vale of Glamorgan funfair could be torn down after planners said it was built without permission.