Regent Superbowl and the indoor market in Great Yarmouth were engulfed by a seven-hour blaze on Friday.
The attraction in Regent Road was on the town's main tourist thoroughfare.
Jim Palmer, of Norfolk Fire and Rescue Service, said: "To enable the town to return to normal, we have to start bringing this building down now."
Great Yarmouth Borough Council said work had been completed on Saturday to make the site structurally safe and to allow firefighters in the ground floor to dampen down hot spots over the weekend.
It said following expert advice, material containing asbestos would now need to be cleared from the site.
"A clean-up crew wearing protective gear are starting to remove the debris and another team has started to board up windows [opposite]," it said in a statement.
A stretch of the pedestrianised road will be blocked until an inspection on Monday and shops inside the cordon cannot re-open until "further notice".
At the blaze's height, the pillar of smoke billowing from the country's oldest working bowling alley could be seen up to 11 miles away.
People and businesses across the area have rallied around to support the market's shop owners, with an online fundraising page already generating around Â£1,700 of its Â£2,000 target.
The building's owner, Philip Thompson, said he believed some of the 40 traders were uninsured and plans to re-build the complex in time for the next summer season.
"I am just so concerned about the people who trade underneath in the market," he said.
"I feel responsible that we've got to get this open as soon as possible.
"I've had to pay Â£25,000 just to make it safe overnight."
An investigation into the cause is ongoing.
Demolition teams have started knocking down a fire-ravaged bowling alley and indoor market complex in the heart of a seaside resort.