The pile, made up of household waste, rubble and industrial refuse, is taller than the nearby houses in Moor Street, Brierley Hill.
The Environment Agency said it will take eight months and cost Â£750,000 to clear the blot on the landscape.
Neighbours have spoken of their relief that it is being taken down.
One woman said there had been a "strong whiff" from the site over the summer which had not been pleasant at all.
The director of the firm which owned the rubble, Refuse Derived Fuel Ltd, was jailed for six months in December 2013 for repeatedly failing to clear it.
He had said the firm could not afford the clean up operation.
The agency said the bill would not fall to taxpayers and would be paid by some of the landowners.
A 42ft (13m) tall pile of rubbish that has cast an enormous shadow over the Black Country for more than three years is finally being cleared.