Robert Fidler, 66, hid the building, which did not have planning permission, for four years after constructing it in Salfords, near Redhill, Surrey.
In 2015, after almost a decade of legal battles, he was told to tear it down by June or face three months in prison.
Pictures showing the farmer and builders removing tiles from the roof were featured in the Daily Mail.
In a brief telephone conversation, Mr Fidler told BBC South East that contrary to reports he had not started knocking down his home.
Reigate and Banstead Borough Council said it had not been made aware he was "demolishing his unauthorised dwelling".
In a statement, a spokesman added: "But if this is the case we are pleased that he appears to be complying with the instructions of the court.
"It is in Mr Fidler's interests to demolish by 6 June as the High Court judge ordered that he will likely a face prison term if he doesn't.
"We have already given Mr Fidler advice about the options available to him for providing alternative accommodation in existing lawful buildings on his site."
Mr Fidler told the BBC in December that he might look at reducing his home in size, possibly to a bungalow, after the winter.
The council first told the farmer to demolish his property in 2007.
In November, a High Court judge gave Mr Fidler a three-month suspended sentence, telling him he would be jailed for his "defiance" if the structure was not pulled down by June.
More than 1,800 people have signed a petition which calls on Reigate and Banstead Council to stop the "wasteful" enforcement.
A farmer who built a mock-Tudor castle behind straw bales has denied he has started to demolish it.