Central Bedfordshire Council said it would serve an enforcement notice on Syed Raza Shah, after a planning inspector rejected his appeal.
Mr Shah was given permission to increase his house in Barton-Le-Clay, Bedfordshire, by 45%, but the council said he had increased it by 165%.
He will now be told to reduce the house to the size originally permitted.
Planning inspector Bridget Campbell said the seven-bedroom building "harms the landscape character of the area".
Councillor Nigel Young, executive member for regeneration on the council, said he welcomed the latest decision in the long-running dispute.
He said it "reflects the concerns of the council and local residents" and "defends our green belt from inappropriate and unlawful development".
The extended house was described as "Hollywood come to Barton" and hard to value by estate agent Ken Ravey from Country Properties in Barton-le-Clay.
"I sold the five bedroom house set in four acres next door about five years ago for Â£1.25m - this house has seven bedrooms and is set in 1.5 acres," he said
"If it were down the road in Harpenden, yes it could be worth Â£5m - but this is Barton-Le Clay."
In May 2013, an inspector granted retrospective planning permission for the home, named Random House, but the council challenged the decision.
The council argued that Mr Shah's alterations amounted to a "new dwelling" and was "inappropriate development" in an area of outstanding natural beauty.
Mr Shah has not yet commented on the planning inspector's decision.
A man who turned a bungalow into a three-storey mansion has been told to reduce it in size, a council has said.