Bren Goillon, of Helhoughton near Fakenham, featured one of the flag's 12 gold stars as a teardrop in response to the UK's decision to leave the EU.
North Norfolk District Council has now ordered him to remove it from the conservation area property.
The council said it understood he felt "passionately" about the issue.
Read more on this story and other Norfolk news
Mr Goillon painted the 15ft by 15ft (4.57m x 4.57m) flag following the June 23 Brexit vote.
He said it was to "express his grief" at the situation.
He said he has been given 28 days to remove the painting on the grounds that it is an "advertisement".
Sue Arnold, North Norfolk District Council's cabinet member for planning, said Mr Goillon has every right to express his views, but added: "The rules governing conservation areas mean he cannot express himself permanently on the side of his house."
A man has been ordered to remove a European flag painted on his house as a symbol of his disappointment at the EU Referendum result.