Ellis was the chairman and major shareholder of Aston Villa for two separate spells, the first being from 1968 to 1975. Ellis was replaced as chairman and finally ousted from the board in 1979. During his absence Aston Villa enjoyed its greatest period of success in modern times, winning the Football League title in 1981 and the European Cup in 1982. In June 1982, Ellis ousted Harry Marshall as chairman of Wolverhampton Wanderers. At the time he noted the club was 24 hours from extinction with debt as high as £2.5m. In November 1982 he was elected to the board of the World Professional Billiards and Snooker Association.