Ellis was born in Ashton-under-Lyne, Lancashire. In his younger days, he played as a centre-half for Sheffield Wednesday in the 1960s. As a teenager he made his debut in the 1966 FA Cup final against Everton, in which his team lost 3–2.[https://web.archive.org/web/20061017014917/http://footballfocus.xsmnet.com/facup/lineups/teams1966.htm]. He then moved on to play for Mansfield Town, Lincoln City, and Watford. It was at Vicarage Road that he became coach and assistant manager to future England boss Graham Taylor. With Ellis' help, Watford climbed the divisions before turning down a new contract in favour of trying his hand at management. When the Blackpool job was advertised, Ellis applied and was given the job on 1 June 1982.