Ellory's father is unknown to him, having left the household before Ellory was born. He was raised by his mother and maternal grandmother, his maternal grandfather having drowned in 1957 in Wales. Ellory's mother died as a result of a pneumonic haemorrhage in late 1971, the victim of a pneumonia epidemic that killed a number of people in the West Midlands. Ellory was then sent to a number of different schools, and finally completed his education at Kingham Hill School in Oxfordshire, a school established by the Barings-Young banking family as a facility for "wayward and orphaned children".