Born in Holycross, County Tipperary on 12 February 1930, Doyle was the first child born to Tim and Margaret Doyle. Six weeks later his mother died from purpureal fever, leaving the young Doyle to be raised by his forty-two-year-old father. After a year Doyle was sent to Dungarvan, County Waterford where he was raised by his aunt. On returning home to Holycross, Doyle was enrolled in the Sacred Heart College for Little Boys in Thurles, a boarding school for young boys who had lost their mothers. After a year spent here, he began his formal education at Gaile National School. As an eleven-year-old Doyle was sent to Thurles CBS where he finished his primary education and received a limited secondary education.