Billington was married to the former Marjorie Anne Brennan. They had four children: Dr. Susan Billington Harper, Anne Billington Fischer, the Rev. James Hadley Billington Jr., and Thomas Keator Billington, as well as 12 grandchildren. Dr. Billington and his daughter Susan are the first father and daughter to both be awarded Rhodes Scholarships and use them to earn DPhils (at Oxford University).