Hardy was born in Cheltenham, Gloucestershire on 29 October 1925. He was married to Elizabeth Fox from 1952 until they divorced in 1956. He was later married to Sally Pearson from 1961 until they divorced in 1986.