His first marriage, in 1952, was to Elizabeth Fox, the daughter of Sir Lionel Fox; they had a son, Paul. This marriage ended in 1956. In 1961 he married Sally Pearson, the daughter of the baronet Sir Neville Pearson and Dame Gladys Cooper as well as a half-sister of John Buckmaster and a sister-in-law of Robert Morley. From this marriage, which ended in 1986, Hardy had two other children, one of whom is Justine Hardy, a journalist, activist and psychotherapist who founded Healing Kashmir.