In Lord Fitzwilliam's later years his marriage was in disarray, partly due to Olive's alcoholism, and at the time of his death he was seeking a divorce in order to marry someone else. From 1946 he had been romantically linked with the widowed Kathleen Cavendish, Marchioness of Hartington, sister of the future U.S. President John F. Kennedy. She was killed with him in an air crash on 13 May 1948, although the nature of their relationship was not made clear in the newspaper accounts at the time.