Whitlam married Margaret Dovey a prominent Australian swimmer and social worker in 1942 and they remained married till her death on 17 March 2012. On 21 October 2014, Whitlam died in Elizabeth Bay, Sydney, New South Wales at the age of 98.