Warburg's first marriage (5 July 1922), to May Nellie Holt (born 20 May 1902, Hampstead, North West London), produced three sons: David (born 22 August 1923 died 10 November 1987), Hew Francis (born 8 April 1925, died 10 April 1983) and Jeremy Fredric (14 October 1928 - 9 June 1986). The marriage ended in divorce in 1932, and on 21 January 1933 Warburg married the painter and designer Pamela Bryer (née de Bayou, widowed) (20 March 1905 - 1978). They had a son who died of a brain haemorrhage within 24 hours of his birth, on 13 March 1933.