He was married in 1940 to the Roman Catholic, Clare Elwes (1905–1998), youngest child of the tenor Gervase Elwes. Their only child was Margaret, known as Maggie Keswick, gardener and author (1941–1995), who founded Maggie's Centres for those suffering from cancer. She had two children with her husband, the landscape architect Charles Jencks, whom she had married in 1978, as his 2nd wife. (Charles Jencks remarried in 2006 Louisa Lane-Fox, former wife of historian Robin Lane Fox and mother of Martha Lane Fox, Baroness Lane-Fox of Soho).