He is the son of Wilfrid George Gerald "W. G. G." Duncan Smith, a decorated Royal Air Force flying ace of the Second World War, and Pamela Summers, a ballerina. His parents married in 1946. One of his maternal great-grandmothers was Ellen Oshey, a Japanese woman living in Beijing who married Pamela's maternal grandfather, Irish merchant seaman Captain Samuel Lewis Shaw. Through Ellen and Samuel, Duncan Smith is related to Canadian CBC wartime broadcaster Peter Stursberg (whose book "No Foreign Bones in China" records their story) and his son, former CBC vice-president Richard Stursberg.