Calder was born in London, and grew up during World War II. His father was Lord Ritchie-Calder. Nigel Calder was a brother of the historian Angus Calder (1942–2008), mathematician Allan Calder and educationist Isla Calder (1946–2000). He was the father of travel writer Simon Calder and four other children. He was married to Elisabeth Palmer from 1954 until his death.