Ustinov was born on 16 April 1921 in Swiss Cottage, London Borough of Camden. He studied at the Westminster School. He was married to Isolde Denham from 1940 until they divorced in 1950. He later married Suzanne Cloutier from 1954 until they divorced in 1971. He was then married to Helene de Lau d'Allemans from 1972 until his death in 2004. He lived in Bursins, Vaud, Switzerland and had four children. 