Lee was born in Belgravia, London, England, on 27 May 1922. He and his older sister Xandra were raised by Estelle Marie and Geoffrey Trollope until their divorce in 1926. Trollope was a professional soldier. Later, while Lee was still a child, his mother married (and later divorced) Harcourt George "Ingle" St. Croix, a banker. Lee studied at Wellington College from age 14 to 17. After this, he worked as an office clerk in a couple of London shipping companies. In 1941, he enlisted in the RAF during World War II. 