At the beginning of the Second World War he married Joan (sometimes known as Zelda) Sainsbury and had one son, Gilbert Richard. After the failure of that marriage, he married Irene (Pip) Ansell but the marriage barely lasted six months. While working in summer season in 1950, at the Winter Gardens in Ventnor on the Isle of Wight, he met Iris Margaret Tully who was also in the show. At the end of the season, they returned to London and set up home together in Iris' flat in Shaftesbury Avenue. Iris changed her name to Emery by deed poll until 1955, a year after she had given birth to his second son, Nicholas William. She and Emery married in 1955.