His first wife – who was also his third wife – was Auguste Marie (Gustl) Jansen, whom he married on 17 June 1921. After a brief marriage to actress Gerda Müller 1927–1929, he was again together with Gustl Jansen. In 1936 Scherchen married Chinese composer Xiao Shuxian in Peking. A daughter, Tona Scherchen, was born to them in 1938. She has also made a name for herself as a composer. The last of his five wives was the Zurich-based Romanian mathematics teacher Pia Andronescu, with whom he had 5 children: Myriam, David, Esther, Nathan and Alexandra.