Cahn's father, Martin Cahn, came from a religious, but assimilated, Jewish family which had included successive heads of Jewish communities in small settlements in Baden. He worked as an accountant for the mirror factory of Robert's maternal grandfather, Hugo Heinemann. Young Robert was raised in a flat in the centre of Fürth, and for three years in a modernist style house on the outskirts. In July 1933, his father's marriage to Else being on the point of collapse and the children having been persecuted on account of their being Jewish, the family fled to Switzerland from where Robert went with his mother and grandfather to Majorca and Martin to London to establish a new business. Cahn joined his father in London in 1936 and was introduced to the musical soirées and cultural circle of friends his father had developed.