At age 7, Fischer and his two older brothers, Raoul and George, were sent to the Swiss boarding school La Châtaigneraie, near the home town of his mother, Renée Tapernoux, in Vevey. At high school he made a pact with a childhood friend, one of them would become a doctor and the other a scientist and then they could cure the ills of the world. While at high school Fischer was admitted to the Geneva Conservatory of Music, he also considered becoming a professional musician.