Fischer was the third of four children born to Cecil and Louella (Roussin) Fischer of Durand, Michigan. His parents were of German, French, Irish-Scot, and English backgrounds. In grade school he started his general music study with violin and piano as his first instruments. At the age of 7 he began to pick out four-part harmony on the piano. After two years of piano lessons the family moved to Grand Rapids, Michigan, where Fischer began composing classical music and making instrumental arrangements for dance bands.