Cahn was born Samuel Cohen in the Lower East Side of New York City, the only son (he had four sisters) of Abraham and Elka Reiss Cohen, who were Jewish immigrants from Galicia, then ruled by Austria-Hungary. His sisters, Sadye, Pearl, Florence, and Evelyn, all studied the piano. His mother did not approve of Sammy studying it though, feeling that the piano was a woman's instrument, so he took violin lessons. After three lessons and following his bar mitzvah, he joined a small dixieland band called Pals of Harmony, which toured the Catskill Mountains in the summer and also played at private parties. This new dream of Cahn's destroyed any hopes his parents had for him to be a professional man.