Frankel was born in London on 31 January 1906, the son of Polish Jewish parents. He began to learn the violin at an early age, showing remarkable talent; at age 14, his piano-playing gifts attracted the attention of the American pianist Victor Benham, who persuaded his parents to let him study music full-time. He spent six months in Germany in 1922, then returned to London, where he won a scholarship from the Worshipful Company of Musicians and attempted his first serious compositions while earning his income as a jazz violinist, pianist and arranger. Known then as Ben Frankel, his jazz work can be heard on recordings by Fred Elizalde's band. He also played violin with Carroll Gibbons and the Savoy Hotel Orpheans.