Ambrose was born in Warsaw in 1896, when it was part of the Russian Empire. After a time the family moved to London. They were Jewish, his father being registered as a "Dealer in rags" in the 1911 UK census, where Ambrose was named as "Barnett' (a "Violin musician student"). He began playing the violin while young, and travelled to New York with his aunt. He began playing professionally, first for Emil Coleman at New York's Reisenweber's restaurant, then in the Palais Royal's big band. After making a success of a stint as bandleader, at the age of 20 he was asked to put together and lead his own fifteen-piece band. After a dispute with his employer, he moved his band to another venue, where they enjoyed considerable popularity.