Judah Bergman was born in Romford Street near Cable Street, St George in the East, Stepney. His father Judah Senior, who found work as a tailor in America, and his mother Mildred immigrated from Odessa. His siblings included Wolf, changed to Willie, Rebecca and Sarah. He was apprenticed as a lather boy in a barber's shop, and began his boxing career at the "Premierland", Back Church Lane, when he was 14. Jewish Berg boxed with a Star of David on his trunks.