Carlebach descended from a well known German rabbinical family. He was the son of Simson Carlebach (1875–1942), a banker, and his wife Resi née Graupe.
Carlebach traced his heritage to an influential German rabbinical family. His father, Simson Carlebach (born in 1875 and deceased in 1942), worked as a banker, and he was married to Resi, who was born into the Graupe family.