Judah, the third of seven children, was given the same name as an older brother who died in infancy. Following a tradition adhered to by some Sephardi, he was named for his paternal grandfather, who performed the "brit milah", or circumcision ceremony. The Benjamins encountered hard times in the Danish West Indies, as normal trade was blocked by the British occupation. In 1813 the Benjamin family moved to Fayetteville, North Carolina, where they had relatives. Philip Benjamin was not financially successful there, and around 1821 moved with his family to Charleston, South Carolina. That city had the largest Jewish community in the United States and a reputation for religious tolerance. Benjamin was learned in his faith but not a successful businessman; Rebecca earned money for the family by operating a fruit stand near the harbor. Phillip Benjamin was a first cousin and business partner of Moses Elias Levy from the West Indies. Levy also immigrated to the United States, in the early 1820s.