At the age of 20, the ambitious Phillips emigrated to the British colonies in North America, sailing from London in November 1756. Working as an indentured servant to pay off his passage, he lived in Charleston, S.C.. He was employed by Moses Lindo, an indigo planter and Sephardic Jew who was part of the growing Jewish community in the city. By the beginning of the 19th century, Charleston was home to the largest and wealthiest Jewish community in North America, a status it would hold until about 1830.