Purvis was born in 1810 in Charleston, South Carolina. His maternal grandparents were Dido Badaraka, a former slave, and Baron Judah, a Jewish American native of Charleston. His mother Harriet Judah was therefore a free woman of color. Purvis's father was a British immigrant. As an adult, Purvis told a reporter about his family: his maternal grandmother Badaraka was kidnapped at age 12 from Morocco, transported to South Carolina on a slave ship, and sold as a slave in Charleston. He described her as a full-blooded Moor, dark-skinned with tightly curled hair. She was freed at age 19 by her master's will. Harriet's father was Baron Judah, born in Charleston of German Jewish and Spanish-Portuguese Jewish descent. Baron was the third of ten children born to Hillel Judah, a German Jewish immigrant, and Charleston native Abigail Seixas, his Spanish-Portuguese Jewish wife.