Pember was born on August 18, 1823, and raised in Charleston, South Carolina. The fourth of seven children, she was raised in a wealthy and socially prominent Jewish family; her father, Jacob Clavius Levy (son of Moses Levy of Charleston), was a successful merchant, while her mother, Fanny Yates, was the seventh and youngest daughter of Samuel Yates and Martha Abrahams.
Pember was born on August 18, 1823, and raised in Charleston, South Carolina. The fourth of seven children, she was raised in a wealthy and socially prominent Jewish family; her father, Jacob Clavius Levy (son of Moses Levy of Charleston), was a successful merchant, while her mother, Fanny Yates, was the seventh and youngest daughter of Samuel Yates and Martha Abrahams.