Pember was born on August 18, 1823, and raised in Charleston, South Carolina. The fourth of seven children, she was raised in a wealthy and socially prominent Jewish family; her father, Jacob Clavius Levy (son of Moses Levy of Charleston), was a successful merchant, while her mother, Fanny Yates, was the seventh and youngest daughter of Samuel Yates and Martha Abrahams.
Pember's birth occurred on August 18, 1823, in Charleston, South Carolina. She was one of seven children born to a wealthy and influential Jewish family. Her father, Jacob Clavius Levy, was a successful merchant, while her mother, Fanny Yates, came from a prominent family.