 He was the son of shipping merchant Benjamin Pell (–1828) and Marianna (née Ferris) Pell (1757-1795), who married in 1778.Among his siblings were Alfred Sands Pell (husband of Adelia Duane, a daughter of James Duane, and father of Robert Livingston Pell); Gilbert Titus Pell (husband of Elizabeth Birkbeck, a daughter of Morris Birkbeck, and father of Morris Birkbeck Pell), and representative in the Illinois Legislature, who was appointed United States envoy to Mexico in the 1850s; and Maria Pell (wife of Jacob T. Walden).His paternal grandparents were Joshua Pell (son of Thomas Pell, 3rd Lord of Pelham Manor and grandson of Sir John Pell, 2nd Lord of Pelham Manor) and Phoebe (née Palmer) Pell (daughter of John Palmer).