William Purvis and Harriet Judah lived together as husband and wife, but racial law prevented their marriage. The couple had three sons: William born in 1806, Robert born in 1810, and Joseph born in 1812. In 1819 the family moved north to Philadelphia, Pennsylvania, where the boys attended the Pennsylvania Abolitionist Society's Clarkson School. William intended to consolidate his business affairs and return with his family to Britain, where he thought his sons would have better opportunities. He died in 1826 before they could move.