Stephen was five when he attended an "infant school". In 1833 he went to the Alleghany Academy, a school for the social elite founded by the Presbyterian clergyman Rev. Joseph Stockton. In 1834, Stephen went to a black church with Olivia Pise, a mulatto servant. In 1836, the family left The White Cottage for ever and moved to Allegheny City when Stephen's father was appointed Collector of the Pennsylvania Canal.