Stevens never married. However, there were rumors about his 20-year relationship (1848–1868) with his widowed housekeeper, Lydia Hamilton Smith (1813–1884). She was a light-skinned African-American; her husband Jacob and at least one of her sons were much darker than she was.