Stevens was born in Danville, Vermont on April 4, 1792. He went to Dartmouth College and the University of Vermont. Stevens was never married and had no children. He was in a relationship with his black housekeeper even when slavery was legal. He wore a wig because he was bald.