Sherman was married to April 21, 1811 in Barre, Vermont. He was married to Aurora Abbott from 1833 until her death in 1883. They had 14 children. Sherman died on September 27, 1903 in Waukegan, Illinois, aged 92.