Sumner was born on January 6, 1811 in Boston, Massachusetts. He studied at Boston Latin School and at Harvard College. Sumner was married to Alice Hopper from 1866 until they divorced in 1873. They had no children. Sumner died on March 11, 1874 in Washington, D.C. from a heart attack, aged 63.