Sumner was married three times. His first marriage was to Mary Ann (née DeWolf) Perry (1795–1834) in 1826. She was the widow of Lt. Raymond H. J. Perry of the United States Navy (brother of well-known Navy officers Oliver Hazard Perry and Matthew Calbraith Perry, all sons of Capt. Christopher Raymond Perry) and a daughter of U.S. Senator from Rhode Island James DeWolf. When he was in Boston, Sumner resided at the family house on Mount Vernon Street in Beacon Hill.