Through his maternal grandmother, Mehitable (née Stoddard) Hyslop, he was part of the Stoddard family, which owned Noddle's Island in Boston Harbor. Sumner's main accomplishment was the development of Noddle's Island as East Boston, an extension of the City of Boston, beginning in 1833, in partnership with Stephen White and Francis J. Oliver. The East Boston Company was created to conduct the development, and The East Boston Timber Company was created to supply wood from upstate New York to shipbuilders whom the Company hoped to attract to the shores of East Boston.