 From his grandmother's first marriage to Richardson Sands (younger brother of Joshua and Comfort Sands), she was the grandmother of fellow banker Samuel Stevens Sands.Among his many prominent relatives was uncle Alexander Hodgdon Stevens, a surgeon, and his first cousins, Lucretia Stevens (née Rhinelander) Jones, the mother of author Edith Wharton.He was educated at Huddard's School in New York City, entering Yale College in 1850 where he graduated four years later in 1854.