Stanford's father was a farmer of some means. Stanford was raised on family farms in the Lisha Kill and Roessleville (after 1836) areas of Watervliet. The family home in Roessleville was called Elm Grove. The Elm Grove home was razed in the 1940s. Stanford attended the common school until 1836 and was tutored at home until 1839. He attended Clinton Liberal Institute, in Clinton, New York, and studied law at Cazenovia Seminary in Cazenovia, New York, in 1841 to 1845. In 1845, he entered the law office of Wheaton, Doolittle and Hadley in Albany.