Washburn was born in Lynn, Massachusetts, on September 7, 1814, the son of Judge Reuben and Hannah Blaney (Thacher) Washburn of Cavendish, Vermont. His family moved to Vermont in 1817 and settled in Ludlow, Vermont in 1825. Washburn attended the public schools of Ludlow and Black River Academy, then attended Dartmouth College. He graduated in 1835, and was a member of Phi Beta Kappa. He was principal of the Haverhill Academy while studying law under his father in Ludlow, and with William Upham in Montpelier. After completing his legal education with a year at Harvard Law School, in 1838 Washburn was admitted to the bar, and he began practicing law in Ludlow in January 1839.