Wadsworth was born to wealthy parents in Geneseo in Livingston County in western New York State. His father, James Wadsworth, was the owner of one of the largest portfolios of cultivated land in the state, and young Wadsworth was groomed to fulfill the responsibilities he would inherit. He attended both Harvard University and Yale University, studied law, and was admitted to the bar, but had no intention of practicing. He spent the majority of his life managing his family's estate. Wadsworth built Hartford House in Geneseo, NY upon his marriage in 1834 to the former Mary Craig Wharton of Philadelphia.