Disappointed, Withers traveled to Missouri with the plan of settling and spending his remaining years there, but was likewise disappointed with that region and soon returned to Lewis County, where he again took up in farming and school teaching. (He was teacher to the future General Thomas “Stonewall” Jackson in 1839 in a school in Lewis County's first court house building.) Beginning in 1840, he served several years as local justice of the peace near Weston. After the death of his wife in 1853 he made his home with his eldest daughter in Parkersburg.