Withers was said to have been the largest slave-owner in Lewis County, owning ten or twelve slaves. Notwithstanding this, he was a Unionist during the Civil War and served as delegate to the First Wheeling Convention of May 1861. Withers fathered a number of children by his mulatto slave Lucy Taylor after his wife died. He subsequently sold her and at least two of the children. In an 1864 issue of "Harper’s Weekly", Col. George H. Hanks published a letter and printed an engraving of a photograph that included Withers’ slave son, Charley Taylor, in an effort to publicize the issue of "white slaves".