The scandal took a political toll, and he was defeated by Democrat George D. Tillman in 1878, and again, narrowly, in 1880. He successfully contested the 1880 result and regained the seat in 1882. In 1884, he was elected to fill a seat in a different district. He was nominated for Senate but defeated by Wade Hampton in 1886. During this period in Congress he supported racial integration legislation, supported a pension for the widow of his former Major General, David Hunter, and advised South Carolina blacks to refrain from emigrating to the North and Midwest or to Liberia.