The Washburn family had always been strongly opposed to slavery. Washburn moved to La Crosse, Wisconsin in 1861 but returned to Washington, D.C., later that year as a delegate in the peace convention that was held in an attempt to prevent the American Civil War. He served in the Union Army during the Civil War, becoming colonel of the 2nd Wisconsin Volunteer Cavalry, on February 6, 1862; brigadier general of Volunteers on July 16, 1862; and major general on November 29, 1862. Washburn had the honor of having his appointment document signed by President Abraham Lincoln. At one point Ulysses S. Grant called Washburn "one of the best administrative officers we have." He commanded the cavalry of the XIII Corps in the opening stages of Ulysses S. Grant's Vicksburg Campaign. Once siege operations had begun against the city of Vicksburg and Grant called for all available forces, Washburn led a detachment of the XVI Corps during the siege of Vicksburg. He commanded the 1st Division in the XIII Corps in Nathanial P. Banks' operations along the Texas Coast leading the expedition against Fort Esperanza in November 1863.