In addition to pursuing a political career, Washburn was active in the Vermont Militia; he was commander of a regiment with the rank of colonel in the late 1830s and early 1840s, and organized a company in anticipation of the American Civil War, which he commanded as a captain. Washburn's company was federalized in 1861 as part of the 1st Vermont Infantry; he was commissioned as the regiment's lieutenant colonel and second-in-command. He became the "de facto" commander after John W. Phelps was promoted to command of a brigade, and he led the regiment during its participation in the Battle of Big Bethel. Washburn was elected adjutant general of the state militia in late 1861, and led it for the rest of the war, including the state's response to the St. Albans Raid.