Upon his exchange in January 1863, Stannard returned to Vermont. He was appointed brigadier general on March 11, 1863, and given command of the 2nd Vermont Brigade in April, joining it in its camps in the defenses of Washington, D.C. The brigade consisted of the 12th, 13th, 14th, 15th, and 16th Vermont Infantry regiments. Stannard's immediate predecessor had been Brig. Gen. Edwin H. Stoughton, who had been captured in his bed at Fairfax Court House during a raid by partisan ranger Col. John S. Mosby. Stannard drilled his new brigade strenuously and his "quiet but effective" command style made him well respected by the men and significantly improved morale in the Brigade.