Walker was born into a prominent Boston family, the son of the economist and politician Amasa Walker, and he graduated from Amherst College at the age of 20. He received a commission to join the 15th Massachusetts Infantry and quickly rose through the ranks as an assistant adjutant general. Walker fought in the Peninsula Campaign and was wounded at the Battle of Chancellorsville but subsequently participated in the Bristoe, Overland, and Richmond-Petersburg Campaigns before being captured by Confederate forces and held at the infamous Libby Prison. In July 1866, he was nominated by President Andrew Johnson and confirmed by the United States Senate for the award of the honorary grade of brevet brigadier general United States Volunteers, to rank from March 13, 1865, when he was age 24.