While recuperating from an injury caused when his horse fell, Ayres considered his military career and realized that artillery officers had a much slower rate of promotion than their colleagues in the infantry. Thus, he arranged for a transfer and became a brigade commander in the 2nd Division of the V Corps as of April 21, 1863. This division was known as the Regular Division because it consisted almost entirely of regular army (versus state volunteers) soldiers and he led its 1st Brigade in the Battle of Chancellorsville. On the first day of the battle (May 1, 1863) his brigade formed the left flank of Sykes' division when it engaged Maj. Gen. Lafayette McLaws' division on the Orange Turnpike. Sykes' division was forced to retreat after being attacked on the right flank by Maj. Gen. Robert E. Rodes' division.