Collyer lost his title in a rematch with Young Barney Aaron on June 13, 1867. The two contested in a hard-fought battle for 68 rounds and 1 hour and 55 minutes at Aquia Creek in Northern Virginia. The fight began at 8:50 am, with around 1500 in attendance and was fought outdoors in an Amphitheater on the banks of Aquia Creek in Northern Virginia. Young Barney won the choice of position and wisely took the side of the ring not facing the sun. By the end of the bout, Collyer's eyes were closed causing his seconds to throw in the sponge. The bare-knuckled boxing was desperate and brutal and both boxers were down in various rounds. Immediately after winning the title, Young Barney Aaron took a long leave from the ring, leaving the crown open for Collyer to reclaim. His bouts with Collyer were considered among his most memorable and significant.