Byrne first offered his services to P.G.T. Beauregard in South Carolina, but upon learning that Fort Sumter had already fallen, he decided to attach his battery to the organization now forming at Camp Boone that would later become famous as the Orphan Brigade. The battery saw its first minor action on the advance into Kentucky led by General Simon B. Buckner. Byrne's Battery, along with the 2nd Kentucky Infantry, led the advance into Kentucky late in 1861 and captured guns and men along the way, along with many of the pro-Union Homeguard of Kentucky and some of regular and volunteer troops. This advance led to the capture of Bowling Green and the subsequent set-up of the later exiled Confederate Government of Kentucky.