Soon afterwards, the Gila City gold deposits ran out and Swilling followed his friend Colonel Jacob Snively to Pinos Altos, where he both mined and ran a saloon and dance hall. When the Union Army withdrew from the New Mexico Territory at the beginning of the Civil War, the men of Pinos Altos formed a militia company they named the Arizona Guards for defense against Apache attack. The secession of Confederate Arizona from the Union was officially declared in 1861, a territory which included all of the New Mexico Territory south of the 34th parallel. Swilling was elected second-in-command of the company, or First Lieutenant, and retained that rank when the Arizona Guards were absorbed into the Confederate Army. Swilling likely fought at the Battle of Pinos Altos, a Confederate victory and a battle which killed his commander, Captain Thomas J. Mastin (1839-1861).