The next recorded events in his life are his marriage at Wetumpka, Alabama in 1852 to Mary Jane Gray and the birth of their daughter Elizabeth a year later. Swilling wrote in 1854 that he suffered serious injuries—a broken skull and a bullet lodged in his back—in unstated circumstances. Those injuries plagued him for the rest of his life and led to a dependency on drugs and alcohol. In 1856, on his 26th birthday, something happened to cause him to leave permanently for the West. There is more than a year's break in the historical record, but Swilling apparently joined the Leach Wagon Road Company at Fort Smith, Arkansas as a teamster in the summer of 1857, probably staying with the slow-moving oxen-drawn wagon train until its arrival a year later at Mesilla, in Traditional Arizona, which was then part of the New Mexico Territory.