Swilling was both a Confederate States Army minuteman and a civilian aid to the United States Army during the American Civil War. He worked in a variety of disciplines throughout his life, including as a teamster, prospector, mine and mill owner, and saloon and dance hall owner, as well as a canal builder, farmer, rancher, and public servant. All of this was accomplished while he suffered from periods of excruciating pain resulting from major injuries he suffered in 1854. He took morphine to assuage the pain, which led to dependency problems for the rest of his life.