A native of the Minneapolis–Saint Paul area, Swardson was born to Pamela and Roger Eric Swardson, and is the youngest of three siblings—he has a sister, Rachel, and a brother, John. Roger Swardson (1934–2003) was an editor and journalist—having written for publications such as the "Cincinnati Enquirer" and "City Pages", as well as founding the "Grand Gazette", a former Saint Paul community newspaper. Roger Swardson also invested in land development in Saint Paul, revitalizing Grand Avenue—an area now known as Victoria Crossing. Roger and Pamela divorced in 1989. Swardson has Swedish ancestry.