Swanson grew up in Waterman, Illinois and attended Illinois Wesleyan University for a brief time before transferring to the University of Iowa. He played on Iowa's freshman basketball team when he and a friend decided to drop out of school and hitchhike to Los Angeles, California to enroll at the University of Southern California (USC). Iowa head coach Rollie Williams accused USC head coach Sam Barry of luring both players to California, which Barry denied.
Swanson originated from Waterman, Illinois and attended Illinois Wesleyan University briefly before switching to the University of Iowa. He participated in Iowa's freshman basketball team when he and a companion decided to abandon school and hitchhike to Los Angeles, California to enroll at the University of Southern California (USC). The Iowa head coach Rollie Williams accused USC head coach Sam Barry of enticing both players to California, which Barry denied.