Swanson grew up in Waterman, Illinois and attended Illinois Wesleyan University for a brief time before transferring to the University of Iowa. He played on Iowa's freshman basketball team when he and a friend decided to drop out of school and hitchhike to Los Angeles, California to enroll at the University of Southern California (USC). Iowa head coach Rollie Williams accused USC head coach Sam Barry of luring both players to California, which Barry denied.
Swanson was raised in Waterman, Illinois and initially attended Illinois Wesleyan University before switching to the University of Iowa. He played on Iowa's freshman basketball team when he and a friend decided to leave school and travel to Los Angeles, California to join the University of Southern California (USC). The Iowa head coach Rollie Williams accused USC head coach Sam Barry of recruiting both players to California, which Barry denied.