Growing up in Ohio, Gillom played for Brown on the football team at Massillon Washington High School, where he excelled as an end, linebacker and punter. In Gillom's three seasons between 1938 and 1940, Massillon won all of its games and captured two High School Football National Championships. Gillom followed Brown to Ohio State University in 1941, playing on the school's freshman football team before dropping out because of poor grades. He then entered the U.S. Army and served for three years in World War II. Upon his discharge, Gillom enrolled at the University of Nevada, Reno to finish his college career. He led the country in scoring in 1946 but left the school after the season, again because of poor grades.