After growing up in Athens, Georgia, but playing high school football in Arlington, Virginia at Washington-Lee High School, Scott played college football at the University of Georgia, where he led Georgia in interceptions in 1967 with six interceptions and 1968 with ten interceptions. In 1967, Scott was named first-team All-SEC defensive back by the Associated Press, and in 1968 again in both the AP and the UPI. The ten interceptions in a season is now second-most in Georgia history behind Terry Hoage’s 12 in 1982. Scott's 175 return yards on interceptions in 1968 is also second all-time for Georgia.