Harris was only offered a scholarship to Ouachita Baptist University (OBU) in Arkadelphia, after his father's roommate at Ouachita convinced head coach Buddy Benson to do it. Harris' father, O.J "Buddy" Harris, also played football for OBU. In college, Cliff became an accomplished defensive back. He played all of the secondary positions and was a quality kick returner. He was named to two All-Arkansas Intercollegiate Conference (AIC) teams and was part of the 1966 AIC Football Championship team during his freshman season. Harris was part of the track and field team and graduated with a double major in math and physics.