After graduating from high school, Tolliver played college football at Texas Tech University and was a member of the Delta Tau Delta fraternity. His hard throwing style led Texas Tech head coach David McWilliams to say, "He throws the ball harder, and with more velocity, than anyone I've ever seen." He redshirted his freshman season in 1984 and became the starting quarterback in 1985. On November 9, 1985, Tolliver had his first breakout performance for the Red Raiders as he threw for a record-setting 422 yards and five touchdowns in a 63–7 win over Texas Christian University (TCU), a feat that helped him become a household name in Texas. As he began his sophomore season, he was considered the lone bright spot on Texas Tech's offense. Tolliver struggled at the start of the year; by the end of September he had only completed 54 of 123 passes. After throwing five interceptions in a game against Baylor, he shrugged off the bad game, saying, "even Betty Crocker burns a cake every now and then." His fortunes continued to sink when in a game in late October against the Rice Owls, he was benched, and backup quarterback Monte McGuire rallied the team to a victory. However, Tolliver regained his starting job and brought his team to the 1986 Independence Bowl as Texas Tech won six games for the first time since 1978. He finished the season with 1,802 passing yards and seven touchdowns.