Olsen is a member of the State of Utah's Sports Hall of Fame, the Utah State University Sports Hall of Fame and USU's All-Century Football Team. In 2000, he was selected by "Sports Illustrated" as one of the State of Utah's Top 50 Athletes of the Century. He was voted to the All-Academic All-America Hall of Fame in 1988. In 1969, he was voted to the Newspaper Enterprise Association All-Time All-America team with collegiate greats such as Bronko Nagurski, Red Grange, Jim Thorpe, and O. J. Simpson, among others.