In football, as a senior, he was a consensus All-American selection (making the vast majority of All-America teams) and was the winner of the Outland Trophy. After Olsen's junior year of 1960 he was also named All-American by the Football Writers Association of America and Newspaper Enterprise Association. He was also All-Conference in both 1960 and 1961. Olsen and Utah State were in the 1960 Sun Bowl, losing to New Mexico State, 20–13. Led by Olsen, the Aggie defense held the New Mexico State Aggies to just 44 rushing yards on 32 carries.