Olsen served as a television color commentator, teaming mostly with Dick Enberg on NBC's coverage of the AFC during the late 1970s and almost all of the 1980s. He and Enberg also teamed for four Super Bowls (XV, XVII, XX and XXIII), as well as nine Rose Bowls from 1980 to 1988. Olsen also worked Super Bowl XIII in 1979 with Curt Gowdy and John Brodie (Enberg was then serving as pre-game/halftime/post-game host). In 1989, Olsen was replaced by Bill Walsh as NBC's lead NFL color commentator. For the 1989 season, Olsen worked with Charlie Jones on NBC's broadcasts. In 1990 and 1991, he moved to CBS Sports doing NFL games with Dick Stockton.