He won the Speedway World Championship three times, in 1971, 1975, and 1978. He also won the World Long Track Championship in 1973. In 1979 Olsen won Speedway World Pairs Championship with Hans Nielsen. In 1978, 1981 and 1983 Olsen was the captain of Denmark's winning Speedway World Team Cup teams, while also finishing second in 1979 and 1982. Denmark could only place third in the 1980 World Team Cup Intercontinental Final and missed a place in the Final held at the Olympic Stadium in Wrocław, Poland, the same track where he made his World Individual Final debut in 1970. In 2015, Olsen was named an FIM Legend for his motorcycling achievements.