Olsen was only a reserve rider for the 1980 World Final at the Ullevi Stadium (won by Michael Lee) and did not get to ride on the night. He won his place as a reserve by beating American Scott Autrey in a run-off for 11th place at the InterContinental Final at White City in London after both riders finished on 6 points. He qualified for what many thought would be his last World Final in 1981 at Wembley, and ended up finishing second behind American Bruce Penhall (the first American World Champion since Jack Milne in 1937). Olsen had the distinction of winning the last Speedway World Final race held at the original Wembley Stadium. In a run-off for second place he beat fellow Dane and Coventry team mate Tommy Knudsen.