Olsen was born in Haderslev, Denmark. He was taught to ride a speedway bike in 1966 by his friend and great rival of the 1970s, New Zealand's Ivan Mauger. He first rode in the UK for the Newcastle Diamonds in the British League from 1967 until 1969 (Mauger was also with Newcastle at the time). He then moved to the Wolverhampton Wolves, where he remained from 1970 until then end of the 1975 season, having avoided a move to Oxford Rebels. Ironically, the Rebels would beat Wolves both home and away to take the Midland Cup