Olsen played three seasons at B 1901, before he moved abroad to play professionally with Belgian club Cercle Brugge K.S.V. in 1972, who had been promoted to the top-flight Belgian First Division championship one year earlier. At Brugge, Olsen played alongside fellow Danish international Benny Nielsen, who had recommended Olsen to Brugge manager Urbain Braems. In Olsen's first year with the club, Cercle Brugge finished in 11th place in the 1972–73 Belgian First Division and went on to establish itself in the mid-table. While at Brugge, Olsen was used as a multi-purpose player, playing every position except from goalkeeper. In 1976, Olsen moved to league rivals R.W.D. Molenbeek, who had won the 1974–75 Belgian First Division. Olsen joined Danish internationals Benny Nielsen and Kresten Bjerre at Molenbeek. His time at Molenbeek featured even better league results, and consistent finishes in the top third of the league.