In his active career, Olsen predominantly played as a defensive midfielder and libero. He played professionally in Belgium and Germany, and won the 1983 UEFA Cup and three Belgian First Division championships with R.S.C. Anderlecht. Olsen played a total 102 matches and scored four goals for the Danish national team from 1970 to 1989, and was named 1983 and 1986 Danish "Player of the Year". He captained the Danish national team in 50 games during the 1980s, and represented Denmark at the 1984 European Championship, 1986 FIFA World Cup, and 1988 European Championship. Towards the end of his active career, Olsen was characterized as the most important player in the history of Danish football.