Olsen's political career was marked by a bitter rivalry with Dean Brown, the two representing the conservative and moderate wings of the South Australian Liberal Party respectively. After the 1982 election and the electoral defeat and retirement of David Tonkin, Olsen defeated Brown for the state Liberal Party leadership and became Leader of the Opposition. Up against the Labor premier John Bannon, Olsen lost both the 1985 election and 1989 election. In the latter election, the Liberals won a majority of the two-party vote (52 percent) with a five-seat swing. However, most of that majority was wasted on landslides in the Liberals' rural heartland. Even with the likely support of the one National Party MP, the Liberals were still one seat short of making Olsen Premier.