The first real test of Olson's career came on March 20, 1950, Olson's record at this point was 40 wins and 2 losses, against the Australian Dave Sands. Olson lost to a close points decision in Sydney. Seven months after this Olson had his first fight against Sugar Ray Robinson, for the lowly regarded Pennsylvania State World Middleweight Title. Olson, who was widely seen as a slow starter, failed to get into the fight, even though Robinson was not having one of his best fights. Olson managed to hold on for 11 rounds before being knocked out. Despite his great record it was clear that Olson was still too inexperienced to be fighting at that level.