Following two walkover wins, Olson put his middleweight title on the line against Robinson, who was once again number 1 contender following his brief retirement, on December 9, 1955. It was Olson's fourth and final defense of his title, but ended as a stunning comeback for Robinson. Olson, who entered the fight as a 3–1 favorite, was knocked out in the second round. The rematch, fought five months later at Wrigley Field, on May 18, 1956, ended similarly with Olson going down in the fourth. Robinson ended the bout with a hard left hook to the body and a right to the jaw. Olson had made the mistake of dropping his right hand after Robinson's hard left, dropping his only defense in an instant. After this second defeat Olson announced his retirement.