In 1955 Olson, who was finding it increasingly difficult to make weight, stepped up to light heavyweight. His first major fight in this category was against former champion Joey Maxim. Olson won the fight easily on points after scoring knockdowns in both the 2nd and 9th rounds. On June 22 Olson challenged 41-year-old Archie Moore for the light heavyweight title, a fight that many believed Olson would easily win. However, Moore was too strong for Olson and won by knockout after only a minute had elapsed of the third round. After this fight Olson began his decline.