In 1954, he had only four fights, retaining the title in a third fight with Maxim, who once again went the 15 round distance, and versus Johnson, whom he knocked out in 14. He also beat highly ranked heavyweight Bob Baker. In 1955, Moore again beat Valdez, who by that time was the no. 1 heavyweight contender, and defended against Bobo Olson, the World Middleweight Champion and future Hall of Famer who was coming off a decision victory over Joey Maxim, by a knockout in three.