On February 20, 1942, Berger lost to Sugar Ray Robinson before 12,000 excited fans in a second-round TKO at Madison Square Garden. Berger had been sent to the mat twice before the referee intervened and stopped the fight, though many fans felt Berger could have continued the bout. It was Berger's only loss by knockout and one of his few career losses. Berger was first down for a count of seven from a left to the chin, and then went down for the second time from another left by Robinson. The referee stopped the fight after a count of two. It was unfortunate for his legacy that one of his most memorable and well attended bouts for American fans was a loss, even if it was to one of the greatest boxers of all time with one of the longest winning streaks in history.