On January 22, 1945, Berger lost to talented black boxer Ike Williams in a fourth-round knockout, 2:51 into the round, at the Arena in Philadelphia, Pennsylvania before a crowd of 5,451. In the first three rounds, Berger was able to elude Williams's constant barrage of left hooks and right crosses by swiftly stepping away out of reach, but the crowd responded more favorably in the fourth when Williams began to connect with his blows at the urging of the referee for the boxers to make more contact. A left hook by Williams flush to Berger's jaw ended the bout. Berger was nine pounds heavier than his opponent, but Williams conditioning and age of 22 to Berger's 28 probably made more a difference in the bout. The telling blow was to the jaw. Williams became the World Lightweight Champion on August 4, 1947.