He lost the World Lightweight Title for the last time against Ike Williams in a six-round TKO on August 4, 1947 at Municipal Stadium in Philadelphia, Pennsylvania. Montgomery was first knocked for a nine count in the sixth, and eventually the referee stopped the bout after Montgomery seemed helpless against the attacks of Williams. The blow that sent Montgomery to the mat for a count of nine in the sixth was a right to the chin by the hard punching Williams. Williams was four years younger, which may have played a role in the advantage he had against the ring weary Montgomery.