Burley did not defend that title, possibly as part of a strategy to win a shot at Henry Armstrong's World Welterweight title. He won the World Colored Middleweight Title in a ninth round technical knockout against Holman Williams at Victory Arena in New Orleans, Louisiana on 14 August 1942. In their rematch for the title two months later at the same venue, Williams won a 15-round decision. Jack Kincaid of the "Times-Picayune" reported that Burley had won nine rounds of the fight and had been the aggressor throughout.