Burley was never granted a world title shot by any of the world welterweight and world middleweight champions of his era and was avoided by many of the top white contenders. Among the fighters who "ducked" Burley were Hall of Famers Billy Conn (who fought Joe Louis for the heavyweight title), Frenchman Marcel Cerdan (who was supposed to face Burley in his American debut) and even Sugar Ray Robinson, considered by many boxing historians as the best pound-for-pound fighter of all time.