Brunson started off by playing in illegal games on Exchange Street in Fort Worth with friend Dwayne Hamilton. Eventually, they began traveling around Texas, Oklahoma, and Louisiana, playing in bigger games, and meeting fellow professionals Amarillo Slim and Sailor Roberts. The illegal games Brunson played in during this time were usually run by criminals who were often members of organized crime, so rules were not always enforced. Brunson has admitted to having a gun pulled on him several times and that he was robbed and beaten.