James William Moore was born on September 14, 1910 on a farm in Troup County, Georgia, just outside the City of Hogansville. He was the son of a Georgia blacksmith, sheriff and streetcar conductor. He began working at a young age to supplement his family's income. He worked at many jobs including as a cotton picker earning 35 cents per 100 pound, managing a fruit stand, and delivering newspapers. His family moved to Detroit when he was 13. There, he found other ways of making money. Moore ran card games and pursued other games of chance. He even pitched pennies. He was very good at gambling games and was a naturally gifted athlete. He was a Triple-A level baseball player in the minor leagues. He once bowled a perfect game, and was a good golfer.