Ferguson was born to the Reverend James E. Ferguson, Sr., and Fannie Ferguson near Salado in south Bell County, Texas. At age 12 he entered Salado College (a preparatory school) but was eventually expelled for disobedience. At 16, he left home and drifted through the states of the American West, working successively in a vineyard, a mine, a barbed wire factory, and at a grain ranch. After he returned to Texas, he studied law in Bell County and was admitted to the bar.