Horton was born in Los Angeles, the youngest of the five children of the former Ella Claudia Robinson (1892-1966) and John Loly Horton (1889–1959), and raised in Rusk in Cherokee County in east Texas. His family often traveled to California to work as migrant fruit-pickers. After he graduated from high school in Gallatin, Texas in 1944, Horton attended Lon Morris Junior College in Jacksonville, Texas on a basketball scholarship. He later attended Seattle University and briefly Baylor University in Waco, although he did not graduate from any of these institutions.