Williams was born in Attapulgus, Georgia, a small city in the far southwest corner of the state in Decatur County. Both of his parents were teenagers committed to a trade institute for the blind in Macon. His mother ran away from the institute upon learning of her pregnancy. At the age of 28, Williams stumbled upon his birth father, "Blind" Willie Wiggins, by accident in Florida. His mother died during childbirth when he was 10 years old. He and his older sister, Theresa, were raised by his mother's parents, Lelar and Turner Williams. Williams was run out of town by a lynch mob at the age of 13 for allegedly consorting with a white girl.