Williams was born in Marion, North Carolina,[3][4] and spent his early years in the small western North Carolina towns of Marion and Spruce Pine. As a child his family relocated to nearby Asheville, where he grew up. In basketball, playing for Coach Buddy Baldwin, he was named all-county and all-conference for two years (1967 and 1968), all-western North Carolina in 1968 and served as captain in the North Carolina Blue-White All-Star Game.[5] Williams has stated that Coach Baldwin was one of the biggest influences in his life.[6]Williams went on to play on the freshman team at the University of North Carolina at Chapel Hill and study the game under coach Dean Smith.