Norman was born in Lincolnton, Georgia on January 4, 1939 to Fessor and Elease "Eloise" Norman (née Booker) as the youngest of ten children. He attended West Charlotte High School in Charlotte, North Carolina where he made the team in his junior season and became a starter in his final year. He enlisted in the Air Force but was granted a release before attending boot camp due to a football scholarship offer from Johnson C. Smith University by then coach Eddie McGirt, without ever having seen him play