She was born Carolyn Robertson in Norfolk, Virginia in 1925. Her father, Leroy Solomon Robertson, was a chef and her mother, Bertha Flanagan Robertson, a seamstress and homemaker. Payton came from a close knit family that emphasized the value of education. Her grandfather, although born into slavery, saw to it that all of his children attended college. Payton's father enrolled her at Bennett College in Greensboro, North Carolina, in 1941 majoring in home economics and graduated in 1945. Payton said that Bennett, a small historically Black women’s college, shaped her aspirations, attitudes, and expectations and gave her a sense of her capabilities as a woman. She remained close to Bennett College during her entire life and established a scholarship fund there in the late 1990s.