Although the Smith family was of relatively little means, her grandmother, aunts, and mother were all well-educated, especially for the level of education accessible to Black women in the 1940s and 1950s. Her grandmother and great-aunts taught in segregated schools in the South before moving north, though her mother was the only one in her family to have received a college diploma, a Bachelor's of Science in education from Fort Valley State University (then Fort Valley State College). Barbara's family were all active readers who emphasized education, inside and outside of school.