Mann, born and raised in Virginia, is the daughter of Robert Munger and Elizabeth Munger. In Mann's introduction for her book "Immediate Family", she "expresses stronger memories for the black woman, Virginia Carter, who oversaw her upbringing than for her own mother". Elizabeth Munger was not a big part of Mann's life, and Elizabeth said "Sally may look like me, but inside she's her father's child." Virginia (Gee-Gee) Carter, born in 1894, raised Mann and her two brothers and was an admirable woman. "Left with six children and a public education system for which she paid taxes but which forbade classes for black children beyond the seventh grade, Gee-Gee managed somehow to send each of them to out-of-state boarding schools and, ultimately, to college." Virginia Carter died in 1994.