Walker is an African American who was born in Eatonton, Georgia. She is the youngest of eight children. Her parents were Willie Lee Walker and Minnie Lou Tallulah Grant. Her father, who was, in her words, "wonderful at math but a terrible farmer," earned only $300 a year from sharecropping and dairy farming. Her mother added to the family income by working as a maid. She worked 11 hours a day for USD $17 per week to help pay for Alice to attend college.