Patterson is the second oldest of seven children and daughter of C. H. Patterson, a professor of psychology, and Frances Spano Patterson. She was born in Chicago and moved with her family to Edina, Minnesota, when she was young, and then to Urbana, Illinois. Her mother died of cancer when Patterson was a freshman in college and the youngest of her siblings was just five years old. This triggered her interest in developmental psychology, a theme which pervaded much of her later work.