Richards was born on September 1, 1933 in Lakeview, Texas. Richards was raised in Austin, Texas. She studied at Baylor University. Richards was married to David Richards from 1953 until they divorced in 1984. They have four children, including former President of Planned Parenthood Cecile Richards.