Simmons was married twice. Her first was to actor Stewart Granger from 1950 to 1960, with whom she had a daughter, Tracy (born 1956). Simmons and Granger became American citizens in 1956. The second was to film director Richard Brooks from 1960 to 1977, with whom she had a second daughter, Kate (born 1961). Both marriages ended in divorce.