Bergen was born in Knoxville, Tennessee to a Southern Baptist family. She was married three times, all ending in divorce. She was first married to Jeffrey Endervelt from 1950 until 1955. She then married theatrical agent/movie producer Freddie Fields from 1956 until 1973. They had three children, two of which her adopted. She was later married to Jeffrey Endervelt from 1982 until 1990. During her marriage to Fields, she became Jewish.