Berle was married to Joyce Mathews from 1941 until they divorced in 1947. They remarried in 1949 only to divorce once again in 1950. Then he married Ruth Cosgrove Rosenthal from 1953 until her death in 1989. Then he married Lorna Adams in 1991. Their marriage would last until his death in 2002. Berle had three children.