Goldsmith has been married to Sharon Hennagin from 1950 until they divorced in 1970. Goldsmith lastly married Carol Heather from 1972 until his death in 2004. He has five children, one of his sons, Joel Goldsmith, died in 2012. 