Scott was married to Carolyn Hughes from 1951 until they divorced in 1955. Later he married Patricia Reed from 1955 until they divorced in 1960. Thirdly he married Colleen Dewhurst from 1960 until they divorced in 1965, then again they married from 1962 through 1972 after they divorced again. Lastly Scott married Trish Van Devere from 1972 until his death in 1999. He and Karen Truesdell had a daughter Michelle.