Williams was married to actress Barbara Ruick from 1956 until her death from a cerebral hemorrhage in 1974. They had three children: Jennifer, Joseph and Mark. In 1980, he married Samantha Winslow. They live in Los Angeles, California.