He was married twice: first in 1945 to vocalist and former Goldwyn girl Gloria Delson with whom he had two children. They divorced after 18 years of marriage. In 1965, she remarried world class tennis player, Mike Franks. In 1970, he married Virginia (Tita) Curtis, a former fashion coordinator for the clothes designer Donald Brooks. He was the father of Laurie Cahn and jazz/fusion guitarist Steve Khan who, early in his career, changed the spelling of his last name to Khan in order to "create a separate identity from famous father" and because he was "so hurt and angry with him for so many childhood things."