In 1945, Kramer had an annulled marriage with actress Marilyn Erskine. From 1950 to 1963, he was married to Anne Pearce. In 1966, Kramer married Karen Sharpe and lasted until Kramer's death in 2001. Kramer had four children, two with Pearce and two with Sharpe.