Silvers was born on May 11, 1911 in Brooklyn, New York. Silver was married to Jo-Carroll Dennison from 1945 until they divorced in 1950. Then he was married to Evelyn Patrick from 1956 until they divorced in 1966. Silvers had five children with Evelyn. Silvers died on November 1, 1985 in Century City, California from heart failure, aged 74.