Swift was born on January 16, 1924 in New York City, New York. He was married to Lenore Loveman until his death in 2010. They had three children. Swift died on April 18, 2010 in New York City, New York from heart failure, aged 86.