Cardy was born on October 20, 1920 in New York City, New York. He studied at the Art Students League of New York, studying life drawing. Cardy was married to Ruth Houghby from 1947 until they divorced in 1969. They had one son. Cardy died on November 3, 2013 from heart failure at his Miami, Florida home, aged 93.