Warden was born on September 18, 1920 in Newark, New Jersey. He is of Dutch-Irish ancestry. Warden was raised in Louisville, Kentucky. He was married to Vanda Dupre from 1958 until his death in 2006. They had one son, Christopher. Warden died on July 19, 2006 from renal failure in New York City, New York, aged 85.