Durante was married to Jeanne Olsen from 1921 until her death in 1943 from heart disease. He was later married to Margie Little from 1960 until his own death in 1980. Little and Durante adopted one daughter, Cecilia Alicia.