Durning was born in Highland Falls, New York on February 28, 1923, the ninth of ten children. His three brothers James (Roger) (1915–2000), Clifford (1916–1994),and Gerald Durning (born 1926), and his sister Frances (born 1919) survived to adulthood, but five sisters lost their lives to scarlet fever and smallpox as children. His mother, Louise (née Leonard; 1894–1982), was a laundress at West Point, and his father, James Durning (1883 – c. 1939), was an Irish immigrant. His mother was also born in Ireland.