In 1947, he married Patricia Scoman. They have three daughters Cheryl Ann Dougherty (born 1947), Vivian Kathleen Dougherty (born 1950), and Maria Irene Dougherty (born 1952). Dougherty and Scoman separated in 1972.