Armstrong was married to Ann Neale from 1952 until they divorced in 1972, they had four children. Later he married Susan M. Guthrie from 1973 until they divorced in 1976. Lastly he married Mary Craven from 1993 until she died in 2003.