Morse was born on May 18, 1931 in Newton, Massachusetts. Morse was married to Carol Ann D'Andreá from 1961 until they divorced in 1981. He later married Elizabeth Robert in 1989. They have five children.