On September 3, 1935, when Phelps was five years old, his mother died of esophageal cancer. After the death of their mother, Phelps and his younger sister were raised by their great-aunt Irene Jordan in Meridian. In 1950, Jordan died in a car accident. On December 25, 1944, Phelps' father married Olive Briggs, and Phelps and his sister were raised by their father and stepmother in Meridian.