Miller was born on February 3, 1920, in Charleston, West Virginia, the son of George E. Miller, a steel company executive and Florence (Armitage) Miller. Soon after his birth, his parents divorced, and he lived with his mother during the Great Depression, attending public school and graduating from Charleston High School in 1937. He moved with his mother and stepfather to Washington D.C. and attended George Washington University for a year. His family practiced Christian Science, which required turning to prayer, rather than medical science, for healing. After his stepfather was transferred to Birmingham, Alabama, Miller transferred to the University of Alabama.