Duvall was born in San Diego, California. His parents were Mildred Virginia Hart and William Howard Duvall. He studied at Severn School in Severna Park, Maryland and The Principia in St. Louis, Missouri. He graduated, in 1953, from Principia College in Elsah, Illinois.