Riley was born on January 2, 1933 in Greenville, South Carolina. He studied at Furman University and at the University of South Carolina. Riley is currently married to Ann Yarborough. They had four children.