Graves was born in Knoxville, Tennessee. He studied at the University of Tennessee, where he played college football. Graves was best known as the head coach of the Florida Gators football team of the University of Florida. He served twenty years as the athletic director of the university.