Reavley was born in Quitman, Texas. He studied at University of Texas, Austin, at Harvard University and at the University of Virginia. He was in the United States Navy from 1942 to 1946. Reavley was married to fellow Fifth Circuit judge Carolyn Dineen King from 2004 until his death.