Cabell was born into a rich Virginian family. His father, Robert Gamble Cabell II (1847–1922), was a physician. His mother, Anne Harris (1859–1915), was the daughter of Col. and Mrs James R. Branch. Cabell's great-grandfather, William H. Cabell, was Governor of Virginia from 1805 to 1808. Cabell was the oldest of three boys . His brothers were Robert Gamble Cabell III (1881–1968) and John Lottier Cabell (1883–1946). His parents separated and were later divorced in 1907.