Morgan was born in Lexington, Kentucky, to Charlton Hunt Morgan and Ellen Key Howard Morgan. Part of a line of Southern plantation and slave owners on his father's side, Morgan was a nephew of Confederate General John Hunt Morgan; his great-grandfather John Wesley Hunt had been the first millionaire west of the Allegheny Mountains. Through his mother, he was the great-grandson of Francis Scott Key, the author of the "Star Spangled Banner", and John Eager Howard, governor and senator from Maryland. Following the Civil War, the family fell on hard times with the temporary loss of civil and some property rights for those who aided the Confederacy. His father had difficulty finding work in politics and spent much of his time coordinating veterans reunions.