Kelley was delivered by his uncle at his parents' home in Toccoa, Georgia, the son of Clora (née Casey) and Ernest David Kelley, who was a Baptist minister. Kelley was named after the pioneering electronics engineer Lee de Forest. He later named his "Star Trek" character's father "David" after his own father. Kelley had an older brother, Ernest Casey Kelley. Kelley was immersed in his father's mission (church) in Conyers, Georgia and told his father that failure would mean "wreck and ruin".