Carroll began dating Charlann Harting near the end of 1950. In mid-1951, they parted ways to attend college – Harting, whose family was better off financially, at the University of Kentucky and Carroll at nearby Paducah Junior College. After their first year, Carroll and Harting decided to get married. The ceremony took place on July 22, 1951, and the couple eventually had four children – Kenneth, Patrice, Bradley, and Ellyn. Ellyn, born June 27, 1975, was the first child born to a Kentucky First Family while they were residing in the Governor's Mansion.