After graduating from the University of Kentucky and spending three years as an Air Force lawyer, Carroll returned to McCracken County, Kentucky where he gained acclaim for leading a campaign to allow the Tennessee Valley Authority to provide low-cost electricity to the county. He was elected to the first of five terms in the Kentucky House of Representatives in 1962 and served as Speaker of that body from 1968 to 1970. He ran for lieutenant governor in 1971 on an informal ticket with former Governor Bert T. Combs. Combs lost in the Democratic primary to Wendell Ford, but Carroll defeated his primary opponents and went on to win the general election. He was elevated to the governorship in December 1974, after Ford unseated Moderate Republican U.S. Sen. Marlow Cook. Carroll won a term as governor in his own right in 1975.