In 2004, Carroll was elected to the Kentucky Senate from District 7, defeating Harold Fletcher – the older brother of then-governor Ernie Fletcher – by a wide margin. The district included all or portions of Anderson, Fayette, Franklin, and Woodford counties. He made headlines in 2007 when he called on Fletcher's lieutenant governor, Steve Pence, to resign for his disloyalty after Pence endorsed Anne Northup in the Republican gubernatorial primary rather than backing Fletcher's re-election bid. Pence refused to resign, citing an investigation of the administration's hiring practices as his reason for refusing to endorse Fletcher. Fletcher won the Republican primary, but lost in the general election to Democrat Steve Beshear.