Garnant ward councillor Kevin Madge will be succeeded as Labour leader by Jeff Edmunds, from Llanelli, who represents the Bigyn ward.
The decision was made at the local party's annual general meeting on Monday.
Mr Madge has led the council in coalition with independent councillors since 2012.
He survived a no confidence vote in February 2014 after the Wales Audit Office ruled the council's chief executive Mark James had received "unlawful" pension payments.
Commenting on Mr Madge's replacement, Plaid Cymru assembly member Rhodri Glyn Thomas said: "I and my colleagues have always maintained that Kevin should have done the honourable thing and resigned his position after the council's unlawful pension payments scandal was exposed.
"It is curious, though, that those who then gave him a vote of confidence have now ousted him from office."
Mr Edmunds's name is expected to be submitted for the leadership of the council at the authority's annual general meeting on 20 May.
The leader of Carmarthenshire council has been voted out after a secret ballot by Labour councillors.