Peter Lewis was brought in at a cost of Â£318,000 to the council 18 months ago to lead the struggling department.
But Somerset County Council terminated his contract last week after a "series of disappointing reviews" into some of its children's homes and centres.
However Mr Lewis said he already handed in his notice and was leaving for "private and personal reasons".
But he said despite being "slow and steady", progress was being made in the department, and "there's lots of evidence of things that are improved".
"That was the advice we were given, to not rush at things and make sure it was sustainable," he told BBC Somerset.
Mr Lewis said improvements included the restructuring of the department and making visits to children on child protection plans on time.
But he admitted: "There were still things that needed to be done".
Mr Lewis said he handed in his notice to his agency Penna three weeks prior to Somerset County Council's announcement that he had been let go.
Somerset County Council said it received no formal notification from Penna, the agency which employed Mr Lewis, of his intention to resign.
Penna said it is not something they are able to comment on.
A Somerset children's services boss who was sacked by the council claims he had already handed in his notice.