Speaking to Parliament's Treasury Committee, he said: "The point at which interest rates may begin to rise is moving closer.
"Once rates begin to adjust, we expect for those adjustments to be at a gradual pace and to a limited extent."
UK rates have been at the record low of 0.5% for more than six years.
Most analysts do not expect the Bank to raise rates until next year at the earliest.
The pound rose by 1.5 cents against the dollar - 1% - to $1.5637, in response to his comments.
Against the euro it was also up 1% at $1.42.4.
Mr Carney said that the Bank would be able to raise rates due to "consistent growth above trend, a firming in domestic costs, counter-balanced by disinflation imported from abroad".
However, Mr Carney also said rates would not increase to the levels seen before the financial crisis.
"I do think there are a variety of factors that mean that the new normal, certainly over the policy horizon over the next three years, is substantially lower than it was previously," he said.
"I see no scenario in which they would move towards historic levels."
Mr Carney was also asked for his view on the statement by the euro group leaders providing for Greece's third bailout in five years.
He said: "What's embedded in that statement will require Herculean efforts from all sides.
"The scale of structural reforms, the scale of fiscal adjustment, the scale of privatisation that will ultimately be required are significant.
"The process by which this agreement was struck, the nature of the agreement, the scale of the challenge underscore the number of institutional shortcomings that still exist with European Monetary Union."
Mr Carney also said the Bank of England was looking into the impact of the introduction of a national living wage announced in last week's Budget by Chancellor George Osborne.
The findings will be revealed in the Bank's quarterly inflation report next month.
Mark Carney said the rise in the wage was likely to push up overall wages by about 0.3%.
But he said the impact on inflation would be limited because not all labour costs borne by businesses would be passed on to consumers.
"In terms of overall economic impact, whether on inflation or activity or productivity, it will modestly move all of those up - modestly but not materially."
The pound rose by more than a cent against the dollar after Bank of England governor Mark Carney said an interest rate rise was getting nearer.