Wrexham, Swansea, Porthmadog and the current Cardiff office will go, with the Merthyr office already closing.
Mr Jones told BBC Wales he did not see how "moving services further away from people" would improve the system.
Her Majesty's Revenue and Customs (HMRC) said it had "too many expensive, isolated and outdated offices".
Mr Jones told BBC Radio Cymru on Friday: "To centralise everything in Cardiff - what sort of message does that send to people in the north and the west?
"We as a government wouldn't centralise everything in Cardiff."
"I don't see how moving services further away from people is going to make things better," he added.
Welsh Conservative leader Andrew RT Davies told BBC Radio Wales there could be a boost from new jobs in Cardiff but HMRC had to "convince people like myself that their service will not deteriorate" and that "local intelligence won't be lost" by centralising services.
Rhodri Evans, from the Federation of Small Businesses in Wales, hoped the changes would solve the "difficulties" firms had dealing with the tax authorities online or by phone.
"Over the long-term, this modernisation programme may bring benefits and efficiency savings," he said.
"In the short-term however, FSB members will be concerned that the closure of these tax offices will simply compound existing problems."
On Thursday, HMRC chief executive Lin Homer said the current tax office structure "makes it difficult for us to collaborate, modernise our ways of working, and make the changes we need to transform our service to customers and clamp down further on the minority who try to cheat the system".
HMRC said it currently had around 2,900 staff in Wales, including 350 in Wrexham, 300 in Swansea, and 20 in Porthmadog.
Plans to close tax offices across Wales and centralise services in a new centre in Cardiff have been criticised by First Minister Carwyn Jones.