Sir Liam Donaldson told the BBC there needed to be financial savings through staff cutbacks.
Last week, Health Minister Simon Hamilton announced his intention to abolish the board but said it could not happen until April 2017.
Sir Liam says it should happen sooner.
"The last thing you want is an organisation which is dying in management terms running the show for years," he said.
Mr Hamilton rejected Sir Liam's idea that an international panel should be given the power to make difficult decisions such as reducing the number of acute hospitals.
Instead, the health minister believes the panel should be made up of Northern Ireland experts and politicians should make the decisions.
"He has talked of political courage and political consensus - so whilst the letter of the recommendation of bringing in an external panel to advise is not being taken - the spirit of it very much is," he added.
Many of the 600 staff at the Health and Social Care Board are expected to be redeployed to the Department of Health or the trusts, but Sir Liam says real savings are needed.
"I would expect to see a reduction in unnecessary management costs because ultimately that same money can be redeployed to frontline patient care where it is very very badly needed," he said.
A senior medical expert who reviewed how the health system is run in Northern Ireland says the Health and Social Care Board should be done away with as soon as possible.