Local media say he will now face trial for criminal abuse of office.
Mr Tomana was suspended back in February 2016.
His removal came after he dropped charges against two army officers accused of plotting to blow up a dairy belonging to Mr Mugabe's wife, Grace.
The two men were among four arrested outside the dairy farm, north of the capital, Harare.
They were found carrying ammonia and petrol bombs.
The men were initially charged with possession of weaponry for sabotage, and with money laundering for terrorism purposes. Charges of treason were added later.
The four were accused of hatching a plan to set up a militia base to the west of Harare, with the aim of unseating Mr Mugabe's government.
Mr Tomana was accused of obstructing the course of justice by releasing two of the suspects, which he denied.
In 2015, Mr Tomana was convicted of abuse of power after refusing to prosecute a lawmaker with Mr Mugabe's Zanu-PF party for raping a child at gunpoint.
He was threatened with 30 days in prison over the matter, but eventually agreed to act.
The legislator, Munyaradzi Kereke, was later convicted of rape and imprisoned.
Zimbabwe's President Robert Mugabe has sacked the country's chief prosecutor, Johannes Tomana, after a tribunal found him guilty of misconduct and incompetence.