He was appointed envoy to the Vatican, with Mr Mursi's office giving no official reason for the switch.
But Mr Mahmoud later said he would stay in his post in defiance of Mr Mursi's order, state-run Mena news agency said.
The row comes a day after 24 supporters of ousted President Hosni Mubarak were acquitted of organising attacks on protesters during last year's uprising.
Hundreds of demonstrators rallied in the capital Cairo against the acquittals.
They accused the judges of "complicity" with the former Egyptian leadership and said they wanted to "purify justice".
The group on trial had been accused of sending men on camels and horses to break up a protest in Cairo in 2011.
In the incident, later called the Battle of the Camels, Mubarak supporters charged protesters in Tahrir Square.
It became one of the most notorious incidents of the uprising and left nearly a dozen people dead.
Some senior members of the old regime were among those accused.
They included Fathi Sorour and Safwat al-Sherif, former speakers of Egypt's two houses of parliament.
Prosecutors said Mr Sherif, who was also the secretary general of Mubarak's National Democratic Party (NPD), had "contacted MPs, members of the NDP and financiers of the party, inciting them to disperse the protests in Tahrir Square by force and violence".
Mr Mahmoud was appointed in July 2006.
Officials quoted by Reuters said that an assistant to the general prosecutor would take up the responsibilities until a new prosecutor general was appointed.
Egyptian President Mohammed Mursi has removed Prosecutor General Abdel Maguid Mahmoud from his post.