Baghdad said the deployment was done without consultation and was a violation of national sovereignty.
Turkish Prime Minister Ahmet Davutoglu defended the move as routine troop rotation at a pre-established camp.
Mosul has been under the control of militants from the so-called Islamic State group since last year.
Turkey deployed hundreds of its forces to the town of Bashiqa to train Iraqi Kurdish forces fighting IS.
"Iraq has the right to use all available options, including resorting to the UN Security Council if these forces are not withdrawn within 48 hours," Prime Minister Haider al-Abadi said in a statement.
Mr Davutoglu wrote to Mr Abadi promising not to send further troops but stopped short of agreeing to a withdrawal.
Turkey enjoys close relations with autonomous Kurdish regions in Iraq, although it views Syrian Kurdish groups over the border as hostile, analysts say.
The fall of Mosul was a key moment in the rise of IS and a Iraqi government offensive to retake the city has been repeatedly put back.
Iraq has threatened to go to the UN if Turkey does not withdraw soldiers it sent to areas near the northern Iraqi city of Mosul within 48 hours.