He was responding to the release of a government report into fraudulent qualifications in the public sector.
The education minister said that some had been using their relatives' school certificates, while others did not appear on official records at all.
President Magufuli has given those on the list until 15 May to resign or face legal action.
"These people occupied government positions but had no qualifications... they robbed us just like other common criminals," Mr Magufuli said, urging officials to "name and shame" offenders.
"We have been working hard to create new jobs while there are people in government who hold fake degrees," he added.
He has also asked newspapers to publish the names of everyone on the list.
There isn't a particular place which is well known for making fake certificates here, but it's easy enough to get hold of one on the black market.
It is very common for people to use fake certificates in Tanzania.
Police raided a house in Dar es Salaam last year and found two individuals with machines, stamps and piles of templates for fake secondary school, birth and even business licence certificates stacked up in a room.
It is also very common for people to use their friends' or family members' certificates.
In this case, when someone gets a job or goes on to higher education, they will actually change their name to match the one on their certificate.
The 9,932 people named in the report account for more than 2% of the country's estimated 450,000 civil servants.
Officials convicted of fraud could face up to seven years in jail.
Last year, Tanzania removed more than 10,000 so-called ghost workers from its public sector payroll in a crackdown on corruption.
Payments to the non-existent employees had been costing the government more than $2m (Â£1.4m) a month, it said.
When President Magufuli came to power two years ago, the government temporarily suspended recruitment in some government sectors while they checked up on existing employees' qualifications.
The report focused exclusively on secondary school qualification and did not examine university degree certificates.
Tanzanian President John Magufuli has sacked nearly 10,000 civil servants for having fake education certificates.