He told supporters that it was shameful that some top officials were earning $18,000 (Â£13,000) a month, while others were paid as little as $140.
Officials not ready to accept the new $7,000 monthly wage limit should "start looking for alternative jobs", he said.
Mr Magufuli has promised to cut wasteful public expenditure in office.
The changes would come into effect in time for the start of the next financial year, which starts on 1 July,  the president said.
He likened the gulf in wages to a few "angels residing in heaven", while the majority "languished as if they were in hell", Tanzania's The Citizen newspaper quotes him as saying.
Junior civil servants would see their salaries increase as part of the new policy, Mr Magufuli added.
The president was speaking from his home town of Chato in north-western Tanzania in his first visit since taking office last October.
He also used his speech to announce a drop in the current income tax rate from 11% to 9%.
"It's true we want to collect tax, but we must also understand what the working class takes home," he said.
On Tuesday, a US government aid agency withdrew $472m (Â£331m) of funding for a Tanzanian electricity project after criticising the government's handling of elections in Zanzibar.
The president appeared to react to the move in his speech by criticising over-reliance on foreign aid:
"We need to stand on our own. Work hard so that Tanzanians can get rid of donor dependence."
Nicknamed the bulldozer, Mr Magufulil has announced a range of cost-cutting measures since coming to power including cancelling official celebrations for independence day.
Tanzanian President John Magufuli has promised to slash the salaries of senior civil servants, cutting the top wage threshold by almost two-thirds.