A letter reading: "Next time it will be a bomb, you Muslim scum," was posted to the Jamia Masjid Abu Bakr mosque and was signed with a number code which is believed to be used by neo-Nazi groups.
Mosque secretary Sabir Hussain called it "appalling and very frightening".
Ch Supt Rob Odell said police had "offered security advice" to the town's mosques.
Mr Hussein claims the Muslim community has felt "under siege" since the publication of the Jay Report into the sexual exploitation of teenage girls by men predominantly of Pakistani heritage.
More on this and other South Yorkshire stories.
He said community cohesion was "very low" in the town following far-right marches.
"There is never a day goes by when we don't get abuse.
"At first we thought the letter might be a prank with somebody messing about, but when we looked at the code and checked it on the internet then that made us think this is very, very serious and not just an ordinary prank."
Rotherham has seen 15 protests over the past two years in reaction to the report, which found at least 1,400 girls were abused between 1997 and 2013.
A bomb threat received by a mosque in Rotherham was "credible", according to police.