The 32-year-old former Ghana midfielder asked Daniele Minelli to stop Sunday's Serie A game at Cagliari.
But he was instead booked for dissent in the 89th minute, prompting the former Portsmouth and Sunderland player to leave the pitch in protest.
He angrily confronted Cagliari fans, shouting: "This is my colour."
Speaking after the game, Muntari said: "The referee should not just stay on the field and blow the whistle, he must do everything.
"He should be aware of these things and set an example.
"I asked him if he had heard the insults. I insisted that he must have the courage to stop the game."
Pescara boss Zdenek Zeman, whose side lost 1-0, said: "He asked the referee to intervene, but he [said he had] neither heard nor seen anything.
"Muntari was right, but he shouldn't have left the pitch. It's not up to us to dole out justice. We can talk a lot about it but then it must be left with the powers that be.
"Today this has happened when Muntari has already played in Italy for many years. We hope that mentalities will change."
Muntari was at AC Milan when then team-mate Kevin-Prince Boateng walked off the pitch because of racist chanting during a friendly with lower-league side Pro Patria in January 2013.
It prompted a wave of support on social media. Fifa, football's world governing body, applauded Boateng's principles, but said it did not condone his decision to walk off.
Pescara are bottom of Serie A and have been relegated to Italy's second tier.
Pescara's Sulley Muntari walked off the pitch in protest when a referee booked him after he claimed he was being racially abused during a league match.