Marko Arnautovic's red card in Wednesday's 0-0 draw with Southampton was the first issued to a Potters player this season.
Hughes told BBC Radio Stoke: "It was simply a mistimed tackle.
"It was not done with intent or violent attitude and should have been viewed as such. But there's no point appealing."
He added: "There's no chance at all that we might get that rescinded.
"Unfortunately, officials come here and think an early yellow card will settle down Stoke. I know for a fact this is what they talk about.
"Those days are long gone. We play a different way and have done for a long time. But the perception is still there that referees come here and they have to calm us down."
Hughes was sent to the stands on referee Anthony Taylor's last visit to Stoke - the 4-0 home defeat by Tottenham in September.
Arnautovic will be suspended for three games for his high challenge on Sofiane Boufal, his second ban of the season having already been shown five yellow cards.
"The ref wasn't great on the night," Hughes said. "He missed a number of decisions and clearly missed an elbow from Shane Long on Bruno Martins Indi. That needs looking at again.
"If you're going to make game-changing decisions like that, then all the other key moments and decisions he has to get right too."
After becoming only the fifth man in English football to both play in and manage 400 top-flight matches, Hughes said that he had seen many better games in that time, but few that had made him more proud.
"To a man, we were excellent," he said. "It could have been very easy to accept our fate and wait for them to score.
"Maybe a lot of people here thought that was going to be the sequence of events, but, in the end, we saw a rearguard action.
"I know I have the players who can dig in and be resilient. We had to do it for 70-odd minutes.
"Apart from one outstanding save from Lee Grant, they did not create too much of note and that's more to do with how well we did."
Manager Mark Hughes believes Stoke City still sometimes suffer from a past reputation when it comes to their treatment from Premier League referees.