No, being scum isn't about moral judgment, it's about doing scummy things.  

White supremacists spewing hateful things is scummy.
Showing up to respond with violence is scummy.
They're BOTH scum.

But in the eyes of the law, physical violence is considered worse than verbal offensiveness.  In fact, most of the latter is actually protected by the law as free speech.