You simply cannot equate neo-Nazis, the KKK, League of the South etc. with the people who show up to protest their hate speech. 
All hate is not created equal. Some of the violence in VA was on the right side of history. Some of it, from the fascists, was not.

Racism is, effectively. illegal in the USA. On the other hand, attempts at racial justice--however halting and half-hearted--have the imprimatur of law and the state. It's pretty easy to take this one step further and say that anti-racism is patriotic, whereas white supremacy is  . . . treasonous, maybe, or at least seditious.

The real failure here is with law enforcement. They didn't control the confrontation. They didn't close off the streets where the unfortunate Ms. Heyer was murdered during a large peaceful march. Freedom of speech doesn't protect the fomenting of violence, It's probably time to have laws that foreclose such possibly dangerous situations as the white supremacist thugs brought to Charlottesville.