"No one should do so for anything less than the reasons we would silence someone's expression today."

I think the point that you're missing, Richard, is that a number of us think that based on their original intent and placement (i.e., assertions of white supremacy before public institutions like courthouses), these statues constitute hate speech.  They're meant to tell Blacks that these are White people's institutions, and so not to expect the organs of the state to do anything about things like discrimination, restriction of your right to vote or lynchings.  Thus, they meet your test above.