At the end of the day the only thing we all can agree on is that the guy with a swastika on his face yelling "I hate n----s" and swinging a baseball bat is a "racist." 

As I've mentioned elsewhere, guys like that, whatever their shortcomings, have had little part in instituting, and profiting by, the social structures that systematically advantage Whites and disadvantage Blacks and other minorities. The main thing they offer is a scapegoat, a convenient distraction from the share we all have in systemic injustice based on race in this near-democracy. And that is why I try to steer an understanding of "racism" away from the highly contestable matter of individual attitudes, personal sin, and towards the impersonal social structures that perpetuate injustices present since before our founding as a nation.  

I don't mean to contradict your view, which seems sound, in itself. I just think that if we actually aspire to a just society, that goal will be better served by a broader approach.