I think the outrage is because we consider ourselves to be fighting racism when we marginalize the overt expression of racism ("Saffron Colonial" food, a swastika tattoo, a confederate flag, a gentrified historically black neighborhood). 
We are, however, failing to do anything about racism when we express outrage. Lobby for affordable housing near business centers to reverse gentrification, and for school funding to not be so heavily dependent on local property taxes (wherein poor neighbhorhood = poor school); reject NIMBYisms when we advocate for the changes in our neighborhoods (which necessarily forces negative changes on neighborhoods too busy/poor to advocate);  hire someone who is NOT a cis-het white man, even if that person is the most qualified (because he's had the most opportunities). 
Sure, boycott a restaurant, but also, examine whether that action is enough to categorize you as "not a racist", or if it just makes you a person who "doesn't want to talk about it"...