The dichotomy you describe with respect to discrimination isn't so much 'progressive' inconsistency as it is 'conservative' hypocrisy.  White supremacist social, political, legal, economic and educational policies and practices are still very much with us.  I can understand were, not really having been on the receiving end of same, that you may not have noticed.

In a 2003 University of Chicago study of over a thousand entry-level employers in New York, Chicago, Milwaukee and Los Angeles preferentially hired white applicants who claimed felony convictions over well-qualified black applicants with no criminal records.  So how does 'color not matter' when a white FELON has a better chance at a job than a law-abiding black citizen?

Saying color SHOULDN'T matter is not the same as saying it doesn't.