It's often because many people - especially the not-so-privileged white left - "forget" that some non-whites have earned privilege.  Sure, we face racism, but economically and in many ways, socially, we ARE more privileged.  I think they are often blind to that - especially if they aren't "exposed" to certain cultures (and I don't mean ethno-culture).  Step into any independent school in Ontario or BC and you'll see it right away.