The comments do a good job identifying many problems with the notion of white privilege.  One problem that I don't think has been mentioned yet is that the name is a counter-productive misnomer.  That is to say, it is named wrong and that wrongness causes issues.  

Do some minorities have a more fraught relationship with public authorities or the majority culture? In many places at many times, unquestionably.  And that is shameful.  But being free from oppression and prejudice is not a privilege, and calling it that is philosophically unsound.