The words "white privilege" are badly chosen, and raise hackles unnecessarily. The creators of the term were looking for a shorthand way of summarizing things like the fact that identical resumes, one named Jamal and the other John, get substantially different response rates from potential employers. That can be tested as often as you like, both by sending the test resumes to real employers or by having volunteers in labs review sample resumes. The people doing the reviews honestly believe that they have no bias -- but the bias is experimentally provable over and over again. If the bias can be triggered by a name on paper, it can certainly be triggered by a face. Whiteness helps you get a job, or in any other social situation involving white decision makers.
The difficulty of getting people to accept this fact (and it is a fact) is that any individual case of white privilege is invisible, both to the employer conferring it and to the recipient. It is not deliberate, but it's there.