I'm sorry, while I agree with your second and third paragraphs, I think to a large extent these terms (White male privilege/colonizers) are indeed pejorative. Perhaps not universally, but largely, yes. I tolerate the former, but the latter is certainly not fair. 

I was born here, and didn't colonize anything. I am not responsible for this historical fact and should not be made to feel different or in some way morally culpable by virtue of my identity. Othering isn't okay when either side does it, and it is not productive. Put more simply, neither you nor anyone else gets to name me. I choose my identity, and I expect that choice to be greeted with respect. 

Remember the term microaggressions? Why would we assume these only work one way? 

I'm never in favour of terms that other people. By definition, that is what is happening here.