Can people be against more racial diversity and not be racist? I think so. Whats wrong with loving your own culture? If too many people from other cultures arrive too quickly you lose your culture.  Small ex: how long is your kid going to celebrate Halloween and go trick or treating if no one else is taking part? 

Another reason to be against high immigration is the increase numbers of people in the labor market bringing wages down especially in the lower skilled jobs.If this is true or not isnt the debate, but I dont think these people that believe this are racist either.

This is without mentioning that the white person takes a back seat, through affirmative action type policies, more and more frequently now when it comes to jobs, promotions, and university admissions. Why would a white person want more people getting in front of him just because of their color?

There are no doubt idiots out there that hate people of color or different religions but I believe their numbers are low