You don't have to go to a mosque to meet Muslims but you shouldn't despise people you've never met. That would be prejudging them which would be prejudiced. Why are you so sure Muslims despise your lifestyle if you've never met any? Don't you think you might be imagining it? 

As for how your taxes are spent, I think you may be overemphasizing how much goes towards recent immigrants. There are supports available for immigrants and sometimes it takes them awhile to become productive contributors to the economy, but immigrants, overall, are a source of economic growth. More people working and buying things from each other makes for a more vibrant economy. I live in Toronto so I see all kinds of immigrants contributing to the economy. In Parkdale where a lot of them start out, there are all kinds of small businesses owned by recent immigrants. I bet they pay plenty of tax not to mention the sales tax they collect. 

If people are calling you racist it's probably because you are.