Indifference to the health of the soul of others is a sin, which is fundamentally what you're suggesting.

We should be infinitely concerned with the health of the soul of others because GOD is infinitely concerned with the state of the soul of others. 

We should "convert" every day to have more and more the heart of Christ.

Our goal should be to love with the very heart of Christ.

Drawing back and drawing "walls" around ourselves is not at all in line with the Gospel message.