"It is difficult to label someone a bigot or a homophobe or whatever without labeling yourself."

as a non-bigot? Imagine your statement if you said "it is difficult to label someone a racist...." 

I am ok labeling people as bigots if they are bigots.