If men who think, or just want to be women user the women's room, then you are wrong. Just because  Trans people don't like their gender does not change the fact of their gender. 

It is unlikely that trans gender people have been able to disguise themselves adequately enough to sneak into many bathrooms unknown. Most are easy to spot.