I do like a good tongue twister. But the rest of your comment, to me, can be easily judged by a simple test. A child can't give consent to sex because they are young enough to be manipulated. We make this act illegal to prevent a child being taken advantage of. 

If a person's brain is wired like the opposite gender and they want to act on that and they are old enough to consent to their own actions, is there a problem? I don't think it should be an issue just because of slippery slope sliding. 

The problem is if, as you say, everyone else has to go along with it despite believing it is sinful or weird or dangerous. People that think having a transgender person in a bathroom is dangerous probably thinks transgendered are perverts and likely to attack a child. This is demonstrably not true. They may be uncomfortable because of "the different" and I have some sympathy for that... humans need time.