Perverts?  Fact is, there are over 200 municipalities in the US alone with bylaws on the books allowing people to use the washroom corresponding to their gender.  Add in several states, however many school districts, and who knows how many private businesses and, if this was going to be a problem this press would be full of hundreds, if not thousands, of cases.  Instead there hasn't been a single case of a transgendered individual causing a problem in a washroom.  Reality is the people at risk of assault (physical and/or sexual) are the transgendered individuals themselves.  Not the general public.

But if this really were about safety you'd be lobbying to ban Sunday Schools and bible camps - after all, kids really have been assaulted in those places.