Given that there are over 200 municipalities in the US that have regs on the books allowing people to use the washroom corresponding to their gender as well as numerous school districts, several states and who knows how many private businesses, if there was going to be a problem you'd expect to be able to find hundreds, if not thousands, of cases.  Instead there hasn't been a single reported case of a transgendered individual causing a problem in a washroom.  Fact is, the people at risk of assault are the transgendered individuals themselves, not the general public.

Moreover there hasn't been a single reported case of someone using transgender protections to gain access to a washroom in order to commit a crime.