It's a problem because I'm a woman and I feel threatened by strangers using a bathroom who are physically stronger and capable of breaking my neck if they wanted to. So wear your dress into the men's room and see how that goes for you.