Let's not forget that the issue here is: did the man touch a woman inappropriately. Presumably you're a woman and should know that every day, women get unwelcome touching, regardless of what they're wearing. 

Taylor Swift is known for modest dress. Does it matter if she's modest or if it's Katy Perry who is not modest? No! You don't touch another person without permission, period!