And if someone says they find you creepy, that's an opinion, not defamation.

My experience of the so-called whisper network is that women are actually very subtle about what they say.  It's usually things like "you might want to be careful with that guy."