It is the social perception - based on thousands of years of observation - that women are able to act in entirely non-sexual behaviors while men do not. A man wearing women's clothing is perceived as being frustrated sexually, so frustrated that he is attempting to be attractive to other men in the hope/expectation of sex. However, women are not perceived as sex pursuers, and so are free to wear men's clothing - or any clothing they want. It is a double standard, but like most double standards with women and men, we can be thankful for it.