But wait, how far back do we take this, and what culture should we go off of? Actually, it wasn't too long ago that judges and/or parliamentary representatives wore wigs, not too much different than the one worn by the person in that picture, although they were all men. And then there's the men who had to play the female parts in Shakespeare's plays, since women weren't allowed to act. And togas seemed to look a lot alike. Should we also ban kilts or sarongs?
What is acceptable to wear is generally determined by culture and time. And, as is very evident by the fact that most women of western cultures now wear jeans and pants more often than skirts or dresses, what is acceptable to wear by either gender changes with time, and is changing even more quickly as we interact more often with the cultures of others and become more accepting of things that are not "normal". Just because something isn't normal, does not make it wrong, or even unnatural.
Technically, what is natural for all of us is to be completely naked.