It's like when a man tells me that women have never complained to him about his ability to please a woman. I don't know a lot of women who would offer such criticism. So just because it isn't vocalized doesn't mean the negative sentiment is not there.
I have broken up with PLENTY of men who turned me off physically. I never told any of them that was the reason.