That's because the only real harm that can come from infidelity is emotional and can only occur AFTER the infidelity is known. Without knowledge of the crime, there is no harm. People can live their entire lives blissfully unaware of infidelity, and their lives are not affected in any way. Thus, it is victimless.
The harm stems only from knowledge of a past action, not the actual action itself.