That is really between the two parties involved in the marriage: rules vary from one married couple to the next.
Some people have open marriages; this works for some people.
Obviously, they are not 'breaking their vows" if monogamy was not part of the vows they made to one another.
Nearly all marriages, even open ones, have rules of one sort or another, but the rules vary from one marriage to another, depending upon what the two parties involved feel comfortable with. So, what entails a 'breaking" of the rules or a breaking of trust in a marriage depends upon what the rules or boundaries were in the first place.
Rules, both formal and informal/implicit, also tend to evolve- either in the direction of becoming more lax or, conversely, becoming more stringent- over the years.