Are we talking about a consensual affair or unsolicited junk pictures? The latter has issues other than fidelity to his wife that are problematic. The former is a morality problem that, yes, I think makes for a candidate I'm less likely to vote for, but to the exact same extent that I'd be less likely to hire somebody to be a sales clerk if I'd heard that he'd slept around on his wife. Information directly relevant to the job has to take precedence (in this case, "Is he competent?" and "Does he support policies that I do?" being the major ones), but if looking at that doesn't resolve the issue, then I could go to fidelity to make a final decision. But "loyalty" isn't actually some universal value that applies to all situations equally.
Then, there's also the issue where sometimes people will see that someone needs to be punished for something and decide that any punishment is valid as long as it gets the job done. I actually don't think that's going on here so much, but I think what happens is that people who do believe that automatically assume that anybody who opposes any particular punishment must therefore oppose any punishment. And people who argue with those sorts of people have been conditioned to think that anybody who supports a particular punishment must be of that mindset, and that therefore they're obligated to point out that punishments ought to be confined to the people directly involved, and that a person ought to receive the same punishment (no more harsh, no less harsh) regardless of social status. So if a politician cheats on his wife, he ought to have to deal with a nasty divorce and the financial and social problems that go with that, just like anyone else who cheats on his wife ought to.
So you essentially have people strawmanning, completely accidentally, because if they were having this argument with a random person on the street they would probably be addressing an actual argument. Maybe. I might be completely wrong on this.