QTEs tend to happen in cinematics. So you're busy watching the cinematic, thinking you can relax a moment from the gameplay, until suddenly no! You have to press X to not die. If you weren't expecting it, you'll scramble and probably fail the first time. And then, in all future cutscenes, you need to be watching for it, instead of simply appreciating the scene, distracting you from what's going on. There is also the tendency that its the "important" fights that get QTE events, with the effect that it is not really you as the player that beat the boss, you just loaded the movie and pressed play (and stop and fast forward and play and play again) to watch the scene where the boss is beaten.
I think that is where most of the irritation comes from, it is almost a betrayal of established expectations (in terms of gaming history, not as established within a particular game or franchise itself).
Some games handle it well or better, some do not. For example, God of War bosses are at least still an action scene, so you're not betrayed by the expectation that you can relax. But it is still a bad idea, because following on-screen directions to press or repeat single buttons is weaker than having full freedom of action to fight The Big Fight.