It's typically caused by the skulker's pals. Artifact hunters often come in groups (never declared: they just trickle in one after the other and are announced as visitors, and I'm not aware of anyone finding the data structure that link them together). Then one of them leaves, only to more or less immediately return as a skulking thief. As soon as the thief attacks a resident or your guards attack the thief the infiltrators go into action, causing mayhem. They can attack your dorfs, while still claiming to e.g. socialize, they can run around, scared of your horrible dorfs, and they cause your dorfs to be scared of the horrible infiltrators, causing widespread interruptions. Since they're not indicated as hostile, it can be a real pain to locate them. You can avoid some of these cascades by allowing skulkers that flee when detected to get away without a fight, but there's no way to avoid a fight when they decide to attack.
The loyalty cascade happens because some of your dorfs have been befriended by some of the infiltrators, and take their side when your dorfs start to defend the fortress.
The infiltrator groups can to be huge early on in the fortress history, including exceeding the visitor cap, but tend to get smaller over time (so once you're capable of dealing with large outbreaks they no longer happen). Artifact hunters (that didn't attack you) can return later as regular mercenaries. I think the number of artifact hunters plummeted together with the rest of the mercs when mercs were redirected into merc companies (which the fortress has no interactions with), though.