Instead of talking about adventurers, let's talk about Commandos instead. The motivations become a lot clearer when you aren't thinking in terms of TTRPGs. Enemy nations would be happy to send a squad of commandos at you. Bandits would be more than happy to do so. They aren't just random guys wandering the landscape, they're men with a purpose.
The result would be similar of course. Champions coming in with the skills to infiltrate, avoiding your traps, sneaking around, staying out of sight when possible, and kicking ass when finally caught.
They'd be showing up to accomplish a SPECIFIC task, not just indiscriminate destruction (though that might happen as a matter of course). They'd be sent to steal an artifact, or adamantine stuff, or raid your vault (for when coins actually matter... one day maybe). They'd be sent to assassinate a noble of yours, probably one of the military ones like Duke or even King. They'd sneak in, kill anyone who gets in their way, and get out, leaving chaos in their wake.
Adventurer raids I'd object to, since they'd be too metagamey for my tastes. Commando raids though... that could be awesome. Of course, the commandos would likely consist of adventurers, and probably got their skills slaying monsters in the countryside...