I haven't tested it directly myself with mercenaries, but I'm betting it's just the pathfinding bug that plagues any fort mode creature that can fly. You'll see the same thing happen if you mod dwarves to fly.
The problem is that the game calculates reachability to a tile based on moving on surfaces, and doesn't take flying (and probably still swimming) into account for that. Even if something can fly somewhere, if there's no walkable path to get there they won't go under orders and likewise won't think they can reach it if there's a job to do there. That means that, yes, a moat will keep things from flying into or out of your fort.
I'm reasonably sure that behavior applies to invaders too, and by my understanding it should. However, if I'm remembering right, if the tile is reachable on foot at all, things can still fly over obstacles to get to it. So, having a bridge over the moat will let things fly over it elsewhere.
The problem in this case is that there's no walkable tiles into the air, so they won't go after flying things by flying after them. Annoying, I know.
It's extra annoying because flying things seem to have a much higher propensity for wandering up trees and onto the tops of buildings and subsequently getting stuck. A major issue I've run into with modded fort mode is that flying things will calculate a path that includes the top of a building (remember, you only need the destination to be reachable on foot, not the whole path), and once they get onto the building for whatever reason they're forced to recalculate their path. Uh oh, no way down, so they just stand there like idiots and dehydrate and die.
I've seen people suggest modding the ability to fly out for anything that's part of your fort, and it's hard to disagree.