The next year, the goblins brought trolls with them, and I repeated the maneuver, trapping every single goblin and every single troll in the hallway full of traps. This appears to be a very repeatable strategy. I'm new to the game, so maybe I'm rehashing old ground, but none of the strategy guides I've seen mention it, and it's so powerful you'd think it would be a stock technique. Thoughts?
Exploiting the AI is already a tried-and-true defense strategy that underpins most of DF play. The fairly basic "double entrance" that has the wagon weaving around traps the enemies dutifully pile into is (at its heart) a pathfinding exploit. I'd suggest they don't include it because they're writing it into the bones of their play.
Is the line-up-behind-the-traps aspect of the invader AI as much of a game balance issue as I think it is?
Yes and no. First, none of the intruders you don't capture will die or disappear - they're there for the long haul, generally. Moreover, if they get loose, they'll pick up where they left off. They're not entirely 'harmless'.
They are, however, rather easily handled, which will remain true until their capabilities expand. They can't dig, they can't phase through the traps, and so they can't progress. They can, however, potentially fill up your cages and leave your side route "full" when the next siege comes calling.
Is that better or worse than simply drowning them all outright?
One way to mitigate the 'leader trapped no one moves' issue is to move to a model with weapon traps interspersed in the cage traps so as to make "hanging with the boss" a non-viable strategy, or simply to reduce the number of leaders captured (which itself mitigates the stalling). Leading with a few weapon traps before the cages can help, as the leaders tend to be up front and get diced.
Most defense methods (floods, siege engines, murder holes, firing galleries) will synergize with cage trap stalling to permit dwarves to continue attacking the siege, if exploiting them is the goal.