The AI isn't very complicated, but the commander bit is something I thought most people are aware of. While bandits and the like won't have a leader, armies involving nobles will most often have a commander, who stands behind the other units and gives the orders to move them forward. It can sometimes be nice to leave them with the commander so they stay in that line while advancing (it's really easy to kill their cavalry's horses when they do this), but the main reason why I prefer having them charge in as soon as possible is to avoid them being grouped together when they reach your army. If they charge earlier, you'll get a steady trickle of enemies due speed/athletics differences, whereas if they're allowed to advance in the line they'll often end up with their ranged units in a good position and all the infantry attacking at once (while the cavalry flank, since they're usually on the edge of the line). Knocking out the commander also gives a morale hit to the enemy's army, making them more likely to retreat and panic.
I think you're right that taking out the commander breaks up the incredibly dumb "march forward in formation by stages"
But, that's hardly AI... I still have only observed a binary thing:
1) Stand in a line and walk forwards, stop, walk forwards, stop, eventually charge
1a) If enough people or the commander die, skip to 2
2) just charge with individual targetting
And 1 is a godsend. I'm like, "Oh good, they've decided to stand there and be cut down for a while." It's usually a reward for facing a large force. I park my men at the starting position and just murder a bunch of clueless morons. Some of them do shift to face me, but they're forced to keep marching in the retarded formation. And most of them keep facing away. It's great, and it's also horrible.