since the game is pretty detailed, i don't think some sort of arbitrary formation bonus(es) would be appropriate or necessary. that said, if you look at ancient formations there was actually only a few different factors that formations did:
for shields:
shoulder to shoulder - allowed large shields to protect someone next to you, as well as yourself. also both brace the shield.
- allowed smaller shields to be braced by someone next to you (like the large shield).
- bucklers don't really gain benefit
shield touching the back of the man in front of you allows non buckler type shields to reinforce the person in front of you against getting knocked down as well as helping the man in front to push forward against opposition.
i am out of time now, but i will continue later, for various weapons and such.
[edit]
spear weapons:
depending on the length of the spear you attack at a certain range, too close and you can't attack them with the point. by having ranks of men deep one is threatened from the thrust of a spear no matter how close he gets to the man in front, since someone behind him will use his spear over the shoulder of the man in front.
when one has adequate room to swing his spear he can use it as a quarterstaff to attack those that get too close, the longer the spear the more unwieldy this is.
having room behind him one may draw the spear back to bring the point closer to him so as to thrust at enemies that get too close, again the longer the spear the more unwieldy this is.
guandao/flail/staffs:
large heavy weapons design for slashing tend to need room to swing, and would not do well with people shoulder to shoulder since one could not get a proper swing in.
projectile weapons:
you need room in front of you in order to use the weapon, thus you would not be able to stand in close rank with others and still use your weapon.
slings/staff slings:
an error in firing would result in launching the projectile earlier or later than normal putting it higher or lower than intended, which if there is a friendly in front of you that you are firing over, you might hit. also like most projectile weapons you need room in front of you in order to properly use the weapon.
wrestling/estoc:
estoc was used as a can opener for armor, and thus involved grappling. grappling would need room to maneuver, allowing for dodging and wide sweeping movements, and room to fall when bringing someone down.
there i think that is it.
what does this mean in dwarf fortress? well something like things being possible for the critter to do other than normal, when standing immediately next to or behind another critter in "formation" (or unable to something they would normally be able to do). hence with say a wedge formation, the bulk of people behind the critter at the very tip of the formation, pushing him and steadying him when needed would allow him push others out of the way easier than normal, and even more so keep from getting pushed away or knocked down.
i would also assume that a critter can't move past another critter unless it allows him to (or gets knocked down/away). also one wouldn't be able to use their weapon when a friendly is in the same square, or ranged weapons if their is a friendly in the square immediately in front of them, or guandao and such weapons if one is to the immediate right and immediate left (maybe even to either side if their is a want for greater restrictions).
for wrestling, i would make it so that a critter loses its defensive bonus from wrestling when next to a critter to the immediate right, left, or back, and would gain attack penalties against opponents with its allies to its immediate right, left, or back. in a situation where a critter in a tight formation tries to grapple a critter in a tight formation, the grappler would have the advantage, since although it is penalized, the grapplee has no defensive wrestling skill (from formation) and thus needs to rely on his other defensive things (which actually might be pretty good with overlapping shields and such).
in summation this would allow effective pike squares, phalanxes, wedges, columns, and lines (assuming you are using the right weapons for such), without having any sort of predefined formations in mind for any sort of bonuses.
[/edit]