The Civil War began in neat lines and ended in trenches and rifle pits. The Confederacy's need to conserve lives and defensive posture lent itself to digging in; by the time of the final campaign it was standard procedure to dig a trench at the end of a day's march. Union forces occasionally were too green to wise up. The "Heavy Artillery Battalions", infantry formations freed up from Union defenses, performed excellent and precise marches shoulder-to-shoulder into Confederate grapeshot. Going to ground was no longer seen as cowardice.
And that's something worth repeating; when you look at a military technique or tactic that was continued well past its usefulness, or see some effort that surely any individual would know better than engage in, consider the extreme social stigma of cowardice. From giving up a seat on a lifeboat in the Titanic disaster to charging into the guns at Ypres, the idea of certain death is less painful than surviving as a coward. To be frank, this is the distinction that makes the Western concept of manhood.
That thread-derailing bombshell aside, the tactical see-saw between open-order and closed-order fighting (scattered or formations) has swung back and forth several times before the advent of gunpowder. The Iliad describes a series of individual duels between social equals in a manner that the samurai would take to ritual extremes. Centuries later the Greek city-states had adopted mass closed-order formations, the phalanx, which amounted to a solid wall of the city's men of property fighting as armored infantry (a bronze helmet is expensive) while the city's poor fought as slingers and javelineers in open order ahead of that line and cavalry from the city's financial elites (horses are very expensive) holding the ends of the phalanx. Taken as a whole this meta-formation, or order of battle, incorporated loose open-order troops for scouting, raiding and screening the heavily armored body of infantry. These infantrymen squared off against their opposing equals in the sort of pitched hand-to-hand battle they were best at, while their cavalry support harassed the enemy's flanks, drove off their skirmishers, or countered the enemy cavalry.
But crucial to understand is that there is more to the battle than simply the men on the field. There is also the "baggage", the food, supplies, tents, clothing, camp attendants, mules, carts, armorers, surgeons and women (of assorted professions or relation) needed to support several thousand men in the countryside. For most of history, this was amassed at the campsite selected the night before the battle. This is the root of protecting the rear; each army has some interest in reaching the enemy's baggage and either looting or burning it. This has long been the interest of cavalrymen as they had the speed to reach and withdraw from the enemy camp. In addition, the baggage is usually on or near to a road, and given the odds, these roads lead to the respective territory of the army. Capture the enemy's baggage, and at least the enemy army will be cold and hungry the next night. The loss of these supplies can bring the rapid end to an army's ability to remain in the field.
The natural solution to defending a point while attempting to attack another point is to stand between the two points. As a matter of geometry, you will be well placed to intercept anyone coming toward your camp. Meanwhile, if you have many people in this contest, it makes sense to form a line facing the point you intend to attack with the point you wish to defend behind you. This line is called the front, its ends the flank, behind it the rear.
Two masses of the same amount of men facing one another will both adopt a line of approximately the same spacing and length. This is due to various dynamics. First, if every yard of the front on your side is ten men deep, but every yard of the enemy is twenty men deep, (his front is half as wide), you are at an advantage. Only the first two or three ranks in a front can be in actual combat, while those behind them are unable to use their weapons. The longer line will bend around the flanks of the shorter line, encircling it and pushing the mass of the enemy in on itself. The longer line is able to use more of its combat power. This resulted in the Roman disaster of Cannae; Hannibal concealed the weakness of his center versus the strength of his flanks, and so the Roman army did not lengthen their line.
Lengthening a line in excess of the enemy's is called extending the flank. One can either extend the flank as well, form a front to the flank, or refuse the flank. A front to the flank creates a new line opposing the enemy's extention at an angle to the main body, creating a J or L shaped line. Gettysburg was a front to a flank. It allows the forces placed on the flank to be closer to the center of their army and reduces how far the need to travel. These are called interior lines. The enemy then has to attack along exterior lines; travelling farther from their body to attack the enemy. This is not only tiring, but in an era of horseback communications, reduces the efficiency of command and resupply. No RTS simulates a message horse riding from the general to the commander of a unit, but if you imagine one being required to issue an order, you would realize the sluggish nature of exterior lines (as well as why the German adoption of the radio allowed the blitzkrieg). Understanding the advantage of interior lines explains a tremendous amount of operational or strategic thinking. Refusing the flank is ignoring the enemy's extension on a flank in order to focus your forces where they will do more good. If your flank is rough terrain or defended by light forces who can harass and retreat, it may not require reinforcement. The battle of Leuctra was an example of an army simply ignoring one of their flanks to overwhelm the opposite.
I'll come back to write more later, I have an afternoon to fritter elsewhere.