The leather armor is no good, it will just mess with the rest of uniform by taking up the space of mailshirts and breastplates, it is really only meant for archers. Also, although this is damn exploity, assigning multiple shields to each dwarf works wonders: they don't have any problem juggling them and make block attempts with all of them in turn (this might get obsoleted by the combat rewrite in the next version)
To give your army breaks, get into the orders screen, the default order is training for 10/10 dwarves. Use the -+\keys to adjust this to a different ratio (I like 7/10) and assign this ration to each of the 12 months. Now each month of the 10, 7 will train and the other 3 will rest (assuming our squads have 10 dwarves in them, if less, adjust accordingly)
You really dont want them working civilian jobs all too much, it takes a surprisingly long time for a dwarf to find all his or her gear, and seemingly much longer when youu desperately need them to be holding a chokepoint. Worse, you might find dwarves dawdling at different rates, so that they meet the goblins one at a time, rather than altogether.
Keep your militia focused on training and they will perform admirably. let them slack off or work civilian jobs and they will likely die.
As far as burrows go, panic rooms are great for uber emergencies (say the militia got slaughtered or there is a plague spreading from the caverns) but 90% of the time, you will want your fort running normally, when a threat is present, and not huddling in a panic room. As such it is very worthwhile to make a burrow encompassing, say, everything in sight but tthe surface and the cavern--remember that burrows can be designated across z-levels. Then when goblins come, the civilian alarm will keep the workers working, but will also keep them from dangerous spots like the surface