I put all my dwarves in squads with no active training months scheduled. Give them a weapon and a shield. If they're not doing their assigned task they go do individual weapon drills rather than sit around in the meeting hall socializing.
If you have a lot of extra idle hands than have three squads on a rotating "train for 2 months, no order for 1 month" schedule so at any time 2/3 are training and 1/3 are available for work. Oh, and fill the training schedule with sets of "Training - Minimum 2" so that you get max sparring. And have their equipment be worn over civilian clothing so that they carry shield and weapon about.
At a minimum you should get civilians with a minimum of weapon and physical training which makes them more ready to get armor and serious training.
I saw it pay off in my fort recently when a dwarf sent out to build a section of road was ambushed just outside the gate. He *outran* the goblins while my gate barracks squads were summoned and the marksdwarves in the overwatch position were activated. And carrying the shield paid off since he blocked a few attacks when a goblin caught up with him momentarily. (The militia from the barracks triggered a second ambush and then obliterated both of them. Goblin, meet Hammerlord. Hammerlord, meet target.)
Note: Of course the dwarf didn't run back inside the gate. Instead he bolted off towards the far gate of the fortress.