I think you can put your dwarves into the military, set them to stay armored when off duty, and then set their schedule to "no orders" to let them carry on with civilian tasks.
I've got an axedwarf right now who had her right arm ripped off by a forgotten beast. She spends all day standing on the spot where she lost her arm, trying to pick up her axe (or perhaps her gauntlet) and failing. At one point a Rutherer came wandering by and she spent a week chasing it all throughout the fort pounding on it with her bronze buckler. Wham, wham, wham, wham, not letting up for a moment, bruising every internal and external organ, harrying it as it crawled piteously through its own vomit trail as it sought any means of escape. Passed right by one of the other axedwarves who was on guard duty, fully armed (heh), but he decided for some reason not to help with the fight. Perhaps the crippled axedwarf was working out her anger over the loss of her arm and the other dwarf didn't want to get between her and her punching bag.
Eventually she got thirsty, and ping! She switched back to being a civilian and let the Rutherer be. Perhaps she finally came to terms with her injury and decided she really needed a stiff drink instead. Poor battered Rutherer crawled off to lick its many, many wounds.
I checked back a month later and she's standing over her axe again, still trying to pick it up. I marked her shield for dumping and had it taken away but that didn't seem to help. Oh well. I'm not going to disturb her further, I saw what happened to the last creature that tried.