Welcome to Tantrum Spirals!
Alright, well, first things first -
Thieves and Snatchers are a rather annoying bunch. If your Dwarves are working outside, there is little to no stopping a Goblin Snatcher from running up to a parent and snatching the baby from their arms. Children may also be snatched as well if a Goblin Snatcher can get to them. However, assuming you can create a checkpoint that all Goblins or Kobolds would have to run though, you can set up animals (or Dwarves - preferably ones without babies or Children if so) to be stationed in an area. If stuff runs next to them they will reveal the enemy.
A Second form of defense you might use is to simply turn your checkpoint into a gate or bridge or something; that with a flick of a lever can be closed. Then when some thief wanders in and reveals himself, you can shut the gate to trap him in. Once trapped it is just a matter of sending troops out to deal with him.
****
The Caravan will eventually leave, assuming they can (if they can't path out of your Fortress to the edge of the map then they can't leave).
****
Insane Dwarves don't calm down - Your Berserk Dwarf must have met one of two ends: He died, or he killed off your entire fort. I assume the former happened in your case.
Note as well that for Berserk Dwarves, there is only one thing your Captain of the Guard could do to such a Dwarf - execute the kill order you gave on said Dwarf. May also be executed by any Military Dwarf as well.