I'm wondering how many people embark and immediately dig down to the first cavern and setup a fort there instead of closer to the surface?
This has actually become my default modus operandi of late. Caverns give wood, land suitable for grazing and farming, and there's something infinitely more dwarfy about a fortress centered around underground caverns than one built close to the surface.
What kind of equipment and dwarf setups should I embark with? I'm thinking of having every dwarf have military skills since they can learn anything else useful later on, but I don't know how much armour is required to give adequate protection against cavern critters.
You really don't need to do anything different embark-wise. Starting with a bunch of military skills/equipment is not really that necessary for settling the first cavern layer (usually around z-levels 25-35). You are unlikely to run into anything tougher than a couple of troglodytes or bugbats, especially if you dig down immediately. What you will want to do is pick a good spot that you can wall off quickly and easily. A lot of caverns have wide open spaces surrounded by little mazy passages. These are perfect. Build a statue in the middle of the wide open space and designate it as a sculpture garden. Your livestock will congregate in this area and graze on the cave moss without having to be pastured. Then just slap up a few well-placed walls in the mazy areas to isolate yourself from the cavern proper. As your fort matures, you can always deconstruct a few of these walls and expand further into the caverns. As another poster mentioned, be mindful of the multiple z-levels of the cavern - you will need to make sure your walls go all the way up to fully protect yourself from flyers.
And do caravans arrive in the cavern, or do I have to make a gigantic ramp for them from the surface down to the cavern?
This is really the one big hitch about cavern-based forts. Caravans will not spawn in the caverns under normal circumstances. What I usually do is build an airlocked trade depot on the surface with a nearby chimney that leads down to the cavern. You still have to haul trade goods up to the depot (or build a burrow for a live-in craftsdwarf near the surface to make goods) but then you can designate purchased goods for dumping down the chimney into the cavern. You can also dump ores down the chimney (since iron ore and fuel generally only occur near the surface).
In 34.11 you will need to take precautions with the chimney, as falling items will now do damage to dwarfs who might be standing at the bottom (in past versions they did not). Also with the new minecart revisions, since ore is so much heavier, the chimney dump is probably less efficient than a good cart setup.