Yeah offense is the best defense, concentrate on making enough armor and weapons for 20 dwarfs, the first ambushes often bring only one squad you will outnumber them 2:1, even recruits will pickup kills. The wounded mean training for your doctors, and the ones who recover can become decent fighters in a few years of training and fighting. Add more recruits to their squads and the drills will start doing something.
Add in a line of 3x3 cage traps as your last line of defense and you don't have to worry about anything until you reach a population of over 80 and the sieges begin. The first sieges will often be really weak too.
Wall of the caverns or if you want access make gates of raising drawbridges (building destroyers can't destroy a raised bridge). Make a burrow that includes all safe areas of your fort and make a civilian alert for it that will keep civilians from running outside in times of trouble.
Don't worry about high numbers of idlers, a 150 dwarf fort can be supplied by a few legendary brewers and growers easily. Just make sure you disable the important labors on any new migrants so that they don't ruin your food production or lower your armor/weapon quality accidentally.
My current fort was defended by only 10 melee and 5 ranged soldiers for most of the time, they were only in iron armor too, as my map has no flux.