On the discussion of toolmakers/armorers/nailmakers, it's worth noting that a toolmaker is going to be better at making weapons than an armorer, since they have a much higher turnery skill. A toolmaker's also going to be entirely better than a wheelwright at making wheels, and a combination of any smith + a boilermaker is likely going to be better at making the plate armors as well.
So while a unit may be useful for autoproduction, if you want to push for a high quality manually, some units are actually better at a task than their specialized counterparts. A toolmaker is currently better for anything requiring turnery than the other smiths because of its equipment and skills, except for the boilermaker for boilermaking. Jewelers of all things are also the most effective for glazing, since they can equip diamonds. Stonecutters and bricklayers use identical equipment, so the difference in masonry/stonecutting is going to come entirely from unit level, which isn't very much (and I don't know if stonecutting is even very relevant).
You can also change a building's specialization very painlessly for no loss of quality that I know of. So if you want to produce some weapons and armor to get just a few units equipped, you don't need to build an entirely new forge, you can just flip a tool factory or whatever back and forth (and the toolmaker will be as good or better without having to train a new unit). I also would not worry so much about building quality. Even with perfect materials and units, it may not even be possible to get some buildings up to a high quality level, and it doesn't matter much for the lower level buildings anyway (like the buildings that only give +5 fire, the difference between +2 and +5 isn't really going to matter). Many buildings give no bonuses or penalties, so quality only affects combat (though a high quality mine for instance will have a lower happiness penalty). But if you plan on upgrading that building, the initial quality will matter.
And a final point, the annoying attributes a unit has do matter a bit (more than the difference between most building qualities anyway), but it's not just he requirements shown on the units page. The unit skills shown are often different from the actions that the unit will actually be making, like the knight apparently requires charasma and wisdom (or whatever those are) but doesn't even have any relevant skills related to those attributes (which would just be strength and agility for combat).