I use a similar setup.
4 "builders" -- mining, engraving, mechanics, architecture
1 craftsdorf -- masonry, carpentry, glass making, bone carving, wood carving, weaving, clothier, leatherworking, stone crafting, crossbow making
1 "laborer" -- farming, butchery, tanning, furnace operating, plant gathering, milling, cooking, brewing, plant processing
1 "uber-dorf -- weaponsmithing, armorsmithing, blacksmithing, metalcrafting, appraising
The metalworker is the leader, and he learns the various clerk duties on the fly. As he's a noble, he should be protected from the hammerer, should the hammerer get loose. He's also the most valuable dorf, so any XP gains in metalworking will stick around for a very long time and be concentrated into a single dorf.
Later on when there are migrants, I stick them into the craftsdorf and laborer castes, usually two laborers per craftsdorf. The original four builders keep the monopoly on mining, mechanics, and engraving so they become highly skilled. And only the dungeon master is allowed to also practice the metalworking skills, again focusing XP into as few dorfs as possible for those valuable skills. The original metalworker stops all hauling tasks and turns basically into another dungeon master once there are sufficient other dorfs that he can specialize more.