As a play on this, and using the BUILDINGDESTROYER tag:
Expand the tag (more than 2 numbers), with each creature receiving a number ranking representing the raw, natural brute strength (and or other abilities) to destroy a construction. (1)
Next, give weapons a BUILDINGDESTROYER tag (if not already, I'm not a RAW expert to say the least). The combined rating of both creature and weapon (e.g. goblin with axe, troll with maul, dragon with no weapon) represents its destructive rating.
Next, give each construction a BUILDING rating. Give each construction material (wood, stone, metals, adamantine as the highest) a BUILDING rating. Combined represents the strength.
Next, each constructed building gains hit point equivalent equal to its base construction, material used and quality of the work.
The equation starts that creature + weapon >= building+material for that attack to even harm the building. E.g. a goblin + axe (total say 3) won't harm a felsite door (5) no matter how long he batters it. With a maul (1+4=5) he now could. He now assaults the door, and over time causes damage reducing hp to nil - destroyed. Similarly, a bare handed troll (5) can do the same. And so on.
(1) as a variation, specific creatures could be granted bonuses against specific constructions or materials e.g. SoF gains +10 to wood materials.