This has been suggested before, but to the extent of my searches they are short, and don't dive into detail. So here's my thoughts:
Wooden constructs: These would be most vulnerable to my system, Any unit with an Axe can destroy a wooden construct. Units of different strengths would have different speeds of cutting through walls, doors, etc. Axes would also speed up the deconstruction of wooden walls.
Branching off, Units above size 4 can kick down wooden doors, while units above size 10 have a chance of splintering the wood causing (possibly) deadly splinters to fly into anyone nearby. A unit with an axe, or a catapult shot would also have a chance of splintering the door. what the doors is hinged to has an effect on the Splintering, loam, soil, and sand walls would always give way and never splinter the door. Doors built into stone walls may splinter, doors built into metal ore walls would have a greater chance, and doors built into adamant walls, and metal would ALWAYS splinter.
The same goes for other wooden constructs, excluding walls, where only a catapult may splinter them.
Stone Constructs: These would be a lot harder to destroy,a unit sizes 5-10 would need a hammer, or siege machine to destroy, units size 11+ can kick it down with enough force. Stone doors and walls would have an effect different from splinter, instead causing dust, pebbles or even chunks of rocks to fly everywhere, Dust would reduce the sight range of dwarves, causing ranged weapons to have a small chance of hitting someone. Pebbles would cause minor bruises and might cause tiny dents in armor. Small chunks may cause minor breaking of limbs of an unarmored person and dent armor. Large chunks may break or mangle limbs, kill, knock out, or suffocate armored or unarmored personnel, with armored more likely to suffocate, or be knocked out, and an unarmored person directly killed.
(large and small chunks may only be caused my heavy siege equipment, or megabeasts)
Metal Constructs: Largely indestructible, needing multiple siege shots to puncture or break a wall, and multiple walls being nearly indestructible. Metal doors could slowly be knocked down with maces and hammers, units size 30+ have a chance of knocking down a door. Metal based megabeasts would not be stopped by walls, instead either melding into the wall/door, or absorbing it. metal doors and walls have a very small chance when hit or destroyed to send metal chips flying out causing cuts, and have possibility of internal damage. in any other case causing metal spikes to appear where to wall was, with a possibility of causing cuts to unarmored personnel moving through the tile.
Adamantine is invulnerable to most attacks, only attacks with sufficient force, and proper material can damage the walls. Also while being the one of the lightest metals in the game, a megabeast, or creature may have enough strength to rip a construct from it's foundation and toss it.
VERMIN: Vermin would take a larger step from gnawing on barrels, and may chew on walls and doors. insect vermin like termites may slowly destroy wooden constructs, rain may also cause slow rot and decay of wooden constructs, if not properly protected.
Racial Differences: Different races would use different walls, with different heights.
Elves use walls of trees and thick foliage that quickly regrows and is incredibly thick and hard to cut, but however VERY weak to fire. may also fall apart or become weaker if the village druid is slain
Humans Use standard stone walls, nothing special, early settlements may use wooden palisades.
Dwarves Also use stone walls but due to mighty dwarvern stoneworking they are stronger than their human counterparts.
Kobolds and Animal peoplesmay use simple wooden fences and sloppy wooden palisades, very poor construction and may be knocked over by small creatures.
UNDEAD(if they ever rise as a Civ) would use decrepit stone walls, with iron reinforcement (if possible) to compensate for their warlike nature.
EDIT: ADAMANTINE REVISION
EDIT2: VERMIN+RACES
Whaddya think?