Also, are walls smooth? If so, then they shouldn't be able to scale walls.
If tied in with some of the alternate suggestions for material-handling, this could mean something...
Even right now, a wall built of rough stone would have many hand-holds. A wall built of stone blocks would (potentially) be less easy to climb, perhaps with a brick wall somewhere in-between (giving the grouting assumed to be used). (At least by non-'gecko-ish' creatures that can climb anything, and possibly even under overhangs as well.)
Ditto a log wall c.f. one made with planks (a.k.a. wood blocks). Metal walls might be the hardest to climb (save for those with suckers, or magnetic capabilities). Ah, and glass. Just suckers (and geckos and their like) for climbing those.
Concentrating
just on stone, the idea (superseded by the latest mining changes, so maybe not how it will ever go) by some of us at one time is that skill in mining lets one produce varying amount of dust, rubble and different sized stones or quarry-quality blocks from mining (also according to what the player designates as wanted from that tile, so quality can be sacrificed for speed). A rubble wall would not only be technically easy to clamber over (or a rubble-partition easy to dismantle, when blocking a tunnel), but would let liquids through, with large-block walls (perhaps with rubble innards) being more resistant to every form of attempt to get through (including siege-engine projectile impacts) and again also harder to climb.
You could not so easily lay down a solid floor on top of a rubble barrier, however. Or it would need to be a careful laying of paving slabs (obtained from quarrying large-scale materials) rather than merely a layering of rock chips that would sufficiently 'surface' solid ground to create paths with reduced impedance of movement by vegetation (if regularly used, at least, kind of like the unsurface roads we have now).
Anyway, that's not all so applicable to climbing. Let's just say better quality walls are harder to scale (unless they're easier, for some particular sub-method of climbing), that smoothing does likewise, although
engraving may make it easier again. Some form of "Trajan's Column" being made by climbing engravers might well be a self-supporting effort.