Hiding half the tile neighboring hidden areas the same way the multi-level shadow is done currently wouldn't be too bad.
hiding the lower 1/4 of the above tile would absolutely be enough.
Hiding part of the tile is an implementation method that I hadn't really considered. It could certainly work.
However, I have an actual demo now:
This is using Meph's tileset plus some additional overrides:
[OVERRIDE:202:T:StoneWallSmoothLRU:3:_MDF_overrides_6:238]
[OVERRIDE:202:T:StoneWallSmoothLRU:6:_MDF_overrides_6:238]
[OVERRIDE:202:T:StoneWallSmoothLRU:7:_MDF_overrides_6:238]
[OVERRIDE:202:T:StoneWallSmoothLRU::_MDF_overrides_6:235]
[OVERRIDE:203:T:StoneWallSmoothLRD:224:_MDF_overrides_6:238]
[OVERRIDE:203:T:StoneWallSmoothLRD:192:_MDF_overrides_6:238]
[OVERRIDE:203:T:StoneWallSmoothLRD:96:_MDF_overrides_6:238]
[OVERRIDE:203:T:StoneWallSmoothLRD::_MDF_overrides_6:236]
[OVERRIDE:185:T:StoneWallSmoothLUD:41:_MDF_overrides_6:219]
[OVERRIDE:185:T:StoneWallSmoothLUD::_MDF_overrides_6:218]
[OVERRIDE:204:T:StoneWallSmoothRUD::_MDF_overrides_6:237]
[OVERRIDE:206:T:StoneWallSmoothLRUD:128:_MDF_overrides_6:221]
[OVERRIDE:206:T:StoneWallSmoothLRUD:4:_MDF_overrides_6:220]
[OVERRIDE:206:T:StoneWallSmoothLRUD::_MDF_overrides_6:225]
I'm no artist, so I didn't try to make completely new tiles to use here; I just reused the existing ones.
The mask is currently 8 bits wide, with one bit per neighbor. The bit is a 1 if the tile is walkable, and a 0 otherwise. This is not quite the right test (I'm sure you'll be able to think of a few ways it will go wrong), but it was easy to implement; it's easy enough to write a better test now that we have something to play with. Also, one bit per neighbor might not be enough (it can't distinguish between a neighbor which is smooth and a neighbor which is raw stone, for one thing).
The code is here:
https://github.com/mifki/df-twbt/pull/82