In-character. In this particular case, dependent on the ways you suppose dwarves should react. (which kind of substracts from the sandbox factor).
I only at one point made a reference to one exact personality-trait and this one has been "loyalty".
Substracting from the sandbox factor? It adds to the sandbox factor - and even more, when there are generated civilizations and races.
With this, society grows more complex. As long as reasons and rules for murder and execution aren't hardcoded there is more sandbox than anyone could imagine. We just need the possibility.
It wouldn't be following your idea either.
Why?
Where are murders? There are accidents, but they really don't have anything to do with grudges. They've got no reason in society but in physics. Those aren't murders, or was I missing something?
Oh, yes, I am talking about FortressMode, of course.
And the hammerer is not really doing execution. He is hammering at random people for stupid reasons. We currently do not have any difference between execution and a fight in self-defence and the current system cannot model this difference - not even with the ethics-tags.
You could rework my system, so the hammerer is the only one, who has the executioner job. And you could even remove the option on the player to order dwarfes to be killed.
For that matter you could add a "butcherable" tag to dwarves and thus get a homemade execution method.
I guess, then my dwarfes would eat meals made out of dwarfen meat. Wouldn't the butcher doing this commit a crime? And still dwarfes wouldn't be murdering each other.
Personally I like the current justice system better. It could use some improvement, but I feel that what you propose alters the very basis of it.
I don't want to alter or remove the current system, I want to add in a new and independent system on top of the old.