I've put some thought into such a system. I've been looking at creating a roguelike with advanced AI, on a level where monsters are capable of living their lives from birth to death, each one having to find food every day and discover how objects work through experimentation (or having someone else tell them). That kind of stuff.
Anyway, I realised early on that any world in which these monsters were allowed to run around within would have to be pretty robust to not either become cluttered with the accumulated crap produced by these monsters or end up a wasteland as they destroy everything. Thus the idea of a closed system wherein nothing could be created or destroyed, only rearranged.
The key here is 'pocket reality'. It needs to have consistent and logical rules that define how things work, but they don't need to mimic our reality's rules exactly (and you couldn't anyway, they're too complex). A simplified model of reality is called for. For example, I came up with the following system to act as a foundation for my pocket reality;
There is the ground. It is divided into tiles, and each tile has only two properties - what is upon the tile, and the energy of the tile. There is a basic 'entropy' mechanic that equalises energy across tiles, so a tile with high energy will transfer some energy every turn to surrounding tiles with lower energy.
Then we add plant life. Plants have one property - their mass. They take the energy from the ground tile beneath them and convert it into mass that they store. When they reach an arbitrary mass quota they convert some of their mass into seeds which they randomly spread to surrounding tiles. Yes, this means plants can never die unless eaten, but these are pocket reality rules!
Monsters can eat the plants and gain the mass the plant had. If a monster moves it transfers a small amount of it's mass into the ground, and if it runs out of mass it dies (so it has to keep eating to move, and it has to keep moving to eat!). When it dies, it adds all of it's mass into the ground, after rotting. The mass now transferred to the ground is the energy the plants eat to gain mass! Ecosystem closed!
You can let the monsters make items out of the plants, in the form of wood. When the wood rots it's added back to the ground. It's a pretty basic system anyway, but it's a start.
You still face the problem of a mischievious player killing either all plants or all monsters. The easiest solution is to make the world big enough that you can't possibly kill things faster than the areas you've just slaughtered your way through are repopulated. That won't work when your model becomes more advanced than ground > plants > monsters > ground though, as all the ecosystem talk identifies. First steps first, I guess?