So many different responders within 24 hours of posting... I guess having a reboot really was a good idea to get more people into the discussion, because the last one became pretty inbred with mostly the same posters over and over again.
OK, first off, I tacked on the feces/urine argument into the second post.
Soil quality need only be tracked for tiles that actually have soil, which won't be many (basically the surface, subsurface soil layers, and the caverns--farming on stone shouldn't be possible without moving soil from elsewhere. A 4x4 embark with 3 caverns, surface and say 4 subsurface soil layers would have (3+1+4)x4x4x48x48 = 300,000 tiles. If you want 4 bytes of data for each tile (water, nitrogen, phosporous and potassium) that's only an extra megabyte of RAM.
Maybe soil acidity could be introduced too? It could be changed by the application of lime.
I haven't gotten to this part of the suggestion yet, since it's part of the "nuts and bolts" I wanted to avoid at first, but one of the things we talked about is why you can't just have binary flags for water and NPK. There needs to be changes over time, and building up soil (and yes, acidity was included, as was biomass, and there were talks about soil drainage and other more esoteric soil qualities) so that crop rotations can make some logical sense. This means having more than just 2 values for every soil, and as a compromise, I was talking about 8 bits per soil nutrient, and six variables, for a total of 24 bits per individually tracked piece of soil. Making larger "chunks" of soil that share their variables would allow for larger values that can have more nuanced changes in fertility over the course of game years without having to track half a megabyte of data just for soil fertility data. After all, tracking an average of 10 tiles of farm with a single value that is two or four times longer in terms of bits will still have a net savings of two to five times the data.
Subterranean ecosystems could semi-plausibly be powered by:
1) decaying organic debris, whether brought in by your dwarves or washed in through the underground river system--this would make caves with no water pretty sterile, which is quite realistic.
2) Chemotrophs or lithotrophs--organisms that build organic matter out of inorganic (see http://en.wikipedia.org/wiki/Chemotroph and http://en.wikipedia.org/wiki/Lithotroph). These might require special conditions such as the presence of particular rocks or hot vents (maybe proximity to magma?)
The problem with decaying organic debris is that there isn't nearly enough of it to explain cavern ecosystems. Chemosynthesis was actually a part of a more advanced "magma farming" system I wanted to explore, but I think becomes a little too complicated, especially since magma and water don't play together too well right now, and I'm already asking for enough major system overhauls.
For now, I think it would be best to have some sort of more invented sci-fi explanation for cavern systems, and having some sort of fungus creature that can just tunnel straight through solid rock and leave behind muddy floors that can spawn even more decomposers picking up its waste would be a good starting point, but there needs to be a way to inject more energy into the system without perpetually increasing the amount of mass, and for that, we need a "magic energy source" in the Arthur C. Clarke sense that "any scientific principle you can't understand is functionally indistinguishable from magic" (paraphrased/rephrased to be made more applicable to the topic at hand). It needs to have a measurable, predictable function to base farming off of it, which means it would behave in a measurable, predictable manner similar to science, but it basically would generate its energy from seemingly nowhere, or at least, generate it in a way that is functionally infinitely renewable without need for destruction of matter in the process to really be able to harness it for our purposes.
Converting infinite magma sources into energy can accomplish this, but not all caverns really hook up to magma vents of their own. Something magical about the caverns themselves (and having "cavern climates" from different flavors or densities of that magic) would be the best means of explaining how all this works.
While i really like the idea of selectively breeding crops, i think it should be a bit more complex than that. Instead of having a single 'better' type of crop, should be different traits you can give it, such as how hardy it is, how fast it grows, and so on. This could also tie into the raws of the plants themselves, having certian traits for growth rate, nutritional needs, ect.
My other thought on this is while procedural advancement is fine for the AI civilizations, when it's a player controlled civ, i think you should have a bit more control, such as ordering your farmers to breed, say.. maybe a faster growing strain of pig tails for example. Then perhaps randomize it, so you may end up with a strain that's just faster growing, or maybe faster growing but a bit more adapted to your exact ecosystem, making it harder to export. Or perhaps you get something else, as accidental discoveries happen occasionally. But, as the player, i feel you should have some say in the direction of reasearch , but perhaps that's an argument for another thread.
Sorry, this is also a problem of my trying to be too abstract when talking about the subject. You are correct in your assessments, and these are things I also want to try to fit in.
The balancing mechanism I want to put into "better crops" is that, generally, they are just more susceptable to pests and more generally delicate. Modern agriculture makes up for this with pesticides, but dwarves obviously have this in very limited, if any, supply. (Unless "dump magma on your fields, that solves everything" counts as a pesticide.)
To make a real-life analogue, the true miracle story is corn. Corn started as a grass, not much different from wheat, but which has been so utterly unnaturally selected as to be almost unrecognizable when compared to its ancient predecessors.
Native Americans relied upon their corn, but there was almost no real variety in corn, and as such, it was very susceptable to crop failures. This is, in fact, believed to be one of the major reasons why the North American Native Americans, which did have an ancient civilization (the Mound-Builders), lost that civilization - a few years of successive crop failures made the people overthrew their leaders, and just went back to more "safe" methods of society, like hunting buffalo or smaller-scale subsistance farming in the areas more capable of sustaining such agriculture.
Also consider the navel orange. There is only one navel orange tree in the whole world because it cannot produce seeds, and was just a freak mutation. It has existed for decades, and more trees are produced by "cloning" it by cutting off a branch, and letting it grow into a new tree. Still, it's just one organism, with just one string of DNA, with no means of evolving defenses while organisms that might prey on it, such as disease, can evolve every year to better exploit the weaknesses of the navel orange tree. They are all the same tree, so if one becomes vulnerable to a disease, they are all helpless against that disease.
fgsfds! Three new posts while typing this one... well, might as well post it before I get caught in a neverending chain of trying to respond...