AAAAHHHH SO MANY WORDS!
Every third paragraph my brain starts going off tangentially to what you're saying, and I sit there and cogitate on it for a few minutes before coming back to find you went in a similar direction I did, or a different direction I wanted to debate you on, or jumped the rails entirely into a new thought that sends me off on another tangent. Or, in short, you make me think too much
I'm not even going to touch the whole spheres/magic energy source/make the caverns make sense thing. Or rather, I'll do it at some later date in some other thread. I'll be in and out of this thread as individual bits provoke my brain cells.
Notably, the first tangent I went off on started when you highlighted how the problem is the (relative) lack of scarcity in DF. My thoughts really started rolling after you mentioned that each player wants DF to be different things. You're right- this is a game, and one of the objects of a game is to have a player make conscious decisions about how to most effectively manage their limited resources to achieve their goals. "A series of interesting choices," as I've heard it eloquently phrased. The thing is, in DF most of the goals beyond "survival" are self-imposed by the player, which means after the early stages of the fortress every player is going to be making different choices depending on what they want to do. At this point I stopped thinking about scarcity and optimal actions in regards to farming and applied it to the way we play DF as a whole.
After chewing it over a bit, I ended up breaking the way we play DF into two groups- World Shapers, and Creature Controllers. Creature Controllers are the players who like manipulating and learning about the inhabitants of the fantasy world that has been generated, both as individuals and as larger social units like family and fortresses; learning about their social lives, maximizing their happiness, optimizing their productivity, interacting with other races/civs/sites, playing with the economy and so forth. What the creatures do is the most interesting to them- the fortress itself is just a place to collect these interesting little dudes. World Shapers, on the other hand, manipulate and investigate the world around them; building megaconstructions, doing large mining projects, playing with eccentricities of physics and generally forcing the environment to bow to their needs. The creatures within the world are interesting in the sense they provide a way to measure and interact with the environment, but if a bunch of miners die, what of it? Obviously, many players are between these two extremes, but roll with me for the sake of argument.
Thinking all of these things was a bit of a paradigm shift for me, since it broke up what I had once considered a unique whole into discrete pieces. It also illustrated that everybody does a little bit of each part as part of surviving and making a sustainable fortress, and that tackling all of these hurdles at once creates the learning cliff we all know and love. What happens is that after everybody struggles to do the necessary bits of the other playstyle, they tend to ignore them in favor of spending their most precious resource (time) on things they have the most fun doing. So after feeling enlightened, I evaluated the suggestion itself in these new terms. Improved Farming enhances the World Shaping aspect of the game, since it necessarily involves learning more about the earth itself, life cycles, weather, water manipulation and so forth as you make further and further improvements.
Now what really excited me is the fact that, by your methods, you've made it possible to have very detailed refinements to this aspect of the game, complete with added challenges and a veritable font of new projects, WITHOUT forcing people who play the game differently to figure it out. If you enjoy this part of the game you'll have a blast assuming each successive challenge, and if you don't you can accept having suboptimal farm output provided it meets the minimum needs of the fortress, and you can supplement your food production with trade in order to spend more time focusing on the little dudes themselves.
At this point, I realized what you had actually done was make a way by which being good at one playstyle made you automatically better at other. You are better at the military aspect of the game because your bountiful self-sustaining farms make you able to ignore the depredations of sieges. You can succeed economically and socially because your awesome fields are producing nice things for your dwarves, and you're growing more than other sites because your unique embark makes you more suited for, say, redroot dye than they are.
And of course, THAT thought made me start pondering ways to get the same kind of progressive difficulty curve, since this ties right back into DF being everything for everyone. I could probably ramble on at this point (as if I haven't already, and I've still got nits to pick with what I've said), so I'll just stop, chew it over, and if it merits further discussion I'll go make my own thread.
And that was just tangent #1! I haven't even started thinking about FARMING YET!
I'll pop in later and make some more on-topic posts- I'll just use this space to thank you for giving me something to ponder at work tomorrow.