WARNING: A bit rambley (and full of parentheticals).
In reading the god awful boring book I have for Cultural Anthropology I ran across a sidebar saying:
Was the invention of Agriculture a Terrible Mistake?
And lists a few costs associated with farming: social inequality, disease, despotism, destruction of the environment from soil exhaustion and chemical poisoning, water pollution, damn and river diversions, to air pollution from tractors.
Now I'm not suggesting that all of these things be implemented, but currently everyone agrees that farming is too easy (I typically have two 6x8 farms (one inside, one outside) set to one crop for 1 season each: one spring, one fall, and STILL end up with more food than I can use) and that there is no downside to farming (other than finding a place to store it) and it only takes 1 or 2 dwarves to keep 100 to 200 feed and drunk. Historically it's taken more than that (5-10%, I think are the numbers, possible exception being Ireland when they grew potatoes pre-famine due to it taking less land to grow the same amount of food, the famine would have been a passing thing if the British hadn't exported 90% of the food and spent more than 50,000 pounds to feel 8 million people).
What farming could use are some specific events that would mean you need more farms, more farmers, crop rotation, and actually fertilizing (hopefully something besides wood to do it with) to achieve the same production.
Ideas can be taken from the list above (disease, for example, could kill all the plants in a farm, soil depletion means that a seed only has a 50-50 chance of sprouting and making a stack of 1, fertilizing too much could cause soil poisoning, whereby nothing will grow until you flood water through the room, which then would be "polluted water" but there's nothing to indicate that right now--[SALTY] might work) or others can be suggested. We're not trying to nerf farming, we're just trying to make it not be a 200+% returns industry (even a novice farmer gets back twice what he spent--idea: reduce the number of seeds per plant to 1, so that a stack of 1 gives 1 seed, and a stack of 5 gives 5, but make it so withered plants also return a seed, because that's what happens: it "went to seed" and isn't edible, but can be used to grow another).
Another idea: farmers must routinely return to the farm to tend to the plants, can't just plant and ignore. This causes a need for more farmers, as each plant needs to be visited several times throughout its life (instead of two: planting and harvesting) and that that care is what determines yield, not the skill of the planter (though it could influence). An idiot planting seeds will get many plants, and lots of fruit, but if he doesn't keep the weeks off and the bugs out he won't end up with much that's edible. See: Sim Farm (it was simple, and actually fun in a way to balance the needs of crops, though sometimes it got a little out of hand when you didn't know WTF you were doing and doing what the icons told you wasn't enough--I'm suggesting that it be automated: plant, remove weeds, water, remove pests, harvest; different plants could have different needs, such as plump helmets (being mushrooms) are plant, water, water, remove weeds, water water, harvest or cave wheat would be plant, water, remove weeds, water, remove pests, fertilize (would effect entire plot), harvest. Each crop would have a list of tasks to be called at certain maturity levels (at day 5: water me) and that if the task doesn't get done in a certain amount of time then the plant's "quality level" goes down and produces a smaller stack).