I'm midway through my longest-running and happiest fort ever (I've had two slightly stressed dwarves ever, and am hovering at 7/8th's of the fortress being -100k stress, mostly because some dwarves die and get replaced by normal stress dwarves), and have some new observations on what exactly is the cause of all this. (Do note that this fort has had basically nothing but down time and me doing moderate micromanagement at stress)
Summary:
Personality change needs fixing (less extreme change for less extreme incidents), give weather acclimation so aboveground forts aren't impossible, don't worry about un-meetable focus needs, and of course, corpse thought stacking.
Causes:
1. The surface. I get it, dwarves don't like vomiting (who does!) from the sun, rain sucks, and seeing standing trees is downright offensive.
2. Sentient corpses. Still. Corpse stockpiles are useless due to how bad they are for stress, and it's just another trap for newbies to fall in, since they're supposed to be the actual way of disposing of these kinds of things but are actually not at all how you should do it unless you want your fort to go insane. Potential solutions to this are all well elaborated. I'm partial to the "corpse thought stacking" solution (which would also make corpse stockpiles viable again) rather than another nerf to the power of corpse thoughts.
3. Personality change. The surface can generate enough bad thoughts on its own, as can sentient corpse. But the real killer is personality change. Sometimes it'll make a dwarf immune to stress, but just as often it'll make them in a constant state of internal rage. This might technically be close to net stress zero, but nobody cares if a dwarf is doing "just fine" or "super uber happy because I'm immune to stress." The problem is that for every dwarf who wasn't a problem before and is now definitely not a problem there's one that wasn't a problem and is now absolutely going to go insane unless you pamper them in a sealed room, and even then not eating prepared fly brain will drive them over the edge. Personality change absolutely needs to be fixed: a dwarf with an insufferable personality every now and then as a rare migrant is fine, but the occasional miasma incident and having to use the surface for anything is basically executing a dwarf or two every time you do it in a bigger fort. Minor incidents should have smaller ranges for their changed attributes (so a dwarf can't go from impervious to the effects of stress to a nervous wreck or vice versa), and major incidents (loss of loved ones, permanent major injury, all that life altering stuff) should be the only things that can ruin a dwarf.
Stopping stress:
1. Avoid the causes. Duh. Automate siege disposal and just abandon the surface. I have yet to fight a siege in this fort mostly due to lack of methods, and I've avoided the surface entirely. This is something that needs to go. The surface can't be something you have to abandon just to keep your dwarves happy. I'd love to see acclimation to the weather be something a dwarf can get (or already have based off personality and beliefs), much like they can become desensitized to death.
2. Little happy thoughts. Offtime, offtime, offtime! This is an intended mechanic working as intended (least, since socializing was fixed). Taverns, temples, libraries, and guildhalls are all great ways of reducing fortwide stress, long as everybody has time to use them.
3. Big happy thoughts. Personality change is busted. Use it to your advantage. Birthing children and gaining siblings are extremely useful in making dwarves less prone to stress. I wouldn't call it overpowered as most forts aren't long-term, and also since most of the personality changes don't affect stress a ton, but it's still a powerful tool in the arsenal of anybody looking for a happy fort. When a dwarf has 10 siblings all born in the fort, you can bet they've got at least 1 or 2 changes that make them less susceptible to stress somehow.
4. Daily happy thoughts. Stuff that happens without your intervention is what keeps the hardier dwarves going, even in a hellish fort. Drinking, sleeping in a good room, dining in a legendary dining hall, all that. Not really anything you can do about this one. If your fort's residents aren't getting these thoughts, you sort of deserve to have your fort unhappy.
Things that aren't a problem:
1. Focus. Unmet needs aren't an issue in my opinion. Being a bit unhappy because you haven't had family time is fine, even for dwarves who don't have family. That's something that would realistically be entirely reasonable. If your literal life goal is raising a family and you constantly yearn for love or whatever, then obviously yes, you're gonna be lonely if you don't have a family. A few of them (like wanting foods that they've never tried before) are a bit silly, but still pass as reasonable complaints most of the time. It can be handwaved as "hey this dwarf heard of a great dish from their mom/dad, and really wants to try it," even for dwarves born in the fort. And they aren't typically problems for overall stress. It's a reminder that you can't make everybody perfectly happy, that there's at least a little bit of autonomy in these dwarves and they're not perfectly content being your slaves as long as you give them their basic needs and a bunch of zones to chill in. From a gameplay perspective it's an issue, but from a simulation perspective I think it's interesting flavor.
2. Military dwarves. Military training is quite powerful for alleviating stress, simple as that. I have three soldiers go insane ever (as a player who likes military a lot), even through 44.10-12 being my newbie days, and those were from personality change reasons more than the "seeing a lot of bodies" reasons, since they were accustomed to death by the time their stress started increasing. Soldiers going insane from invader corpses isn't the issue, it's the civilians doing cleanup.
Much of the stress system is pretty tough on new players, as while the needs/focus stuff gives you a decent idea of "hey, I should make this for my dwarves" there's still a lot of gimmicks involved in not having your fort go insane. It's entirely possible to have a happy fort as it stands, which is where I can see players who say stress isn't that broken are coming from, except they're wrong, because alleviating stress issues currently involve ignoring a good portion of the game (abandoning surface, not using corpse stockpiles), making game-y choices rather than ones that would be reasonable (who knew non-stop military training would be good for dwarves who are unhappy from being worked to death? also, atom smashing is basically required), and just in general doing that gimmicky stuff to get by. My current fort is happy because I'm doing all of those gimmicks. My two stressed dwarves happened during, news flash, a time where everybody was seeing sentient corpses. I've otherwise been lucky to only have semi-fragile personalities from inevitable incidents over the long term, and most of those have died from other reasons over the years.
Thank you for coming to my DWARF talk.