I haven't played in a little while so some of them may not longer be applicable, but one thing I found about Dwarf Fortress is that it has an extremely steep learning curve, but once you learn it then your odds of losing drops to near zero (short of inevitable lag death, or bad luck). Once you learn to feed your dwarves and keep them reliably happy the only threat is invaders which are easy to deal with using traps, walls, archers, etc.
So, in the past, I would install a mod that added a huge amount of enemy races, some of them extremely strong (I forget the name now) and upped the chance of being attacked.
Then, I put the following rules into place:
1) I could never completely wall off my base, there always had to be an opening, I could use floodgates if I wanted to, and doors of course, but never walls or drawbridges.
2) I could not use traps. Instead 100% of my defense must be using militia.
3) I could not use "Danger rooms" (don't know if those are still common, but they were a way to get your dwarves extremely highly trained at combat quickly).
4) I could not use magma for smelting.
Those were the big ones, basically the idea was to make combat, or more specifically base defending, much more challenging.
Another unrelated challenge I tried once, which was great fun, was where you take your initial seven dwarves, dig down underground and carry in all your initial belongings (the stuff from the wagon), then you wall off the way back out and play the rest of the game never returning to the surface. While obviously you can't buy more stuff from traders, the big issue is you cannot get more dwarves (they'll still arrive, you just let them die somewhere). When each of your seven dwarves are incredibly valuable many decisions become more complicated and dangerous, for example near the only close water-source lived a cave crocodile. I had to kill it, it was glorious, even made armor out of it's hide to celebrate.
Eventually I lost it all to a forgotten beast which poisoned us all and ended things but it was a good run.
One final one, popular in the old days, was surviving in a glacier environment. In this case the challenge was water really, which took some doing (you'd have to dig out a pit, drop it down, so the ice would melt, then use that). Now though, with underground lakes, this one isn't as big of an issue.
And, of course, there is the old standby if setting up camp over an aquifer (the more layers the harder it is). This is the best one, perhaps, because after you master it you have a lot more choices of where to set up a base as well as an endless source of water if you need it.
Also, not exactly a challenge, but a while back I stopped using the various programs to manage dwarf jobs (not sure what the current one is, at the time it was Therapist). I ended up liking it much more this way, food for thought.