I think that part of it is also that Toady wants to maintain the cliff-face of new challenge as a cliff-face, not as a greased sheet of plate glass with sharpened obsidian fragments.
Imagine if a new player encountered a challenge that was engineered to challenge a veteran player. Something that would pose the same level of difficulty to an army of multi-legendary champions that the first goblin siege does to new players. Their fort would be wiped, they'd get frustrated, they'd leave. They would be struggling enough with keeping their dwarves non-tantruming, nevermind keeping a spare army of fully-equipped champions lying around.
Dwarf fortress doesn't have levels, or missions, or any of the usual ways around this problem. It doesn't have them because it's trying to be a world simulator, not an RTS. That's part of the reason that I like it.
Now, that means that to get the difficulty scaling, you need to scale the whole world up to be more difficult. That's where the modding, and the megaprojects, and the challenges come in, and I don't see the problem with getting difficulty that way. Goblins aren't tough enough? Use Orcs, or Doombringer goblins. Comfortable getting everyone fed? Double the growing duration of everything. It lets people (and through the community, everyone) fine-tune the difficulty to where they want it, and that's great.
Now, maybe some of the more common mods could be implemented as world-gen options (food durations, enemies available, and so on). But I think that the current balance is fine.