Okay, I've worked this over in my head for a bit, and this is what I got. In case you need to be warned this far into this thread HFS spoilers to follow.
These ideas are kept fairly mythology neutral, where possible, but there are going to be some sub-suggestions that come under countermeasures that some of you may find objectionable. We already know, you don't have to tell us.
The smash/burn/kill/defile demons in the pits and the horrible nightmare evil spirit creatures are the same beasties. If you kill the demons, the nightmares stop. This allows the dwarves to fight the demons on their own terms, in the traditional DF manner.
In fact, once you open the pits and the legions start storming out, the nightmares and whatever get put on hold in favor of a more direct application of death and horror. The psychic torments were just a stopgap that the demons applied, because they couldn't attack the dwarves in the more visceral fashion that they prefer. (This one does assume that the demons are unable or unwilling to tunnel to the dwarves, but still aware that they are "out there".)
The nightmares are whatnot should be an inconvenience at most.They should take a couple of dwarves here and there, through death and madness, and increase the chances of a spiral, but they shouldn't be a fortress ending cataclysm in their own right. They should mostly be about fear and atmosphere, rather than any sort of mechanical challenge.
From a game-play standpoint, the battle with the physical manifestations of evil is the climax, and the haunting stage is just the build-up that adds anticipation and flavor to that moment. In that regard they should slowly grow more intense over time, to push that sense of building drama, and eventually encourage the players into a confrontation with the horror that lives next door.
The nightmares should probably start with the first square of admantium mined(or even exposed), and row faster the more squares you mine. I know this seems harsh to those of you who are used to mining isolated veins with complete impunity, but I think that this adds the sort of enjoyable tension and "one more square" mentality that digging too deeply and too greedily should be about. Again, the horrors shouldn't be as much of a threat as a full on glowing pit scenario.
An experienced HFS miner might even be able to guess how many demons are in the pit below, from the level of horrors he experiences. This might seem like a bug, but it lends itself to a nice "Ooh, this must be a big one."
Countermeasures:
You should never be able to eliminate the effects of the call of Pit'thulhu on your fortress, but there should be a number of ways to mitigate the effects.
Doctors, priests, philosophers and medics should all be able to train themselves in the art of recognizing the symptoms of demonic possession and oppression. People are going to come to them with these problems, and they need to be able to figure out what they are dealing with.
Since the demon will just start harassing a new victim, there's no point in burning them, but that doesn't mean that your dwarves know that. Mandating the burning of the possessed would be a flavorful thing for athourity figures to do. This might give dwarves a happy thought, because they feel that something is being done about the problem.
Likewise, even if amulets and priestly blessings don't do anything, they might give dwarves the courage to soldier on through these troubled times, without taking an axe to the guy next door, when he rearranges his furniture at 3 AM.
However, it would be cool if the various religions did have some means of invoking divine protection, or abjuring harmful spirits. They could provide you with physical objects, like blessed amulets shaped like the totem animal of your god, sacred unguents(like holy water, sacred perfumes and anointing oils), scrolls inscribed with quotes from prophets, and the finger bones of a famous archbishop, or they could give you something immaterial like a prayer to say before sleeping, and a verbal blessing of their own.
These could make the nightmares come less often, or at least keep you from turning your children into artifact mittens. Over time, the demon's hold on a dwarf grows stronger, and you need more warding to keep it out of his skull. Thankfully, your priests are getting plenty of practice, so they should be up to the task, unless one of them has an accident, or get's possessed himself.
Alternatively, the fortress might try to appease their diabolical neighbors with worship and offerings. These could stop the growth of the problem, or even cause a single demon to stop causing trouble, so long as the offerings continue. As these are the histories of geed and trouble, the demons should demand an ever increasing amount of tribute, lest they visit horrors upon you once again.
The demands could be made known through the ravings of the possessed, or the cult leaders might have some means of auguring the demons will, without being completely lost to madness and possession(though it should probably be an eventual side effect, if they keep on communing psychically with demons). Some might even pursue possession, for the power it would offer them over the other cultists.
More Fun for everyone!