From what I understand, the forced of Hell don't follow the normal rules. Normally, a creature is defined in a raw file, including their population. This population can be 10,000 for near infinite, or it can be 2-3 in the case of cave dragons, but there's still a limit out there, and you can kill every single creature.
Demons, on the other hand, have no direct file, and are more like titans and forgotten beasts, in that they're mostly randomly generated, and have no population limit. While there's a limit to how many will be on the map at one time, there's no limit to how many will come at you over time. This can be good or bad, depending on your view. On the one hand, there's always a foe for you to face, on the other hand, they fly and swim in magma and water, and will kill your FPS harder than all the cats in the world if allowed to gather. Thus, either by violence or FPS, hell will destroy you.
The only way to combat hell, is to wall it off. You can build walls at the edges of the map, but only underground. On the surface it's only within 5 squares, but underground it's up to the edge. However, on a 3x3 embark there should be 188 edge tiles. Assuming hell to be 3-4 Z deep, counting the small hills, that's about 650+ tiles of wall that you need to build, plus small amounts of scaffolding to reach the above-ground-level walls, all while being assaulted by a constant stream of demons and the barrage of "Urist McMason cancels build: Interrupted by Demon".