You're forgetting that items & creatures use memory too. Like the stones you just dug out, wandering animals, or even ambushers sneaking around out there with all their gear.
Forgetting? Not really. You're right, but it's not enough to cause the FPS death that everyone knows and fears.
http://www.bay12forums.com/smf/index.php?topic=104643.210 Proven in this thread, by the venerable NW_Kohaku.
There's something else that does it - and that "something else" could be, oddly enough, walls and empty space. Yes, having to choose which item of 5k to walk to slows it down - but it's intermittent. They only do that once, and then they have a path. FPS death implies that the framerate slowing is constant. I've seen that a forgotten beast with toxic blood causing all my dwarves to vomit everywhere can cause a five second delay every two seconds on my laptop. But for those two seconds in between - everything goes smoothly. Not FPS death. I always figured it wasn't the pathing algorithms that were doing it... somehow I never thought it could be simply rock.
Think about it - for all the tags that a single article of clothing has, you're sitting on a huge embark. I can't remember the exact numbers, and DF is closed, as I'm about to go to bed, so let's go low and pretend for a moment that your average embark is 100 tiles on each side. Now let's pretend that your embark extends downwards 100 z's - you got unlucky and the magma's close to the bottom. That's 1,000,000 slots, and as floors and walls are counted separately - 2,000,000. Every unmined stone needs tags and values - thinking minimum and streamlined (and there may be some I forgot): State (solid, liquid, etc), Material, heat, mined-out percentage, location, and everything that's splashed on it. Assuming that a stone with nothing mined out of it has no tag for that, that's still four values per stone minimum. Seven, if you're counting location as X-Y-Z. This bumps it up to seven to thirteen million values sitting around.
Now of course this is a ridiculously inefficient way of doing things, and Toady, as a programmer, would've realized that from the beginning. Thus, if areas of the map yet to be revealed are simply tagged with a single character: S/L/G (solid/liquid/gas (aka air)) along with, of course, separate values for trees and so on so that you can simulate the cavern growth, you lower it to close to only 1,000,000 characters. You could even use binary: 00/01/10. Nothing's smaller than that. That's far, far faster than using separate tags for everything, and takes up a lot less space. However, if all that expands as far as memory goes when you expose it... it could easily reach around 364,000,000 characters relative to binary. That would explain why everyone says "ohhhhhh, don't breach the caverns unless you want your FPS to go away!!!". There's a reason for it. And it's not just pathfinding AI. I never really thought that was a likely answer to it, anyway, to be honest.