Pathfinding is one of the biggest memory hogs in DF. To deal with this, Toady used an algorithm that assumes that there *is* a path between from where an entity is to where they want to go. This makes things go a little faster... unless there is no path at all, when it's way, way slower.
So there's effectively a list somewhere that lists all the tiles that can be reached from the current tile, updated each time you do something that could change this fact. (It's not actually a list, but suffice to say that when you look at two tiles you can tell if there's a path between them without actually finding the path). There is only one such list: not a seperate one for dwarves and pets.
So a pet-impassible door between a pet and its destination doesn't change the fact that the pet thinks it can get there. So it tries to find a path, finds one, and starts walking. Eventually it hits the door, and can't open it. So it stops moving, and checks "is there a path between here and where I want to go." Again, there's only one list so the computer answers "yes" and tells the pet to go through the door. It tries, and again can't open the door. This continues until the pet decides to go somewhere else, or until the door opens. A "pet-impassible" door is basically a door that requires you to turn a handle, instead of a door that swings when you push it. A pet can't turn a handle, but if someone else (say, a dwarf) comes through and uses the door, the pet can follow through on their heels.
It should also be noted that pets don't obey the "traffic" specifications, though. So if you really want to trap animals you can make a "short" route with a pet-impassible door and a "restricted" designation and a "long" route with a "high traffic" designation (and possibly also pet-impassible for good measure). This would mean the pets would all cluster around the "short" door, but the dwarves would all use the "long" route so nobody would ever open it. You still lose processing power to pets constantly re-calculating their route, but it usually does pretty well at keeping pets trapped. (Pits with no door at all also work, but then dwarves can't get in either.)