As usual, I only know 3.5e... Sorry if PF is different.
Not to mention that it can't really be used as a Get Out of Encounter Free card, considering that you have to leave from the same place you entered eventually. If the enemy had anyone capable of knowing what you did, that's reason enough for the DM to have had them set up traps and prepared to ambush the party when they left.
Open up the rope trick hole, see the traps, negate them... leave harmlessly... all thanks to a single spell.
Negate them... how? Can't cast across the border, which means the traps are just as deadly as any other traps. The rogue has to partly leave the hole to attempt to disarm, becoming vulnerable (and visible).
Besides think about what you are all saying. In order to balance ONE spell you have to overwhelmingly put the entire dungeons resources against him, have creatures with full encyclopedic magical knowledge (Which only a select few have), masterful trap making skills capable of laying magical traps in meer hours (Which none have... Where traps come from are often a DM fiat), all for one wizard that could have half-assed his training. Even ancient dragons couldn't pull off what you are referring to.
If anything you are proving just how overpowered it is by saying the exact requirements to stop a single spell.
Remember Rope Trick doesn't "Just end" they can look out the hole... along other things ("Ohh look they are making traps, Fireball... Sorry still in perfect safety!")... honestly I am not even trying to break the system, this is just a single spell. I haven't even got into combinations or some of the other little tricks wizards have.
Fireball from a rope trick is very iffy. I know fireballs and other evocations bypass spell resistance, but it's still casting a spell across the border. Shouldn't work in my opinion.
And it's a level 2 spell, you don't need "full encyclopedic magical knowledge". Some shmuck with spellcraft of 2 has a 25% chance to identify the spell precisely. But anyone can quite reasonably say "Hey, they climbed that rope into an invisible hole. Maybe it's still there".
Keep in mind "The rope can be climbed by only one person at a time." The climb DC for a rope trick rope is specifically 5, which means there's very little risk... But, don't forget armor check penalty or strength modifier to Climb! Paladin and spindly spellcasters might want some cross-class ranks.
Let's say our cowardly wizard casts rope trick and rolls a 10. -2 STR, but the DC's only 5. He can now move... 1/4 of his speed up the rope. For a normal creature, 7 feet up. Well within vertical reach of inquisitive enemies who just watched the wizard climb up a rope into nothing. They could stand underneath and *jump* to see what's inside, or just blindfight into it.
Assuming initiative is such that the whole party moves before the enemy get a chance (probably by holding turns) they might be able to move or sprint to and up the rope. As long as they can move up to the rope and make the climb check, within their movement allowance. If they can hit a 10 instead of a 5, they can even climb at half instead of a quarter. 20-speed creatures might need to do that, even if the rope is the minimum 5ft.
It doesn't say what type of action it is to pull the rope up, which I think makes it a standard action by default... Either way, if someone didn't have to act or sprint, they might be able to pull off an amazingly coordinated escape and hide the evidence...
Into the 5-30ft high hole, and I'd love to see a level 3 party that made it up a 30ft rope like that without courting disaster. No, practically speaking it's going to be within the 10ft vertical reach of medium creatures.
Which I guess is where things get really weird and undefined. The space can hold 8 creatures (including the rope), but it's not clear how *big* it is. I guess arguably it's unbounded, so the party members can simply back away from the hole and be safe... but then enemies may enter if it's not at capacity. If they can vertical-leap high enough to reach the hole, I'd say they can grab the edge and climb up (harder than with a rope, but still). Or fish the rope out with a weapon... Which interestingly frees up a space within the... space.
The hole is impressively stealthy unless *you do it in front of people*, in which case it's very vulnerable to intruders. Which matches its purpose as a safe place to sleep, not escape. And once detected, it's quite vulnerable to assault or ambush. The occupants do have the advantage of being invisible, but they're also cornered and can't cast across (nobody can, but still). And climbing inside is not even trivial, particularly in a combat situation.
Basically I'll put it this way...
The game doesn't attempt to negate Fighters, Barbarians, Rogues, Clerics, Druids, Bards, Rangers, or monks (Rogues are negated because... well 3.5 doesn't do rogues well)... The only characters 3.5 guns for, where at least half the monsters in existence are geared towards dealing with... are Wizards... And even then Wizards STILL can easily anti-negate... Any Wizard worth his salt ignores spell resistance and can defeat spell immunity.
Even Order of the Stick makes jokes about it.
---
All in all my experience with Wizards is that players who use them, and know them well enough, are intentionally trying not to break the game. All the while knowing exactly how to break the game, unless the DM intentionally makes things up to prevent that ("Ohh the villain is a super wizard, or the villain has a magic castle that blocks you")
This much is pretty true! Rope trick is not a great example though.