Then you're just handwaving away the issue? Disappointing. At the end of the day, Rule 0 gets used because the DM can't handle players using core game rules as written. Why not simply ban the teleportation school of magic too? Or say that nobody's allowed to play a spellcaster? Doesn't sound like much fun to me, but every table's different. At our table, we're all 100% for playing the game within the written rules. The DM rolls his dice in the open, and if that means your character eats a critical when they're on low hp, you'd better have prepared your backup PC. By the same token, if there's a class feature that's part of your character's build, it's considered bad form to say you're banning it simply because it makes winning easier.
Because they are VERY useful and perfectly viable... until players try to cheap their way through it.
And Teleport does have some pretty firmly established counters such as "You teleported into a wall" that would be punishing if the DM used. Often just saying "For some reason, you do not know of, you cannot seem to teleport to that location as if it is cut off" instead of "It looks like they remodeled and your standing in a wall"
It is a damned if you do, damned if you don't.
Basically it is like the Scrying spells. If used standardly they are perfectly fine. Once you try to talk to the main villain's Ex-Girlfriend and then Scry and die the final boss... THEN your approaching the area where the DM has to go "No, the villain is in an anti-scry field.
---
That and... Who do you think you are? Some nobodies?
EVERYONE should know who you are if your teleporting everywhere and using Scry and die tactics. If a villain doesn't have a way to stop you from Scrying or teleporting right up to them, they are probably not going to do anything until they do.