You'd need IC. JUST TCon, no matter with how nice metals would lose so hard. For one thing, they'd get raped on sight by Ars, and they'd be unable to do ANYTHING to Thaumaturges. If you're going to do a tech class, it has to have something damn nice to compete with the custom opness that Ars gets and wands of fuckyou that TC4 has. Ars is stronger as far as raw magic goes, but TC4 has delicious automation and mass mining.
TC wands don't do knockback, do they? Yea, they can be brutal, but with pretty poor armor choices and the inability to keep melee away from them thaumaturges become glass cannons. Quantium + Nano saber is pretty much unbeatable. True TC gets a degree of automation, but they are the only class that does and suffers from other setbacks like inability to share research, meaning all the ore in the world won't help you if nobody is online at the time to give you what you need.
Ars Magica is tough to overcome, but like I said there is a config option to lock people into a single skill tree, limiting what spell options they would have. There is also an option to increase or decrease the base damage of all AM spells, so if they are too good it is easy to tone them down.
I feel this would work best if your class was assigned randomly between what ever teams had the lowest numbers, keeping the size even.
How useful can vanilla invisibility potion (and other vanilla potions) + enchanted diamond sword be in such themed PVP setting?
Depends on what they are against. Even while invisible you can see their sword and attack at that. Some people would have access to very unique ways to kill you without having to even see you (Seriously, Vampiric poppets are terrifying), might be able to just teleport away. Might be able to just tank it out even.