Baton pass teams are generally incredibly obvious. So Drifblim would be good since no one uses it for baton passing. Keep in mind the moment someone sees some baton pass-starting shenanigans, if they know what they're doing you have at most 2 more turns to switch out or you're fucked. Might be even less if they had something out ready to stop you in the first place. Quite a big problem considering this Drifblim build involves using so many freaking things before using baton pass.
Baton passing tends to be looked on as pretty nooby for this reason.
Off the top of my head, you need to watch for:
1. Taunt obviously. Goes through substitute as well.
2. Anything with priority. Pranksters especially.
3. Perish song. It gets transferred with baton pass and is pretty hilarious.
4. Enemy set up sweeper. What's the point of your baton passer if Gyrados is now behind a substitute and applied dragon dance twice while you pranced about.
5. Roar, dragon tail, whirlwind, etc.
6. Pursuit would obliterate Drifblim specifically. No one really uses this any more though.
7. Spore user. Even hypnosis or anything that causes sleep will cause trouble.
8. Something slower than the baton passer can OHKO your baton pass switch the moment you do it.
9. Haze users. Getting rather common.
10. Crits. All the evasion and defensive stacking in the world won't save you from crits.
11. Unaware gives no fucks about your baton pass team.
12. Trick and switcheroo users can royally fuck you up. They can either choice the baton passer or choice the baton recipient. Either way the entire plan goes to hell.
13. Prepare to cry when trick room appears.
14. Toxic or burn can cripple a baton pass switch in, especially against wall teams. For example, in Driftbloom's case, a Gliscor could easily apply toxic on the switch and just substitute + protect everything until the passer switches or dies.
Pretty sure there's more, but as you can see... there's way too many counters to stop a baton pass team. I purposely run baton pass teams on Showdown just to see all the different ways it can die and I guarantee you there are tonnes.