I'd say "drugs" in this context are more strongly directed toward psychoactive or hallucinatory stuff than anything else. There's a difference between recreational and medical drugs... or drug usage, at least. Abuse is abuse regardless of the intended usage of the material, though. Still, there's a difference between drug use for pleasure and other sorts.
As for me, I just don't like being around anyone that's on
anything that fucks with behavior, especially if I've known them when they're not under the effects. To wit,
No joke, and it's been a bit of a stumbling block with trying to get into my major, but being around people who are drunk or while they're becoming drunk is incredibly unsettling to me. Y'have no goddamn idea, makes my entire neural schema start to flip out. Even tipsy is something I have trouble wanting to stay anywhere near. Same for any behavior effecting drug, as well as cognitive degeneration due to disease or age. It's like... just a little buzzed? "No," goes Frumple's mind, "Person, you have just fucking poisoned yourself and now I'm watching you lose cognitive capability and control, flat out the most fundamental aspect of what makes you, you. This is existential horror writ libation, and now, self, I am going to gibber and suggest you claw at the walls and escape."
Not just alcohol, as noted. Heavy behavioral shifts due to willfully ingested chemicals makes huge parts of my psych start to flip the fuck out. Now, when they're not under the effects I don't really give a shit, but when they are...
And yeah, I indulge in caffeine occasionally, though I've been steadily reducing intake. Last time I took painkillers was... while passing a kidney stone? Yeah, think that was it. I'm really cautious with their use because they've almost killed me once or twice. Doc gave standard dosage, turns out I was sensitive enough it almost flatlined my blood pressure (and me). I seriously don't put antibiotics in the same category as anything that really fits into recreational categories, though. But recreational, psychoactive, hallucinogenic? No mas. The closest I've came was a couple of attempts at (prescribed, taken specifically as directions dictated.) antidepressants when I was younger, and both of those fucked me up so bad I'm now utterly terrified of having another try at it.
Would say it's safe to say someone that doesn't want to interact with people that willfully and regularly poison themselves for kicks isn't quite what I'd call an asshole move, though. Some people can deal with the sorts of personalities that do that, some can't. Wanting to avoid that interaction is a mostly neutral thing, I'd say.