I'm afraid you shouldn't have got me started on this.
There is little to no evidence that pot is harmful in any way (besides the fact that smoke is harmful in general). The study that found "conclusive" evidence that THC (the active ingredient) reduces your cognitive abilities were done on monkeys (err...primeapes?), giving them increasing doses daily, and they had to give them something like 100 times the amount found in your average joint to notice any change. Proof that statistics can support anything you want them to support if you quote them vaguely enough.
That said, I'm a BC native (BC bud! famous worldwide!) who's smoked pot quite regularly for...5 years, I guess, and I can tell you firsthand it does make you stupider. Very slightly, very slowly, but it does. Also lazy, but it does that immediately.
I'd say your risk of any adverse affects from sitting in a smoky room occasionally are almost immeasurably low. It's also pretty unlikely for you to get noticeably high from sitting in a smoky room, unless you have a rather low tolerance or it's really good weed.
Erowid.com has a lot of good information on mind altering substances if you want some reading, and it's mostly unbiased. It has a slight drugs-are-super-fun bent to it though.