I suspect it's possible to do without a tavern, but I see no reason why you'd want to, and if you did you'd probably increase the fortress stress level consideraby. Taverns are basically essential to satisfy needs to Socialize. If you're worried about visitors, you can make the tavern off limits to them, and if you're worried about bar room brawls you can refrain from appointing poisoners (a.k.a. "bar keeper" and "performer") to them. Dwarves need to Pray as well as Socialize: it's not an either/or choice.
I don't use poisoners, but I do allow visitors. I have to kill off visitors occasionally as they tantrum (the stress system is hard on visitors as well), and there is a risk/chance of getting infiltrators*, which can be interesting and frustrating at the same time.
*Infiltrators are the results of some bugginess in the artifact quest system. Questers often come on groups, and sometimes one of them leaves the fortress and returns as stealthed thieves while the rest of the group remains inside the fortress. When the thief is engaged (not detected, but when either it attacks or gets attacked) the infiltrators become stealthed hostiles: some of them continue with their business (e.g. socializing) while still scaring locals, others run around in panic while still scaring locals, and some attack locals while still pretending to go about their business. Since none of the infiltrators are marked as hostile, they're hard to deal with, but I've found that active militia stationed close to them can detect them and eliminate them. I've also had a case of an infiltrator who ceased to scare locals after a while, and the militia sent there didn't detect anything odd.
Raven and Shonai_Dweller posted while I wrote:
There was a bug where guests could turn hostile when a siege showed up. This bug has mostly been fixed, but there's a recent report in the bug tracker showing this behavior (in that case pitting a prisoner results in madness).