... lucky you? If you're saying you've never actually been in an area that had congregations of violently bigoted people, anyway. I've been living in and around them all my life, and I've seen the results of what happens when you do pretty much anything but join in or stay silent -- and sometimes not even the latter'll save you. Still places in the states where asking "why" would literally get you killed and definitely get you shitkicked, and plenty of places elsewhere.
Lucky me indeed, for a few things at least. When you are getting people lynched and such, that is physical assault and should be dealt with the law. When asking 'why' leads to statements of 'I'm going to kill you,' that is also the time for the police to get involved, as clear intent of an upcoming crime has been given.
Secondly, you really don't need to live near the local bigot's meeting spot where they dastardly plot the next way to insult everyone to live near local bigots. It is just a matter of distinction that you don't need to form everything into groups and cabals for it to be harmful.
Wouldn't that heuristic render public speaking or broadcast a complete impossibility?
In such cases where it is spoken through a private medium, it is very easy to show that you have consented when you change the channel to watch that show, or attend that speech. The consent-ideal would be to void the validity of being able to harass individuals or start shouting at strangers in public, whilst also preserving their right to express themselves.