EH:
The view espoused-- that there is more involved in public discourse than the two people speaking (Specifically, that other people have a right to hear the public discourse)-- has no bearing on any perceived notion of "rightness" or "wrongness" in the speech.
You are confusing a "Right to hear" with a "Requirement to listen." They are not the same thing.
When a person decides to shut down speech of another, because they do not wish to hear it, they are denying other people the opportunity to hear (or read) that speech. This is one of the multitudinous arguments laid out by Mill, and he does a very good job of it too. (Being a classical empiricist.) Instead of shutting somebody down, therefore, the most societally forward thinking way to approach the problem is to recuse ONESELF, not to shut the OTHER down. This removes yourself from the conversation, and thus allows you to not hear any further dialog-- without removing the ability of other people to listen to the guy on the soapbox, if they so choose.
In instances where the other person is harassing you, and will not relent when you attempt such recusal, the various anti-harassment laws on the books are a good place to start from. This can physically prevent the person from being in your proximity, for instance. It does not prevent them from speaking to other people though, which is why it is preferential to using socially dynamic ways of shutting people down. (like the mansplaining phenomenon)
I WOULD go so far as to say "It is in your best interests to listen, even when you disagree with what is being said", rather than do the recusal and duck out-- At least if you are actually interested in obtaining empirical truths about the topics of the discourse. Mill mentions this as well, giving several noteworthy examples. He refers to it as "the combination of opposites." He supposes that each side will have some portion of the empirical truth, but with a lot of subjective bias, favoring the individual speaker's internal world views. To reach a neutral opinion of truth, one must wade through the quagmire of all opposing views, and distill the verifiable essences from all parties. This means that if your goal is truth or knowledge, you cannot just summarily dismiss another person's rhetoric. It *IS* mentally taxing, but if your goal is truth, it is the only way forward.
Combined with the prior, any person seeking such truth has a requirement to listen and hear the views being expressed, even if you yourself do not. Shutting down that speech, because you find it offensive, prevents the truth seeker from getting that speech, and denies them valuable sources of information. (There are all kinds of information to be gleaned from even purely propagandist speech, that is completely non-factual on the face of what is being discussed! The language of the propagandist, the sources they cite, and a number of other things are all of value to a person seeking universal and neutral truth of the topic!!)