Yea, you're right, mostly it goes like this
"A thing"
"Your thing is offensive and hurtful [to me]. [Please] stop it."
"Lol I can say what I want"
"No you can't"
"reeee muh free speech"
Fixed. It only ever comes up when someone says that people should be stopped from speaking, which happens an awful lot actually.
Your experience differs from mine, which could very well be due to miscommunication.
See, the people most often accused of "fighting free speech" that way are leftists and many of those (particularly the ones I am concerned about because I see them as peers) are
pacifists or close to being pacifists. If they say something along the lines of "you can't" (which is very unlikely to be the actual choice of words) they mean to tell you that it's not something you can do and
still be considered a decent human being. The assumption is that people do want to be decent human beings and thus would avoid doing something that makes them something else.
At least that is the case that I was talking about in the first place. I'm an anarchist and pacifist, I don't even
consider actively doing anything but looking at someone with disdain and making a snide (or angry) comment. And somehow that always is depicted as being worse than whatever I happen to criticize – be it rape jokes, homophobia, assertions that certain groups of people should not be allowed to make their own decisions, calls for murder or genocide…
FREE SPEECH!!! you know? Because my free speech ends where their's begins or something like that.