Take our hypothetical "nigger"-word user. Locally, it doesn't mean much. Maybe, rather than brown skin, it's come to mean someone who "works hard for little reward". (I'm sure you can follow the potential for that sort of linguistic drift)
The people in our hypothetical example have long since forgotten the real meaning of the word - they aren't familiar with its slavery connotations, aren't even really aware there were areas were slaves were predominantly black and called niggers, or where nigger was used as a slur. In essence, they don't even know "niggers" are a thing beyond "a person who works hard for little reward".
That's fine, then - they can use that word amongst themselves. But when they enter an international stage, and tell someone who's working too hard "you should relax! Stop acting like such a nigger." it's obviously going to get a response. And the proper course of action, when the history of the word is explained to him, is to stop using it around that audience. And perhaps be more aware of the other meanings for the word, to learn something, and perhaps avoid using it all in areas where it could be a problem. An apology isn't needed, but would help assuage aggressive feelings on both sides.
In essence:
It is alright for you to use it where you are to mean what the locals use it for. But when its pointed out in other company that the word is offensive, or even potentially offensive, one should probably make an attempt to not use it.
And come on, Roo-fucker isn't a slur, we all know what you get on to on that side of the water.
(But really a playful insult is something very different than what we are talking about, here. And there ARE a decent number of Roma in New Zealand, at least, and probably Australia as well. So even if you don't see it as an insult locally, you may very well interact with people who do and just don't say anything because of the response we're getting right now.)