Well, no. It's not "This thing signifies one sound and one sound only." It's a "... What? Okay, that's another one for the ' pile," scenario.
What would be the point of that. It is assumed that the language in question has both pronounceable and unpronounceable sounds, and the former outnumber the latter (else what would be the point in a Latin orthography anyway). If there are several unpronounceable sounds, it makes sense to have several symbols for them, because while the human readers can't pronounce them, they can distinguish between them, and therefore read (which is the point of a Latin orthography for Chtulhuese).
There is literally no scenario in which the "another one for the pile" approach makes sense. None. If you're going that way, there's no point to having an orthography, since Rule Zero for any orthographic system is "Thy graphemes shall have phonetic meaning that can be derived from their shape or optionally, position, but it shall be a concrete meaning in the end." If an apostrophe in any position could be any of your, say, three unpronounceable sounds, then there's no point to having an apostrophe. Nothing will do just fine, since the apostrophe does nothing for increasing a reader's comprehension anyway if it has no precise meaning.
On the other hand, if we have several different symbols for Cthulhu noises (' for mandible rattle, " for side-spiracle hiss) then a human reader can distinguish between, say "kth"ha'" 'tree', "kth'ha'" 'forest', "kth"ha"" 'water' and "kth"ha"" 'ocean'. With an apostrophe, those would have been indistinguishable, and you'd get meaningless texts. The meaningless texts would be supposedly written by humans (since they're in Latin). Now you must assume your humans are slightly dumb and can't make up a functional transcription system for Cthulhuese.
I might see a reason. Say that the average human speaker, when confronted with such an ' is supposed to just leave a short gap in the word where normally one of the alien sounds goes (which might not be in the human hearing range). Why? Because to a native listener this does convey information, namely 'here goes a sound the puny human cannot make', which might help him interpret what said puny human is trying to say. Sure, the academics of that world probably have a whole system to represent the various unpronouncable sounds, but to the layman that has no real added value. Seeing a '&' or a '¥' or a '#' (using random symbols here), while each of them indicate the same thing for practical purposes, won't help him speak or understand the language better. Though I don't know the setting or the situation it was used, so this might not be valid.
Say dirtfarmer Bob sees an alien he knows approaching, he might say "hello Ma-(pause)-ry", giving "Ma'ry", even though its real name is 'Ma¥ry', where the '¥' sound is one dirtfarmer Bob can't pronounce.
Another reason might be that these humans simply haven't mapped the various alien sounds yet, and are representing them with ' until they can look into it better.
Yes, but if you're gonna pronounce them, then having at least an approximation for alien sounds is better, no? When you try to pronounce something foreign, you try to approximate since presumably, native speakers need to understand you. I mean, if you have no spiracles, you can just tap your sides with your arms and hiss, but just making a pause there isn't really the best idea. Why would he address her as ma-pause-ry if that adds nothing for both of them - for him it'd be easier without a pause, and she presumably knows that he has no spiracles and can't pronounce it, so why not just drop it altogether?
I must also reiterate that creatures that share most of their phonetic inventory with humans but have additional sounds that humans can't even begin to approximate are pretty improbable in the first place.
The last argument holds some weight, but then there are races that were supposedly familiar to mankind for a long time. For those, the "not mapped yet" hypothesis makes no sense. Besides, really, the more such unpronounceable sounds your aliens have, the less point there is for a human to speak their language instead of just writing on a markerboard or something, because with the number of unpronounceable and unadapted fragments increasing, his comprehensibility to them would decrease.
Sorry, I'm just feeling argumentative today.
Yeah, but again, what if some of their sounds are beyond human hearing by a few hundred hertz or something? Hard to reproduce a sound you can't even hear (though that loses some weight since it appears to be a sci-fi setting, they could use equipment to record it and then try to find some way to emulate it, maybe).
As to what it adds: say the alien couldn't make the letter 'r' in any way. If it said 'both' it might mean 'both' or 'broth', and I wouldn't be sure. If it said 'b'oth' then I know there's supposed to be a sound there, so I might understand him a bit easier, cause the list of words it might have meant decreases. I admit the example with the name wasn't the best one, but still, you see my point? There's loads of words in English where, if you drop a letter/letters, you get a different word. Might also be for the alien language. So, the pauze indicates missing letters/characters/sounds/whatevs to help interpretation. Note that the 'missing sound' indication doesn't really have to be a pauze, they might waggle their eyebrows instead, and this is indicated with the '.
True about the fact that sharing most of the phonetic inventory 'except some extra bits' might not be extremely plausible, but then again, you're reading sci fi with aliens. You want it hard? Go read the NASA webpage
That last point I can easily concede, cause I don't actually know for sure what novels you were talking about (did you mention it yet?), so I dunno how familiar they 'ought' to be. I don't even know if they are actually speaking said language, or just found a voice recording of it they were transcribing, or if they know the aliens so well they get together for drinks on the weekend. Without context, I'm kinda arguing in a void, so I presumed a very general case.
And don't worry about it, not like you aren't right in saying it's probably lazyness of the author. Even
if we find a reasonable explanation, it's unlikely that's what he actually had in mind.