It's also functionally fear of heights. The part of your brain that does distance judgement doesn't compensate for being able to float freely.
This. Inclusive with unfamiliarity maybe :O
It's like, you get into a lake and then peer down and see...the darkness underneath, stretching for the whole vision of your eyes. Vast, and maybe with a point of basis for reference (like that speck of dirt floating near your goggles/eyewear) to emphasize it.
It invokes fear that way--usually unfamiliarity though. And that the brain works with concepts and meaning at a very quick speed that it needn't be in conscious thought if you're familiar with concepts...though displaced into other environments. So it may cause some kind of different understanding.
(eg Dark expanse under you = stability whatnow? = omg I'm alone and can I handle this?! <- Deep sea divers are familiar that there's something in there -> Explorers and historical anecdotes do include the "fear of the unknown" which was contrasted with familiarity = maybe it's just the expanse of darkness and the "depth of nothing" <- this can be contrasted with added knowledge, like 'I see nothing because that means [something]", like the 'vanishing point of sunlight' due to depth, or a really dark stone
)
And yes, it's generally a common reaction--maybe not as you described it but common in its basic structuring.