The "noneuclidean geometry" that lovecraft was yabbering about is more or less easy for most scifi readers to grasp:
Our 4 dimensional spacetime reality has "objects" inside it that are cross-sections of objects that exist in higher dimensional spaces. EG, how an MRI scan slice of a brain is. (Our brains are 3D objects, the MRI scan is a 2D slice of that 3D object. The eldritch abominations are similar to the MRI scan slice, except they are a 4D spacetime slice instead of a 2D timeless slice.)
Because of their existence in a higher dimensional spacial context, to them, the shortest distance between any two points may be signficantly shorter than for us, who are bound to our 4D spacetime. EG, the famous "Folding space" metaphor. (Draw two dots on a sheet of paper. What is the shortest distance between them? Well, you can draw a line on the paper between the two points (Shortest 2D distance), or you can make the distance 0, by curling the paper over on itself, so that two dots touch. (Shortest 3D path.)
Attempting to understand (and make use of) this higher dimensional gemoetric understanding of these creatures somehow causes madness in lovecraft's novellas, but that's another story completely. (I've touched on it previously already in other threads. Lovecraft apparently thought people were completely unable to handle such things at all. They go crazy at the smallest thing.)
One interesting feature of such a non-euclidian manifestation would be that it may appear to completely change form/structure, depending on the viewing angle, kinda like a hologram foil cup does, only with real manifestations (to us) instead of mere plays of light.