Personally, I think we are just as mindless as a ANSI C "Hello World!" program running on DOS.
There's a loosely and transiently collected bunch of atoms and informatic signals, here, identifying itself as "me" or "I". This 'me' is sitting and interacting with (among other things) a separate set that 'me' calls a "keyboard", having received impulses from another set described as a "monitor". All of these are obeying (most probably deterministic) physical laws in their internal functioning and interact in countless ways with each other as well as all the other collections (and meta-collections) which ultimately composes the environment of their existence. The "I" collection definitely has impulses within itself representing a similar world-model as apparently contained by the unknown entity (provisionally entitled "dree12") whose existence is only indicated through a number of proxy interactions.
That is assuming the "I" is truly correct in its world-model even of the keyboard, never mind the external entity, but that way lies territory for which the classic Cogito argument needs to be invoked.
In other words, I agree with you. But then I was always going to, the way I've been continuously programmed and re-programmed by the universe.
I imagined trying to help the creature process the nature of its existence as a computer program, showing it video feeds of the "real world", and wondering what that experience would be like.
Unless the creature existed in a simulated 3d environment with a standard "movie type" input stream... it would probably be about as overwhelming and non-understandable as feeding a million DB of white noise and flashes of light into a human's sensorium.
I assumed a 3d environment, and setting up a plane with a texture linked to a live camera video feed.
To what degree is the simulated creature an exact environmental simulation? Does it fully perceive "line of sight" objects and only them? In seeing a flat image of a live video feed, projected as per a flat plane, does it not only have difficulty in understanding that
that parallelogram is a perspective view of a rectangular wall, but also that (unlike its own world) there are things that are partially obscured by other objects (if not outright hidden) and are furthermore not plainly labelled within its consciousness with an id-tag of some kind (whether or not they are to be consciously ignored) and has to be analysed in a way far beyond the level of "That is <screen1>, upon it is <texture_livefeed1>". Is the problem not so much that, Matrix-like, it has to learn to "see the code", but that it
is seeing the code and it has to learn to "see the representation". And match the application of that representation of its internal world with the dimension-shaved representation of an external world being given in a completely ineffable manner.
And in trying to create a simulated world that we recognise and can work with, there are a number of layers of abstraction between "what the code does" and its representation to us as a 'petting zoo' with which we can interact.
There are a number of aspects to even the background simulation which are anthropocentric. I'm not talking about bipedalism, here, but that there are a number of imported aspects (common to most simulated or 'gamiverse' worlds). There's a "down" and an "up", I'm pretty sure, which has no reason to exist (certainly has no basis regarding large amounts of matter concentrating a force of attraction towards a particular surface) but is a sop to our environment. Euclidean geometry is almost always a given, as is the existence of paths that are either looped or have exactly two ends.
If it's a 3D world being represented, that just panders to us even more. Why should the world not be 4D in nature? The maths and code behind working out the interactions in such a world isn't any more complicated than for a 3D world, or a 2D one. Memory issues, of course. Plus representing the world to the viewers, i.e. us. Although (again) the maths for making a 3D approximation of a 4D world is similar to that which we already need to apply to a 3D world (or the first approximation of the 4D one) in order to represent upon a 2D display device... (I think there's at least one person on this forum with a rotating tesseract as their avatar image, which does exactly that in taking a 4D entity and bringing it down to 2D.)
But this eventually leads back to my "old chestnut", the Conway's Game Of Life board containing such complex patterns that 'they' consider themselves conscious. Never mind that we can poke various pixels on and off if we want, check the state of the entire board (freezing it at will, reverting it, wiping it). If a mass of pixels that is an 'entity', or 'consciousness', does exist it won't look like a stick-figure man write-huge, 'throbbing veins' of pixels pulsing up and down its arms, picking up a "pixelated rock" group of cells and throwing it. It'll be a mass of information that moves by "translating" itself across the grid and as well as that growing and shrinking with no reason to obey any Conservation of Mass/Energy laws such as appear to work within our own universe. It will interact with its environment by the stable (or meta-stable) state of 'its' own cells meshing with cells currently not part of 'itself' and the surface meshing translating (for both the being and the interacted object) into ripples of altered states passing through the respective 'masses'.
(Hypothetically, I imagine that the "skin" of a Conway-being would be a mass of 'feeler' tendril structures, continually maintained and/or regenerated by the underlying substrate. As and when these feelers 'feel' something else this disrupts them, sends a ripple down to the substrate which absorbs it (translating it into a subtle disturbance in a form of transmitted signal, probably far more complex than a "glider stream", to travel towards/form part of the creature's consciousness/awareness) and then reinforce the tendril to prevent its re-assimilation or loss. But that's probably anthropomorphising things far too much and it'd be nothing like that anyway.)
And thus would we recognise 'consciousness'? We have the unique power to look across the entire substance of a mass of cells, but could we truly say "that group of cells is thinking"? To us, we can rewind, poke, prod, etc, and the consciousness would know nothing about it. One 'time line' may cease to continue even while another run-through from a past position now has a 'creature' with a pixel changed here, there... Or whole segments of it whited (or blacked) out through an over-hasty application of a "paintbrush". If, indeed, it was restarted.
From a theological point of view (as a comparison, not wishing to start
that debate again), the omnipotent deity could be doing all kinds of things to us right now. Maybe "gravity" is God interfering with every single particle, in every single tick, making sure that they all obey some force that is not programmed into the original simulation. You cannot even trust that the book you put down just now won't suddenly become a raging bull elephant if we are being actively prodded and poked. But we would have to accept it as part of our universe that such things happen. Gravity is natural because it happens, paperbacks turning into preposterously petulant pachyderms is not natural because (at least from my current knowledge) it does not.
A Conway-creature with an analytical mind would derive what it could of its universe. It would never derive a law of conservation of mass, energy or momentum, because none of these exist in its world. It may derive a "speed of light" equivalent, which may be a measurement that we know as one-cell-per-tick (the absolute fastest that any change can propagate across a mass of cells) except that this would require a mass of cells primed to pass on this change at that speed. "In a vacuum" (in a 'cell-off' expanse) the fastest speed is the speed of a glider cell (or equivalent) travelling through this expanse. Which is of an order of one diagonal cell every four ticks, IIRC, though I think there are ones that move one cell non-diagonally in two ticks. However, there would be no way to examine individual tiles as being "on" or "off", and the scale of the "tick" would be far shorter than any "observational equipment" could be 'built' to directly measure, themselves being constructs who might have a resonance at the scale of the ticks, but could not actually count the individuals.
The Conway-creature taking upon itself the role of an experimental geometer would be able to establish that his world has an orthogonal bias (there is an underlying grid system), due to the relationship between the horizontal/vertical expanses and the diagonals. Not that it could imagine a diagonal distance as Root(2) the distance of the equivalent axial distances, as Manhattan or pseudo-Manhattan metrics only are relevant in such distance calculations. Either way, consider this in the same way as we don't really know whether anyone currently understands the universe correctly (theory of relativity, quantum theory, string theory, M-theory... all good models and are good (capital-T) Theories for some aspect or other (or several!) of our universe, but may not be "The Answer". And the fine details of the universe may be as unknowable, certainly unprovable, even if some GUT comes up that is completely accurate.)
TL;DR; As far as I can see, simulated consciousness in an "our world" type of simulated environment are abstracted beyond their own understanding. Simulated consciousnesses in an environment more suited to themselves are abstracted beyond
our understanding. There may be a half'n'half solution, but I anticipate compromises on both sides of the board to bridge the gap between one-trick ponies (even if it's a pretty good trick of pretending to be a fully interactive 'pet dog'-like being with apparent emotions, et al) and the truly esoteric kind which we may never even realise the full nature of.