I think the "paperclip maximizer" as it's commonly related has been mutated into a completely different argument. In the original version, there's no construct about not knowing how many paperclips it has made, it's making a completely different point altogether than the bastardized version that spread around later. This was the wording of the original version:
https://en.wikipedia.org/wiki/Instrumental_convergenceSuppose we have an AI whose only goal is to make as many paper clips as possible. The AI will realize quickly that it would be much better if there were no humans because humans might decide to switch it off. Because if humans do so, there would be fewer paper clips. Also, human bodies contain a lot of atoms that could be made into paper clips. The future that the AI would be trying to gear towards would be one in which there were a lot of paper clips but no humans.
Notice that there's no mention of 100,000 or 1,000,000 paper clips here, unlike the later version which spread. And there's no mention of the AI not realizing it's already
made "enough" paperclips, because that concept is completely missing from the original formulation.
The
entire point here is that "weak AI" could be
really dangerous, because it has the "blind machine" aspect of traditional technology, along with the self-learning / growth capacity of AI. e.g. if you get in the way of an automated machine, it could fuck you up. Now, add in the capacity for self-learning and growth into that machine, and it's quite possible that a system designed to maximize something-or-other could go off the rails if allowed to grow by itself. This is basically the point:
https://wiki.lesswrong.com/wiki/Paperclip_maximizerThe artificial general intelligence (AGI) won't revise or otherwise change its goals, since changing its goals would result in fewer paperclips being made in the future, and that opposes its current goal. It has one simple goal of maximizing the number of paperclips; human life, learning, joy, and so on are not specified as goals. An AGI is simply an optimization process—a goal-seeker, a utility-function-maximizer. Its values can be completely alien to ours. If its utility function is to maximize paperclips, then it will do exactly that.
So, it could be extremely "clever" from one point of view, but being extremely clever at achieving stuff does not in any way imply the need for conscious self-awareness. Basically it would an an alien intelligence with 100% task-focus at the exclusion of all other values.