Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Reelya]

In the hypothetical, the machine is always capable of obtaining more resources, so saying it'll run out of CPU isn't really going along with the scenario. Paperclips is obviously just the fluff, it's meant to be about the possible ways any self-growing AI with hard-wired goals could fuck up.

The main point about the bayesian actor thing, is that there's always some finite probability that you've failed to achieve some goal (e.g. if you made 1 million paper clips, what if some were stolen or faulty, they need checking). If you read the article, he makes the point that more and more resources would be expended by such a system in an attempt to minimize uncertainty, and like a blind machine making infinite paperclips, it won't sensibly cut off the expenditure of resources to minimize uncertainty either, unless that's explicitly built into it as a goal, or it conflicts with another goal.

[Tack]

I have to say, I own Superintelligence, and the phrase:

though I suppose that makes for a dryer and more boring book to sell.

Severely underestimates how frustrating this book is to try and read.
This being said I don't generally go in for reading scholarly articles, but boy it flashes me back to trying to hunt down citations for an assignment by poring through word count chaff.

[Reelya]

Yes, but it has to consider obtaining said resources. It can't do that if 100% of all its processing is being used for reward signals.

That sounds like circular logic there. If 100% of it's processing is reward signals, it couldn't make even 1 paperclip, let alone do anything.

Saying that such a machine couldn't obtain additional resources to use towards any of it's goals, under any circumstances doesn't seem like a sensible concept.

e.g. the machine must allocate some of it's processing power towards making paperclips. And, if it takes more than an immediate short-term view of paperclips (e.g. tries to maximize the rate at which paperclips can be made) it must therefore decide to allocate some proportion of it's processing power towards increasing it's capacity. The exact balance of processing power to current production vs future production would be decided by the machine, but such as system would definitely be capable of deciding it needed additional resources.

[Tack]

Yes but Reelya it'll be hard enough getting the owl to make paperclips. Why bother speculating about containment?

[wierd]

Sounds like a design/implementation issue.

The AI needs to have processing available to handle its reward/pain signals (It could be argued that NOT producing a paperclip is 'painful' to the AI, because it has a negative reward weight) but also needs processing ability available to evaluate changes to its methodology, or to devise new methodologies. (It is a MAXIMIZER, after all, it has to devise new strategies to continue maximizing its core goal.)

Depending on how much of its processing power is devoted to each task, it could be very ineffectual/unable to perform its mazimizing task and spend all its time 'suffering', or be unable to adapt to a change in situation that causes its initial strategy to fail.

[Reelya]

I think bringing ideas about reward/pain signals implies a very specific type of neural-network architecture with reinforced learning? I don't think that's warranted. That implies a specific architecture, which isn't part of the original concept.

[wierd]

I think it is kinda requires that some kind of handwave is required there.

The supposition is that the AI could go completely rampant, and turn every molecule of metal in the universe into paperclips.

To accomplish such an end game, the maximizer needs to devise space travel, WITHOUT USING METAL, (Since paperclip production has higher priority than spaceship construction), meaning that the AI will have powerful disincentives to form a strategy that causes it to leave its parent planet to seek more raw materials before it completely exhausts local supply--- even to be aware that it has nearly exhausted local supply in the first place.

A dumb algorithm will simply use its default strategy until it runs out of resources, then go into failure mode constantly looking (and failing to find) available resources to construct paperclips from.  The very notion of moving to another location would require computations far removed from its primary goal, which it has no incentive to try. At best, a simple algorithm would attempt some kind of linear mutation, which is more likely to completely halt the processor than it is to create a local solution to the local problem the algorithm is facing when it runs out of metal to make paperclips from.

[wierd]

Indeed. It is the most efficient "solution".  (Paperclip signal created, no energy spent processing metal, only the energy to flip a few bits)

It needs to be explicitly forbidden. (A linear mutation methodology is more likely to result in wireheading (if just accidentally due to off by one errors, etc) than actual workable solutions, which the decision weighting system will then select for barring some other conditions.

[wierd]

You need to better define what constitutes a paperclip, and use as many diagnostic features as possible to prevent single mutations from producing the undesired outcome.

EG, a paperclip is made of matter, and that matter has a mass which is measurable. Is there something in the output tray? (Did the tray cycle/is this a new paperclip being measured?) Does it weigh the right amount, more or less? Was material consumed to produce it? etc...

These would be questions where even a single one being a "no" results in "Paperclip=NO", and failure to get the reward signal-- but this again gets to the halting problem in a way; For the same reasons you cannot be certain that any arbitrary input will not halt the computer, you cannot be certain that no arbitrary mutation will not circumvent the restriction, and enable wireheading.

The best you can do is prevent mutations to the decision weighting system (Read only memory, with fault tolerance), and require agreement with that system before a signal is considered valid.

[Helgoland]

The problem is how you'd actually go and forbid it. Restrict direct control over its input, and it would go after indirect control. If it can't mimic the data counter telling it how many paperclips it produced, it can hack the counter and usurp control that way.

"Seize the means of production," if you will.

Taking this back to the human level: Communism == opium?

[wierd]

Not so much communism, so much as being able to (and getting away with) redefining what it means to meet a goal. (See 1984, and soviet communism, and their prenchant for redefining production goals so that even when a serious shortage happened, it was a !!success!!)  That is a powerful motivator to focus more and more on the fabrication of falsehoods instead of actually seeking real success, because it is divorced from reality and its constraints.

[Tack]

Blockchain wireheading?

[wierd]

Only if you disregard the requirements of the blockchain...

[Bumber]

The supposition is that the AI could go completely rampant, and turn every molecule of metal in the universe into paperclips.

To accomplish such an end game, the maximizer needs to devise space travel, WITHOUT USING METAL, ...

Unless the metal is paperclips.

[Starver]

The supposition is that the AI could go completely rampant, and turn every molecule of metal in the universe into paperclips.

To accomplish such an end game, the maximizer needs to devise space travel, WITHOUT USING METAL, ...

Unless the metal is paperclips.

No metal needed, just the odd human...