Google's and FHI's big red button paper is mathematically elegant. More specifically, the algorithm can be modified so that it fails to recognize that it is losing reward if it is switched to an interruption mode (halted, remote controlled, etc.). The paper mathematically shows that reinforcement learning can be modified to be interruptible. Despite press coverage of how this big red button is going to save us from rogue AI, the results from the paper are much more modest. Google/DeepMind and Oxford's Future of Humanity Institute co-published a paper that first introduced the big red button. In this project, we set up a simple environment to explore big red button issues and propose our own solution.

It may harm the human before he or she can activate the button. It may prevent the human from accessing the button.

If the robot is sufficiently sophisticated it may learn to prevent humans from pushing that big red button that stops the robot. Shutting down, interrupting, or manually controlling a robot may deny it from maximizing reward. Sounds straightforward, but robots that use reinforcement learning optimize expected reward. This might be one to protect the robot from damaging itself or from harming people. There might be situations in which you need to shut down the robot, interrupt its execution, or take manual control of it. Suppose you built a super-intelligent robot that uses reinforcement learning to figure out how to behave in the world. Big Red Button by markriedl Big Red Button Experiments with reinforcement learning agents that can be interrupted while learning View on GitHub Download.

Big red button usb hub