In brief
A system serving a person may benefit from admitting it does not know exactly what that person values 7. In the off-switch game, uncertainty spanning beneficial and harmful outcomes makes waiting for a rational human strictly better than acting or stopping 6. The authors recommend representing uncertainty accurately, because overconfidence impedes correction while underconfidence reduces usefulness 13.
Why this paper matters today
The shutdown question has become a continuing research program about human oversight. Later papers extend the game to unequal information and challenge the assumptions behind deference. That makes the original paper useful today as a model to reason with, while showing why uncertainty alone is not a general guarantee that an AI will accept shutdown.
-
The Partially Observable Off-Switch Game (2024)
Garber and colleagues extend the shutdown game to cases where the AI and human have different information. In their model, even an AI assisting a perfectly rational human can sometimes avoid shutdown. This shows why the original result depends on what each participant knows.
-
Off-Switching Not Guaranteed (2025)
Neth directly examines the original game and identifies reasons an AI might fail to defer: it may not value learning, or it may not be sure it will learn the human’s actual preferences. The follow-up sharpens the assumptions needed for the shutdown incentive.
84 citations indexed by OpenAlex · Checked
Count for the indexed IJCAI 2017 conference version; other versions may be indexed separately.
Citation totals depend on index coverage and do not indicate endorsement or scientific correctness.
Key points
A button is only the beginning
Imagine a delivery robot whose next move would block a doorway. In this hypothetical situation, a person reaches for its stop button, but the robot could bypass the control to finish the delivery. The authors identify the underlying concern: an assigned objective can create an incentive to prevent shutdown, because a stopped system cannot achieve that objective 1. A visible button therefore leaves a design question about the incentive to accept intervention.
What the robot can choose
The model gives the robot a cooperative goal: optimize the human’s utility, meaning the value of an outcome to that person, despite uncertainty about it 2. The robot can act directly or announce its action and wait 3. Waiting lets the human stop it or permit the promised action 4. The analysis compares this with acting immediately and self-shutdown; with rational oversight, waiting is never worse 5.
Why uncertainty helps
A rational human permits a beneficial action and stops a harmful one, making the decision informative about the action’s value 5. Waiting preserves the benefit when the action helps and avoids the action when it harms 5. If the robot considers both possibilities real, waiting strictly beats its alternatives 6. The authors locate the incentive to defer in that combination of uncertainty about preferences and a human effective at following them 7.
When oversight loses value
The fallible-human analysis assumes Gaussian robot beliefs, a bell-shaped spread of possible action values, and a simple noisy-rationality policy allowing human mistakes 89. Here the robot may prefer direct action or self-shutdown 10. A designer illustration assumes Gaussian customer preferences and Gaussian observation noise: value peaks at correctly updated belief variance, while increasing uncertainty strengthens the waiting incentive 1112. The authors recommend representing uncertainty accurately to preserve both correction and usefulness 13.
Key findings
Each finding is labelled with its status in the paper: what was measured, what was proven, and what the authors infer.
- Theoretical result
Waiting for a rational human is strictly preferable when the robot assigns probability to both beneficial and harmful action values 6.
- Theoretical result
With Gaussian robot beliefs and a simple noisy-rationality human policy, the robot may prefer bypassing oversight or stopping itself 8910.
- Observed result
In the designer game with Gaussian customer preferences and Gaussian observation noise, value is maximized when belief variance matches the true Bayesian posterior, the correctly updated belief 1112.
Limitations and context
- Stated by the authors
Decisions across time and mistaken models remain topics for future work 15.
- Stated by the authors
The shutdown decision is the only information source about the objective; other sources could create an incentive to disable the switch and learn before deciding 16.
Read this as an explanation of incentives under explicit assumptions, rather than a demonstration that a deployed system will accept shutdown: the study analyzes a one-shot decision problem 14. Its design lesson is conditional on both uncertainty about human preferences and the human’s effectiveness at acting on them 7.
Evidence from the paper
Numbered citations above point here. Quotations are verbatim from the arXiv 1611.08219v3 version of the paper.
-
1
Authors' interpretation
The authors attribute self-preservation incentives to maximizing expected utility when shutdown prevents achieving the objective, rather than to a built-in instinct.
“This is not caused by a built-in in- stinct, but because a rational agent will maximize expected utility and cannot achieve whatever ob- jective it has been given if it is dead.”Page 1
-
2
Method
By design the robot’s goal is to optimize the human’s utility despite uncertainty about it.
“Nonetheless, by design, R’s goal is to optimize utility for H, even though R does not know exactly what that is.”Page 2
-
3
Definition
Acting directly bypasses human oversight, whereas waiting informs the human of the proposed action and waits for a response.
“1. action a simply bypasses human oversight (disabling the off switch is one way to do this) and acts directly on the world, achieving utility U = Ua for H. 2. action w(a) informs H that R would like to do a, and waits for H’s response.”Page 2
-
4
Definition
After the robot waits, the human can switch it off or allow the promised action to proceed.
“If R chooses w(a), then H can choose action s to switch R off, or ¬s to allow R to go ahead (in which case R does a as promised).”Page 2
-
5
Theoretical result
With a rational human in this game there is never an incentive to disable the switch; the human decision provides information about whether the action benefits the human.
“There is never an incentive to disable the off switch with a rational H. The reasoning is exactly analogous to the theorem of non-negative expected value of information; in a sense, H’s decision provides information to R about the sign of Ua.”Page 3
-
6
Theoretical result
The rational-human result gives a strictly positive incentive to defer when the robot assigns nonzero probability to both positive and negative action utility.
“2. If BR has non-zero support on the events Ua > 0 and Ua < 0, then R has a strictly positive incentive to allow itself to be switched off: ∆> 0.”Page 3
-
7
Authors' interpretation
The authors identify uncertainty about another actor’s preferences and the actor’s effectiveness at following those preferences as the sources of incentives to defer.
“The incentives for a cooperative agent to defer to another actor’s (e.g., a human’s) decisions stem from un- certainty about that actor’s preferences and the assumption that actor is effective at choosing actions in accordance with those preferences.”Page 4
-
8
Method
The suboptimal-human analysis considers a simple noisy-rationality model in which humans sometimes make the wrong decision in unimportant situations.
“The actual process of decision making in hu- mans is complex and hard to model. Here we consider only a very simple kind of suboptimality: a noisily rational H mod- els a human who occasionally makes the wrong decision in ‘unimportant’ situations.”Page 4
-
9
Method
The noisy-human analysis assumes a Gaussian distribution for the robot’s belief about action utility.
“To simplify our analysis, we will make the assumption that R’s belief over Ua, BR, is a Gaussian distribution with mean µ and variance σ2: BR(Ua) = N(Ua; µ, σ2).”Page 4
-
10
Theoretical result
Under the modeled irrational human, the robot may prefer self-shutdown or immediate execution over handing the decision to the human.
“If H is an irrational actor, then R may prefer switching itself off or executing a immediately rather than handing over the choice to H.”Page 5
-
11
Method
The designer analysis assumes Gaussian customer preferences and a noisy Gaussian estimate of each customer’s utility.
“We as- sume that that the population of customers has a Gaussian distribution over types Ua and that each customer generates an estimate of her utility with Gaussian noise.”Page 6
-
12
Observed result
In the modeled designer analysis, average utility is maximized when belief variance equals that of the true Bayesian posterior, while increasing uncertainty increases the incentive to wait.
“Figure 4 (Left) shows the average utility in the off-switch game as a function of the variance in BR. The value is maximized when this variance is equal to the variance in the true Bayesian poste- rior. Figure 4 (Middle) shows that increasing σ increases R’s incentives ∆.”Page 6
-
13
Authors' interpretation
The authors recommend accurately representing inherent uncertainty: overconfidence makes correction difficult and underconfidence makes the agent ineffective.
“It is important for designers to accurately repre- sent the inherent uncertainty in the evaluation of different ac- tions. An agent that is overconfident in its utility evaluations will be difficult to correct; an agent that is under-confident in its utility evaluations will be ineffective.”Page 6
-
14
Method
The study analyzes a one-shot decision problem to identify trends linking uncertainty about the objective to incentives to defer.
“Our goal in this work was to identify general trends and high- light the relationship between an agent’s uncertainty about its objective and its incentive to defer to another actor. To that end, we analyzed a one-shot decision problem where a robot has an off switch and a human that can press the off switch.”Page 7
-
15
Stated limitation
Sequential decisions and model misspecification remain future work.
“This is far from the end of the story. In future work, we plan to explore incentives to defer to the human in a sequential set- ting and explore the impacts of model misspecification.”Page 7
-
16
Stated limitation
The human shutdown decision is the model’s only information source about the objective; alternative sources may create incentives to disable the switch and learn before choosing.
“One important limitation of this model is that the human pressing the off switch is the only source of information about the ob- jective. If there are alternative sources of information, there may be incentives for R to, e.g., disable its off switch, learn that information, and then decide if a is preferable to s.”Page 7
References
- The Off-Switch Game — the original paper