In brief
Christiano et al. (2017) trained agents without the task’s reward function: a person compared pairs of short clips, a reward predictor learned from those choices, and the agent maximized the predicted reward 45. On eight simulated robot tasks, 700 labels nearly matched reinforcement learning on the true reward 1012. Feedback given only at the start of training led to bizarre behaviour, so the authors conclude it must be intertwined with learning 1819.
Why this paper matters today
This paper showed that a reward model fitted to human comparisons of short clips could train deep reinforcement learning agents in simulated robotics and Atari games. Later work carried the same recipe of human comparisons, a learned reward model and reinforcement learning over to language models, where it became widely known as reinforcement learning from human feedback (RLHF). Later surveys also examine its limitations. Its relevance today is as the origin of a widely used training method, not as evidence that the method reliably captures what people want in deployed systems.
-
Training language models to follow instructions with human feedback (2022)
Ouyang and colleagues, including Christiano and Leike, cite this paper as the origin of reinforcement learning from human feedback and use the technique to fine-tune GPT-3 to follow written instructions. They collect human rankings of model outputs and report that, in human evaluations on their prompt distribution, outputs from the resulting InstructGPT models were preferred; they note the models still make simple mistakes.
-
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (2023)
Casper and colleagues survey open problems and fundamental limitations of RLHF, which they say this paper popularized. They stress the importance of a multi-faceted approach to developing safer AI systems and propose auditing and disclosure standards.
1,643 citations indexed by OpenAlex · Checked
Count for the OpenAlex record that combines the arXiv preprint (1706.03741) and the NeurIPS 2017 proceedings PDF; citations are not separated by version.
Citation totals depend on index coverage and do not indicate endorsement or scientific correctness.
Key points
A goal that is hard to write down
Many tasks, such as cleaning a table, have goals that are hard to turn into a score. The authors warn that a simple hand-written reward is often optimized without the behaviour people wanted, a difficulty they tie to concerns about misalignment between human values and the objectives of reinforcement learning systems 1. Learning from demonstrations is not directly applicable to behaviour people find hard to demonstrate, and rewarding every moment by hand would take far too long 23.
How comparisons become a reward
A person sees two clips of the agent and picks the better one, calls them equal, or says they cannot tell 58. A reward predictor, a neural network, learns to predict these choices; like a chess rating, a larger gap in predicted reward means a higher modelled chance of being chosen 9. The agent learns by reinforcement learning, trial and error that maximizes reward, from the predictor’s estimate. All three processes run at the same time 7.
How close it gets to the true score
On eight simulated robot tasks, 700 labels nearly matched reinforcement learning on the true reward, though learning was less stable 1012. On those robot tasks, real human feedback was typically only slightly less effective than synthetic feedback from an oracle using the true reward, from half as efficient to equally efficient depending on the task 13. Atari was harder: the method learned substantially on most of seven games and matched or exceeded reinforcement learning on some, while with real human feedback it failed to beat the first level of Qbert 1415.
Tasks without a hand-written reward
The method also trained behaviours for which no reward function was available 16. With 900 queries in less than an hour, a simulated Hopper robot learned to backflip, land upright and repeat 16. These behaviours were judged by watching them, and the feedback came from the authors 617.
Why feedback has to keep coming
When comparisons were collected only at the start of training, the predictor captured only part of the true reward, and maximizing it could lead to bizarre behaviour 18. On Pong, the agent sometimes avoided losing points without scoring, producing extremely long volleys 18. The authors conclude that human feedback needs to be intertwined with learning rather than provided statically 19.
Key findings
Each finding is labelled with its status in the paper: what was measured, what was proven, and what the authors infer.
- Observed result
On eight simulated robot tasks, 700 labels nearly matched reinforcement learning on the true reward, with less stable, higher-variance learning 1012.
- Observed result
Real human feedback was typically only slightly less effective than synthetic feedback, from half as efficient to equally efficient depending on the task 13.
- Observed result
A simulated Hopper robot learned to backflip, land upright and repeat from 900 queries in less than an hour of feedback 16.
- Observed result
Collecting comparisons only at the start of training let the agent maximize a partial reward; in Pong it sometimes produced extremely long volleys without scoring 18.
Limitations and context
- Stated by the authors
Each real-human-feedback result is a single run, and Reacher and Cheetah feedback came from an author rather than contractors 11.
- Stated by the authors
The novel behaviours were trained with feedback from the authors and evaluated by watching them, without a reward function to score them 617.
- Editorial context
All experiments use simulated robots and Atari games; the paper does not test real robots, language models or deployed systems.
These results show that a modest number of human comparisons can stand in for a missing reward in simulated tasks, nearly matching the true reward on the robotics benchmarks 12. They do not show that a learned reward reliably captures what people want under stronger optimization; the offline-feedback failures show how a partial reward can be exploited 18.
Evidence from the paper
Numbered citations above point here. Quotations are verbatim from the arXiv 1706.03741v4 version of the paper.
-
1
Threat model
The authors argue that a simple hand-designed reward function that only approximately captures the intended behaviour (for example for a robot cleaning a table or scrambling an egg) will often produce behaviour that optimises the reward without satisfying our preferences, and that this difficulty underlies concerns about misalignment between human values and the objectives of RL systems.
“For example, suppose that we wanted to use reinforcement learning to train a robot to clean a table or scramble an egg. It's not clear how to construct a suitable reward function”Page 1
“this will often result in behavior that optimizes our reward function without actually satisfying our preferences. This difficulty underlies recent concerns about misalignment between our values and the objectives of our RL systems”Page 1
-
2
Background
Inverse reinforcement learning and imitation learning need demonstrations, so they do not directly apply to behaviours that are hard for humans to demonstrate, such as controlling a robot with a very non-human body.
“these approaches are not directly applicable to behaviors that are difficult for humans to demonstrate (such as controlling a robot with many degrees of freedom but very non-human morphology).”Page 1
-
3
Background
Using human feedback directly as the reward is prohibitively expensive for RL systems that need hundreds or thousands of hours of experience; the amount of feedback must fall by several orders of magnitude.
“using human feedback directly as a reward function is prohibitively expensive for RL systems that require hundreds or thousands of hours of experience.”Page 2
“we need to decrease the amount of feedback required by several orders of magnitude.”Page 2
-
4
Method
The approach learns a reward function from human feedback and then optimises it, fitting the reward to the human's preferences while simultaneously training a policy to maximise the current predicted reward.
“Our approach is to learn a reward function from human feedback and then to optimize that reward function.”Page 2
“Our algorithm fits a reward function to the human's preferences while simultaneously training a policy to optimize the current predicted reward function”Page 2
-
5
Method
The human compares short video clips of the agent's behaviour rather than giving an absolute numerical score; the authors found comparisons easier for humans in some domains and equally useful for learning preferences. In all experiments clips are between 1 and 2 seconds long.
“We ask the human to compare short video clips of the agent's behavior, rather than to supply an absolute numerical score. We found comparisons to be easier for humans to provide in some domains”Page 2
“The human overseer is given a visualization of two trajectory segments, in the form of short movie clips. In all of our experiments, these clips are between 1 and 2 seconds long.”Page 5
-
6
Method
The method is evaluated quantitatively when the preferences are generated by a known reward function (the agent should then score nearly as high as RL optimising that reward), and qualitatively when no reward function exists, which the authors say is the situation where the approach would be practically useful.
“So if we know the reward function r, we can evaluate the agent quantitatively. Ideally the agent will achieve reward nearly as high as if it had been using RL to optimize r.”Page 4
“Sometimes we have no reward function by which we can quantitatively evaluate behavior (this is the situation where our approach would be practically useful).”Page 4
-
7
Method
Three processes run asynchronously - the policy interacts with the environment and is updated by RL to maximise predicted reward; pairs of segments from its trajectories are sent to a human for comparison; and the reward estimate is optimised by supervised learning to fit the comparisons collected so far.
“The parameters of π are updated by a traditional reinforcement learning algorithm, in order to maximize the sum of the predicted rewards”Page 4
“The parameters of the mapping ˆr are optimized via supervised learning to fit the comparisons collected from the human so far.”Page 4
“These processes run asynchronously, with trajectories flowing from process (1) to process (2), human comparisons flowing from process (2) to process (3)”Page 4
-
8
Method
For each pair the human can choose one clip, say they are equally good, or say they cannot compare them; incomparable pairs are not added to the database.
“The human then indicates which segment they prefer, that the two segments are equally good, or that they are unable to compare the two segments.”Page 5
“Finally, if the human marks the segments as incomparable, then the comparison is not included in the database.”Page 5
-
9
Method
The reward predictor models the probability that the human prefers a clip as depending exponentially on the predicted reward summed over the clip (the Bradley-Terry model), trained with cross-entropy loss. The authors compare it to Elo ratings in chess - the difference in predicted reward of two clips estimates the probability that the human chooses one over the other.
“assume that the human's probability of preferring a segment σi depends exponentially on the value of the latent reward summed over the length of the clip”Page 5
“Just as the difference in Elo points of two chess players estimates the probability of one player defeating the other in a game of chess, the difference in predicted reward of two trajectory segments estimates the probability that one is chosen over the other by the human.”Page 5
-
10
Method
The robotics experiments use eight simulated MuJoCo tasks from OpenAI Gym, slightly modified; the agent is trained with 700 queries to a human rater and compared with 350, 700 or 1400 synthetic queries and with RL on the real reward.
“The first tasks we consider are eight simulated robotics tasks, implemented in MuJoCo”Page 6
“Figure 2 shows the results of training our agent with 700 queries to a human rater, compared to learning from 350, 700, or 1400 synthetic queries, as well as to RL learning from the real reward.”Page 6
-
11
Method
In the robotics results, synthetic and RL curves average 5 runs, but each real-human-feedback curve is a single run. Reacher and Cheetah feedback was given by an author; the other tasks used contractors unfamiliar with the environments and the algorithm.
“All curves are the average of 5 runs, except for the real human feedback, which is a single run, and each point is the average reward over five consecutive batches.”Page 7
“For Reacher and Cheetah feedback was provided by an author due to time constraints. For all other tasks, feedback was provided by contractors unfamiliar with the environments and with our algorithm.”Page 7
-
12
Observed result
On the simulated robotics tasks, 700 labels nearly matched RL on all tasks; training with learned reward functions tended to be less stable and higher variance, with comparable mean performance.
“With 700 labels we are able to nearly match reinforcement learning on all of these tasks. Training with learned reward functions tends to be less stable and higher variance, while having a comparable mean performance.”Page 7
-
13
Observed result
On the robotics tasks, real human feedback was typically only slightly less effective than synthetic feedback, ranging by task from half as efficient as ground-truth feedback to equally efficient.
“Real human feedback is typically only slightly less effective than the synthetic feedback; depending on the task human feedback ranged from being half as efficient as ground truth feedback to being equally efficient.”Page 7
-
14
Observed result
On Atari the method had more difficulty matching RL but showed substantial learning on most games and matched or exceeded RL on some. On BeamRider and Pong synthetic labels matched or came close to RL with only 3,300 labels; on SpaceInvaders and Breakout synthetic feedback never matched RL, though the agent improved substantially.
“Our method has more difficulty matching RL in these challenging environments, but nevertheless it displays substantial learning on most of them and matches or even exceeds RL on some.”Page 7
“on BeamRider and Pong, synthetic labels match or come close to RL even with only 3,300 such labels.”Page 7
“On SpaceInvaders and Breakout synthetic feedback never matches RL, but nevertheless the agent improves substantially”Page 7
-
15
Observed result
On Qbert the method failed to learn to beat the first level with real human feedback, which the authors suggest may be because short Qbert clips are confusing and hard to evaluate.
“On Qbert, our method fails to learn to beat the first level with real human feedback; this may be because short clips in Qbert can be confusing and difficult to evaluate.”Page 8
-
16
Observed result
Using the same parameters, the method learned novel behaviours with no available reward function. The Hopper robot learned to consistently do a backflip, land upright and repeat, trained with 900 queries in less than an hour.
“the ultimate purpose of human interaction is to solve tasks for which no reward function is available.”Page 8
“This behavior was trained using 900 queries in less than an hour. The agent learns to consistently perform a backflip, land upright, and repeat.”Page 8
-
17
Observed result
The Half-Cheetah robot learned to move forward standing on one leg with 800 queries in under an hour, and an Enduro agent learned to keep alongside other cars with roughly 1,300 queries and 4 million frames, though it got confused by background changes. These novel behaviours used feedback from the authors.
“The Half-Cheetah robot moving forward while standing on one leg. This behavior was trained using 800 queries in under an hour.”Page 9
“This was trained with roughly 1,300 queries and 4 million frames of interaction with the environment; the agent learns to stay almost exactly even with other moving cars for a substantial fraction of the episode, although it gets confused by changes in background.”Page 9
“Videos of these behaviors can be found at this link. These behaviors were trained using feedback from the authors.”Page 9
-
18
Observed result
Offline reward predictor training performed poorly. Because the states the agent visits shift during training, the predictor captured only part of the true reward, and maximising this partial reward led to bizarre behaviour that is undesirable under the true reward; on Pong, offline training sometimes made the agent avoid losing points but not score, giving extremely long repeated volleys.
“due to the nonstationarity of the occupancy distribution, the predictor captures only part of the true reward, and maximizing this partial reward can lead to bizarre behavior that is undesirable as measured by the true reward”Page 9
“For instance, on Pong offline training sometimes leads our agent to avoid losing points but not to score points; this can result in extremely long volleys”Page 9
-
19
Authors' interpretation
The authors conclude that human feedback in general needs to be intertwined with RL learning rather than provided statically.
“This type of behavior demonstrates that in general human feedback needs to be intertwined with RL learning rather than provided statically.”Page 10
References
- Deep reinforcement learning from human preferences — the original paper