Reinforcement Learning from Human Feedback explained with math derivations and the PyTorch code.
In this video, I will explain Reinforcement Learning from Human Feedback (RLHF) which is used to align, among others, models like ChatGPT. I will start by introducing how Language Models work and what we mean by AI alignment. In the second part of the video, I will derive from first principles the Policy Gradient Optimization algorithm, by explaining also the problems with the gradient calculation. I will describe the techniques used to reduce the variance of the estimator (by introducing the baseline) and how Off-Policy learning can make the training tractable.
I will also describe how to build the reward model and explain the loss function of the reward model.
To calculate the gradient of the policy, we need to calculate the log probabilities of the state-action pairs (the trajectories), the value function and the rewards, and the advantage terms (through Generalized Advantage Estimation): I will explain visually every step.
After explaining Gradient Policy Optimization, I will introduce the Proximal Policy Optimization algorithm and its loss function, explaining all the details, including the loss of the value head and the entropy.
In the last part of the video, I go through the implementation of RLHF/PPO, explaining line-by-line the entire process.
For every mathematical formula, I will always given a visual intuition to help those who lack the mathematical background.
PPO paper: Schulman, J., Wolski, F., Dhariwal, P., Radford, A. and Klimov, O., 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. - https://arxiv.org/abs/1707.06347
InstructGPT paper: Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A. and Schulman, J., 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35, pp.27730-27744. - https://arxiv.org/abs/2203.02155
Generalized Advantage Estimation paper: Schulman, J., Moritz, P., Levine, S., Jordan, M. and Abbeel, P., 2015. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438. - https://arxiv.org/abs/1506.02438
Slides PDF and commented code: https://github.com/hkproj/rlhf-ppo
Chapters
00:00:00 - Introduction
00:03:52 - Intro to Language Models
00:05:53 - AI Alignment
00:06:48 - Intro to RL
00:09:44 - RL for Language Models
00:11:01 - Reward model
00:20:39 - Trajectories (RL)
00:29:33 - Trajectories (Language Models)
00:31:29 - Policy Gradient Optimization
00:41:36 - REINFORCE algorithm
00:44:08 - REINFORCE algorithm (Language Models)
00:45:15 - Calculating the log probabilities
00:49:15 - Calculating the rewards
00:50:42 - Problems with Gradient Policy Optimization: variance
00:56:00 - Rewards to go
00:59:19 - Baseline
01:02:49 - Value function estimation
01:04:30 - Advantage function
01:10:54 - Generalized Advantage Estimation
01:19:50 - Advantage function (Language Models)
01:21:59 - Problems with Gradient Policy Optimization: sampling
01:24:08 - Importance Sampling
01:27:56 - Off-Policy Learning
01:33:02 - Proximal Policy Optimization (loss)
01:40:59 - Reward hacking (KL divergence)
01:43:56 - Code walkthrough
02:13:26 - Conclusion
In this video, I will explain Reinforcement Learning from Human Feedback (RLHF) which is used to align, among others, models like ChatGPT. I will start by introducing how Language Models work and what we mean by AI alignment. In the second part of the video, I will derive from first principles the Policy Gradient Optimization algorithm, by explaining also the problems with the gradient calculation. I will describe the techniques used to reduce the variance of the estimator (by introducing the baseline) and how Off-Policy learning can make the training tractable.
I will also describe how to build the reward model and explain the loss function of the reward model.
To calculate the gradient of the policy, we need to calculate the log probabilities of the state-action pairs (the trajectories), the value function and the rewards, and the advantage terms (through Generalized Advantage Estimation): I will explain visually every step.
After explaining Gradient Policy Optimization, I will introduce the Proximal Policy Optimization algorithm and its loss function, explaining all the details, including the loss of the value head and the entropy.
In the last part of the video, I go through the implementation of RLHF/PPO, explaining line-by-line the entire process.
For every mathematical formula, I will always given a visual intuition to help those who lack the mathematical background.
PPO paper: Schulman, J., Wolski, F., Dhariwal, P., Radford, A. and Klimov, O., 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. – https://arxiv.org/abs/1707.06347
InstructGPT paper: Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A. and Schulman, J., 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35, pp.27730-27744. – https://arxiv.org/abs/2203.02155
Generalized Advantage Estimation paper: Schulman, J., Moritz, P., Levine, S., Jordan, M. and Abbeel, P., 2015. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438. – https://arxiv.org/abs/1506.02438
Slides PDF and commented code: https://github.com/hkproj/rlhf-ppo
Chapters
00:00:00 – Introduction
00:03:52 – Intro to Language Models
00:05:53 – AI Alignment
00:06:48 – Intro to RL
00:09:44 – RL for Language Models
00:11:01 – Reward model
00:20:39 – Trajectories (RL)
00:29:33 – Trajectories (Language Models)
00:31:29 – Policy Gradient Optimization
00:41:36 – REINFORCE algorithm
00:44:08 – REINFORCE algorithm (Language Models)
00:45:15 – Calculating the log probabilities
00:49:15 – Calculating the rewards
00:50:42 – Problems with Gradient Policy Optimization: variance
00:56:00 – Rewards to go
00:59:19 – Baseline
01:02:49 – Value function estimation
01:04:30 – Advantage function
01:10:54 – Generalized Advantage Estimation
01:19:50 – Advantage function (Language Models)
01:21:59 – Problems with Gradient Policy Optimization: sampling
01:24:08 – Importance Sampling
01:27:56 – Off-Policy Learning
01:33:02 – Proximal Policy Optimization (loss)
01:40:59 – Reward hacking (KL divergence)
01:43:56 – Code walkthrough
02:13:26 – Conclusion
