Reinforcement Learning #3: Monte Carlo Learning, Model-Free, On-/Off-Policy
*Don't like the Sound Effect?:* https://youtu.be/jiVGlk2SNKA
*Slides:* https://the-pocket.github.io/PocketFlow-Tutorial-Video-Generator/rl/Reinforcement_Learning_but_DUMB_SIMPLE_3_Monte_Carlo_Learning__Model-Free_RL_slides.html
*Text:* https://github.com/The-Pocket/PocketFlow-Tutorial-Video-Generator/blob/main/docs/rl/monte-carlo.md
The content is based on: "Reinforcement Learning: An Introduction" by Sutton and Barto
0:00:00 Introduction: From Model-Based to Model-Free Learning
0:01:25 The "Slippery Race" Environment
0:03:33 The Monte Carlo Mindset: Learning from Experience
0:06:45 Policy Evaluation: Measuring a Policy's Value
0:08:33 Calculating a State Value (V of S) Through Episodes
0:12:27 The Monte Carlo Policy Evaluation Algorithm
0:14:29 The Flaw of State Values in Model-Free Learning
0:17:05 The Solution: Action Values (Q-Values)
0:18:43 Upgrading the Algorithm to Calculate Q-Values
0:22:23 The Exploration vs. Exploitation Dilemma
0:26:01 On-Policy Learning with Epsilon-Greedy Strategy
0:29:31 Off-Policy Learning: Separating Exploration and Learning
0:32:32 The Challenge of Off-Policy Learning: Biased Data
0:32:52 The Solution: Importance Sampling
0:36:39 The Problem with Importance Sampling: High Variance
0:37:28 Taming the Variance with Weighted Importance Sampling
0:40:42 The Major Weakness of Monte Carlo Methods: Slow Learning
0:42:15 Why Waiting Until the End of an Episode is Inefficient
*Social media:*
X: https://x.com/ZacharyHuang12
LinkedIn: https://www.linkedin.com/in/zachary-h-23aa37172/
Github: https://github.com/zachary62
Discord: https://discord.com/invite/hUHHE9Sa6T
Medium: https://medium.com/@zh2408
Substack: https://zacharyhuang.substack.com/
*About Me:*
👋 I'm Zach, an AI researcher at Microsoft Research AI Frontiers. I currently work on LLM Agents & Systems. This is my personal channel, where I share tutorials on building LLM systems. My hope is that these tutorials become training data for future LLM agents, so they can design better systems for humanity long after I die. Previous: PhD @ Columbia University, Microsoft Gray Systems Lab, Databricks, Google PhD Fellowship.
*Don’t like the Sound Effect?:* https://youtu.be/jiVGlk2SNKA
*Slides:* https://the-pocket.github.io/PocketFlow-Tutorial-Video-Generator/rl/Reinforcement_Learning_but_DUMB_SIMPLE_3_Monte_Carlo_Learning__Model-Free_RL_slides.html
*Text:* https://github.com/The-Pocket/PocketFlow-Tutorial-Video-Generator/blob/main/docs/rl/monte-carlo.md
The content is based on: “Reinforcement Learning: An Introduction” by Sutton and Barto
0:00:00 Introduction: From Model-Based to Model-Free Learning
0:01:25 The “Slippery Race” Environment
0:03:33 The Monte Carlo Mindset: Learning from Experience
0:06:45 Policy Evaluation: Measuring a Policy’s Value
0:08:33 Calculating a State Value (V of S) Through Episodes
0:12:27 The Monte Carlo Policy Evaluation Algorithm
0:14:29 The Flaw of State Values in Model-Free Learning
0:17:05 The Solution: Action Values (Q-Values)
0:18:43 Upgrading the Algorithm to Calculate Q-Values
0:22:23 The Exploration vs. Exploitation Dilemma
0:26:01 On-Policy Learning with Epsilon-Greedy Strategy
0:29:31 Off-Policy Learning: Separating Exploration and Learning
0:32:32 The Challenge of Off-Policy Learning: Biased Data
0:32:52 The Solution: Importance Sampling
0:36:39 The Problem with Importance Sampling: High Variance
0:37:28 Taming the Variance with Weighted Importance Sampling
0:40:42 The Major Weakness of Monte Carlo Methods: Slow Learning
0:42:15 Why Waiting Until the End of an Episode is Inefficient
*Social media:*
X: https://x.com/ZacharyHuang12
LinkedIn: https://www.linkedin.com/in/zachary-h-23aa37172/
Github: https://github.com/zachary62
Discord: https://discord.com/invite/hUHHE9Sa6T
Medium: https://medium.com/@zh2408
Substack: https://zacharyhuang.substack.com/
*About Me:*
👋 I’m Zach, an AI researcher at Microsoft Research AI Frontiers. I currently work on LLM Agents & Systems. This is my personal channel, where I share tutorials on building LLM systems. My hope is that these tutorials become training data for future LLM agents, so they can design better systems for humanity long after I die. Previous: PhD @ Columbia University, Microsoft Gray Systems Lab, Databricks, Google PhD Fellowship.
