Reinforcement learning (RL) has surged in popularity over the past few months, largely thanks to Large\u202fReasoning\u202fModels\u202f(LRMs) and test\u2011time scaling techniques. A solid understanding of RL fundamentals often makes the difference between a model that merely trains and one that converges stably and efficiently when applied to LLMs.<\/p>\n\n\n\n
To refresh my own knowledge, I\u2019ve been revisiting the basics and prototyping a few canonical algorithms. One of my resources is Prof.\u202fErnest\u202fRyu\u2019s UCLA YouTube course<\/a>\u2014it leans theoretical, but does a great job unpacking core ideas such as value functions, advantage estimates, policy gradients, Generalized Advantage Estimation (GAE), Trust Region Policy Optimization (TRPO), Proximal Policy Optimization (PPO), and more.<\/p>\n\n\n\n To ground the theory, I implemented several algorithms in Google\u202fColab using Gym environments. The code is in my this github repo<\/a>, codes are kept deliberately pure\u2011Python and PyTorch for clarity. Below is a quick look at what each notebook covers:<\/p>\n\n\n\n Please note that LRMs primarily rely on PPO and its variants (e.g.,\u202fGRPO). PPO can leverage a strong pre\u2011trained policy\u2014in the LRM setting, the language model\u2019s next\u2011token predictor. By contrast, DQN does not learn a separate policy network; instead, its policy is the simple rule at=arg\u2061max_at\u2061 Q(st,at). Consequently, DQN cannot benefit from a pre\u2011trained policy in the same way.<\/p>\n","protected":false},"excerpt":{"rendered":" Reinforcement learning (RL) has surged in popularity over the past few months, largely thanks to Large\u202fReasoning\u202fModels\u202f(LRMs) and test\u2011time scaling techniques. A solid understanding of RL fundamentals often makes the difference between a model that merely trains and one that converges stably and efficiently when applied to LLMs. To refresh my own knowledge, I\u2019ve been revisiting … Read more<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[17,16],"class_list":["post-86","post","type-post","status-publish","format-standard","hentry","category-uncategorized","tag-reinforcement-learning","tag-technical-notes"],"_links":{"self":[{"href":"https:\/\/firooz.us\/blog\/index.php\/wp-json\/wp\/v2\/posts\/86","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/firooz.us\/blog\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/firooz.us\/blog\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/firooz.us\/blog\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/firooz.us\/blog\/index.php\/wp-json\/wp\/v2\/comments?post=86"}],"version-history":[{"count":1,"href":"https:\/\/firooz.us\/blog\/index.php\/wp-json\/wp\/v2\/posts\/86\/revisions"}],"predecessor-version":[{"id":89,"href":"https:\/\/firooz.us\/blog\/index.php\/wp-json\/wp\/v2\/posts\/86\/revisions\/89"}],"wp:attachment":[{"href":"https:\/\/firooz.us\/blog\/index.php\/wp-json\/wp\/v2\/media?parent=86"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/firooz.us\/blog\/index.php\/wp-json\/wp\/v2\/categories?post=86"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/firooz.us\/blog\/index.php\/wp-json\/wp\/v2\/tags?post=86"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}RL_Basics_CartPole_REINFORCE.ipynb<\/code><\/strong><\/p>\n\n\n\n\n
CartPole\u2011v1<\/code><\/li>\n\n\n\n\n
\n
\n\n\n\nRL_Sparse_reward_Actor_Critic_and_PPO.ipynb<\/code><\/strong><\/p>\n\n\n\n\n
FrozenLake\u2011v1<\/code> (sparse\u2011reward setting)<\/li>\n\n\n\n\n
\n
\n\n\n\nRL_DQN.ipynb<\/code><\/strong><\/p>\n\n\n\n\n
LunarLander\u2011v2<\/code><\/li>\n\n\n\n\n
\n