{
  "video_id": "XMnxKGVnEUc",
  "channel_slug": "umarjamilai",
  "channel_handle": "umarjamilai",
  "title": "Paper: DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning",
  "duration_seconds": 4777.0,
  "url": "https://www.youtube.com/watch?v=XMnxKGVnEUc",
  "upload_date": "",
  "transcript": "perfect wonderful uh do you also are you\nalso humy\ninside yes yeah wonderful so we can have\nlike three streams or two streams only I\ndon't know how that\nworks okay let's start\nguys\num\nso the goal of today's paper reading is\nto go through the Deep seek R1 paper and\nuh\nwhat we will be seeing is first of all\nthe biggest difficulty people have in\norder to um understand this paper is the\nreinforcement learning part uh what why\nit is difficult because we are they use\nanother algorithm called the grpo and uh\nthe goal of this initial part of this\npaper reading is actually to give you\nthe background knowledge that is is\nneeded to understand the\npaper so uh I will be using the slides\nthat I used for making my video on\nreinforcement learning from Human\nfeedback so to understand what is the\nconnection between language models and\nuh reinforcement learning so let's go to\nthat slides so let's review very fast\nvery very fast language models so as you\nknow language models are generative\nmodels uh which only have one simple\nfunction which only have one simple\nobjective which is to uh tell us what is\nthe next likely token given an input\nprompt so if we are given for example\nthe input prompt Shanghai is a city in\nand we feed it to the language model the\nlanguage model will generate a\nprobability distribution over what it\nthinks me means the\n[Music]\num what it thinks is the next likely\ntoken that is coherent with the prompt\nthat we have given the language model\nand usually we sample the to for example\nif we fit this prompt to language model\nit will give us that maybe the next\nlikely token is the word China or\nBeijing or cat or pizza and then we\nchoose what we believe is the most\nlikely based on its probability\ndistribtion on probability score score\num then we take the this word we put it\nback in the prompt and we ask again the\nlanguage model what is the next next\nword and we do this Pro this job\niteratively to generate a full uh\ntext uh for example to generate the um\nthe response to where is shangai we\nfirst ask the language model where\nshangai it will tell us okay the best\nnext likely word is shahai then we put\nit back in the input of the language\nmodel and it will tell the next likely\ntoken is e etc etc I'm also making a\nsimplified assumption here that says\nthat each token is a word and each word\nis a token this is not the case in\nlanguage models but for our explanation\nwe will think like in this\nway okay now we know what is the\nlanguage model what is reinforcement\nlearning we go very very simple through\nwhat is reinforcement learning and then\nwe find the connection between language\nmodels and reinforcement learning now\nreinforcement learning is an area of\nartificial intelligence I I don't\nremember if it belongs to machine\nlearning in particular but it's an area\nof artificial intelligence that is uh\ntasked with optimizing the behavior of\nan agent and the behavior of an agent is\ncalled a\npolicy uh which is the decision making\nof an agent in such a way that the agent\nperforms actions that maximize the\nreward it gets from performing these\nactions in an environment for example\nfor example I have a cat so uh this cat\nlikes to eat meat like most cats and the\ncat is in my house and the cat will be\nconsidered of reinforcement learning\nagent the cat can make some decisions on\nhow it wants to move in the house and\nthis will be the policy of the cat so uh\nthe policy tell the cat if the cat\nshould move up down left or right in the\nhouse now this is the my house you\ncannot see the border so I will draw\nthem because I don't know why\nthey the borders are not drawn here and\nyou should think of this environment as\nbeing a grid environment made up of\ncells so like the\nfollowing like\nthis\num what is the goal of the cat the goal\nof the cat is to arrive to the meat so\nthe the cat can at each position in the\nhouse it can make some uh choices some\nperform some actions we say technically\nuh and the actions that the cat can can\nchoose at each position in the the house\nis move up down left or right what we\nwant we want to make sure that the cat\nlearns to perform the serious of actions\nthat lead him with very uh likely to the\nmeat while avoiding the things that the\ncat is scared about which is the broom\nand the best tub because no cat likes to\ntake\nshower so we designed first of all a\nreward system for this reinforcement\nlearning agent because we want to train\nthis reinforcement learning agent to\nchoose which actions to perform in each\nposition in the house based on some\nreward so one reward that we could do uh\none reward model could be this one for\nexample if the cat moves to an empty\ncell it receives a reward of zero if the\ncat moves to the broom it receives a\nreward of minus one if it moves to the\nbtub it receives a reward of minus 10\nhowever if after performing a series of\nactions the cat arrives to the meat then\nthe cat receives a big reward of plus\nthe decision making of this cat is\ngoverned by a model that we will call\nthe policy of this cat and the goal of\nthe policy is to choose an action given\nthe current state and the action that\nthe cat can choose is stochastic it\nmeans that this policy gives us a\ndistribution over all the possible\naction that the cat can take so if if we\nhave imagine we have a very optimized\npolicy if the cat is here the good\npolicy should tell us that with with\nvery high probability score we should\nmove down because that's one way to\nmaximize the reward and with very low\nprobability we should move left because\nthat will take us with to towards the\nthe the\nbtub um the another thing for example\nthat this policy should do is if the cat\nis here for example it should not move\nright so the probability Associated to\nthe action move right should be low and\nmaybe the probability associated with\nthe action move down should be a little\nhigher this is what a policy is now what\nis the connection between language\nmodels and reinforcement learning so\nfirst of all what is the goal in\nreinforcement learning is to train this\npolicy so to train this decision making\nof this cat of this agent in order to\nchoose the proper actions at every\npossible state in the environment such\nthat it maximizes the reward that the\nagent can\nget uh so a good policy for the cat\nwould be a policy that always lead the\ncat to the meat no matter where the cat\nis now let's go let's connect the\nreinforcement learning with language\nmodels so language model is also kind of\na policy because the language model\nevery time you feed a prompt to the\nlanguage model the language model has to\nchoose an action to perform which is\nwhat token should come after this\nprompt um so in this case we talk about\nState and action the state is the state\nin which the reinforcement agent is in\nthe case of the cat is the position of\nthe cat inside of the house in case of\nthe language model the state is The\nPrompt itself that you feed and the\naction is the distribution over all the\nnext token that the language model can\nchoose\nfrom in the case of language model so we\nalso want to train the language model to\nperform its action actions or to choose\nthe next tokens in a particular way\naccording to some reward that we uh that\nwe can build reward model that we can\nbuild in the case specifically case of\nre learning from Human feedback we want\nthe language model uh to generate text\nusing particular rules for example when\nwe do language model alignment we are\nfirst of all how language models are\ntrained uh usually we have a\npre-training part where we feed a lot of\ninformation to the language model so we\nthrow a lot of data like the entire\nWikipedia the entire web the entire I\ndon't know stock over flow and Lead code\neverything and the language model learns\nhow to kind of uh the structure of the\nlanguage it learns a little bit of\nChinese a little bit of English a little\nbit of Japanese because we throw every\ndata possible that we have at the\nlanguage model then we do a little bit\nof fine-tuning so we train the language\nmodel to generate high quality data so\nwe increase the likelihood of generating\nhigh quality outputs instead of just\nthrowing whatever is on the internet but\nthen we do also an alignment part in\nwhich we want the language model to\nfollow instructions so we want the\nlanguage model to adhere to some\nstandards for example what makes the\nlanguage model conversational is the\nconversation um is the instruction fine\ntuning which means that we train the\nlanguage model to follow a particular\nformat so always for example greet the\nuser always um be helpful always uh\nnever use curse word etc etc etc and\nthis job is done through the\nreinforcement learning from Human\nfeedback which includes many kind of\nalgorithm like P DPO Etc and grpo is one\nof\nthem what we do usually in language\nmodels to train the language model to\nfollow instructions is we\num generate a data set of uh\ninstructions uh in which we we first gen\nhave some list of questions and then we\nask the language model to generate some\nanswers and then we ask some\nprofessional annotators to choose which\nanswer they would like the language\nmodel to generate more and which one\nthey don't like the language model to\ngenerate and the the goal of\nreinforcement learning from Human\nfeedback is to make sure that the\nlanguage model uh will generate more\nanswers like the ones that are chosen by\nthe annotators and less likely to\ngenerate answers that are not chosen by\nthe\nannotators this um this this is uh\ncalled the reward model of the of the\nrefer learning from Human\nfeedback okay now that we have\nunderstood a little bit the connection\nbetween language models and the\nreinforcement learning framework Let's\nuh move on to the paper\nI just want to do a little review of\nwhat we have seen so far so now we know\nwhat are language models they are\nlanguage they are models that generate\nthe probability over what is the next\nlikely what is\nthe next token the probability\ndistribution of what is the what should\nbe the next token based on the input we\nknow what is reinforcement learning\nwhich is a framework for training the\npolicy of an agent in order to choose\nactions that maximize its reward what is\nthe connection between reinforcement\nlearning and language model is that the\nlanguage model itself is a policy\nbecause it makes decisions it takes\nactions in choosing what is the next\ntoken and we want the language model to\nchoose tokens so the next token in such\na way that it follows some standards\nwhich are according to a data set of\npreferences that we usually\nbuild um this data set of preferences is\nconverted into a reward model but we\nwill not be covering the reward model\nfor now at least so now let's go to the\ndeeps paper now in the D in the deeps R1\npaper what they do is they start with a\na pre-rain model they they use the deeps\nV3 base model which I believe is a 600\nbillion uh parameter\nmodel and then they want this DP this\nmodel to uh perform better at\nreasoning what does it mean to perform\nbetter at reasoning it we want the\nlanguage model to find a way to solve\ncomplex Problems by bring breaking them\ninto smaller uh steps that can be um\nthat are easier for the language model\nto manage and the way they do it is also\nthrough reinforcement\nlearning uh let's go to the paper so\nlet's go here let's go here let's go\nhere okay first of all they say that uh\nin this section we explor the so let me\nuse the in this section we explore the\npotential of language models to develop\nreasoning capabilities without any\nsupervised data focusing on their self-\nevolution through a pure reinforcement\nlearning process so what is supervised\ndata when we train language models we\nalso try to build some very high quality\ndata set of what the language model\nshould be generating because as we said\nbefore we have like multiple stages of\ntraining one is the pre-training in\nwhich we just throw random data from the\nweb to the language model but then we\nwant the language model to generate high\nquality data so we have this kind of the\nsupervised fine-tuning part and they\nskip this part here they just take the\nbase model um and then they want the\nbase model to develop the reasoning\ncapability just by using uh\nreinforcement learning which means that\nwe want to incentivize the language\nmodel through a reward system to develop\nby itself what is the sequence of token\nthat should lead to so for the language\nmodel to acquire as much reward as\npossible it's like I take my cat and and\nI want the cat to solve math problems\nand I what I can do I can just play with\nhow many biscuit I can give to the cat\nso if I build my um reward model in such\na way that the cat is\nincentivized to solve math problems then\nby the because the cat wants to get the\nbiscuit the cat will develop whatever\nskill it needs to develop in order to\nget maximize the number of biscuit it\ngets now of course the cat will never\ndevelop it because the underlying kind\nof uh lacks the capability of learning\ncertain things but that's not the\nproblem in uh language models because\nbig language models have a lot of uh\ncapabilities in developing novel\nskills now the algorithm that they use\nin um in deeps R1 is called the grpo\nalgorithm if you look at my video on\nenforcement learning from F feedback\nhistorically we have always used the PO\nalgorithm more recently the DPO algorith\nthere are also other algorithms like the\nO RPO what they use here is called the\ngrpo algorithm and it's a very similar\nto the PO but slightly different and we\nwill see\nhow um let's see what does the grpo\nalgorithm does well uh as we saw before\nuh when we do the uh when we do uh\nreinforcement learning on a language\nmodel we have a data set of preferences\nso we have some questions then we ask\nthe language model to generate multiple\nanswers and then we ask annotators to\nchoose which answer they like and then\nwe train a language a reward model that\num that gives the signal to the language\nmodel to understand which um if the\nanswer that the language model is\ngenerating is uh good or\nbad in this case with the grpo we have\nthe following objective now if you have\nnever seen clip or if you sorry if you\nhave never seen po this sounds quite\nscary but let's try to break it down uh\nstep by step what are we doing here is\nwe want to optimize a policy so the\npolicy is the language model itself and\nthe policy is always denoted with the\nletter Pi like the Greek letter Pi so\nthis Pi of theta is the policy so it is\nthe language model that we are trying to\noptimize we want this policy to be\ntrained to maximize the following\nobjective what is this objective this\nobjective is saying that if I have a\nlist of questions that belong to some\ndatabase of questions\nand uh we sample some output from our\npolicy using these\nquestions then based on some reward that\nthis output um that this output of the\nlanguage model get from our reward\nsystem we should train the language\nmodel to give more weight to those\nactions that result in good reward and\nto give less weights so are less likely\nthe our language model should be less\nlikely to take those actions that lead\nto bad\nreward and the way we do it is as\nfollows so um we take basically what we\nare doing is okay here you see all the\npolicy and the new policy for now let's\nignore that that's because we do\nsomething called offline learning so we\nwill not be covering that at least now\nfor now what we want to do ignore the\nthe the the uh denominator here so the\npi old just concentrate on the um on the\nterm Pi what we are saying is that we\ngenerate the log probabilities of the\noutput so what is the log probability of\nthe output let's do it step by step\nactually\nokay imagine we have the following\nquestion uh for example imagine we ask\nthe language model where is Shanghai\nso and the language model generates\nbecause as we saw we sample a few\nquestions from a database of questions\nand then language mod and then we\ngenerate multiple a outputs using this\nthis question using our language model\nso these outputs are called o and there\nare G of them this is the group in\nGPO maybe the language model the first\ntime will generate let's say uh Shanghai\nis in\nChina so I let me just write Shanghai is\nin\nChina another output could be for\nexample the sky is blue\nand another output could be um Shanghai\nis\nbeautiful imagine we have some magic uh\nreward system that\nassigns um or imagine that a reward\nsystem is a human\nbeing this human being will very likely\nto give a very high reward to this\nanswer zero reward to this answer and\nmaybe not completely zero but nearly\nzero\nscore to this answer why because this at\nleast talks about Shanghai this doesn't\ntalk about anything related to Shanghai\nand this actually uh answers the\nquestions now when we have a language\nmodel we have a question and the\ngenerated answer we have what is known\nas a trajectory a trajectory is a list\nof actions that the language model has\ntaken why because the language model was\ngiven this question as input and the\nlanguage model took an action chose an\naction to generate the first token which\nis shanhai then this Shanghai was put\nback into the language model and then\nthe language model choose another action\nwhich is the token is and then this is\nwas put back into the language model and\nthen the language model chose another\naction which is in etc etc this is a\ntrajectory at each step of the\ngeneration process the language model\nchose an action based on the probability\ndistribution that it generated actually\nit's not the language model that chooses\nis it's our sampling strategy that\nchooses the particular token so usually\nwe can use the gridy strategy or the top\nP strategy or\nwhatever so uh at each step we have\nchosen some action based on the\ndistribution of the language uh that the\nlanguage model generated for each\nstate so we have a log probability a\nprobability associated with the word\nShanghai un conditioned on the question\nwhere is Shanghai we have a probability\nassociated with the word is conditioned\non the input where is Shanghai question\nmark Shanghai then we have a probability\nassociated with the word in conditioned\non the input where is shangai question\nmark Shanghai is blah blah blah Etc ET\nso we have a list of probabilities what\nwe are doing here is that we want the um\nthis is the product of all the log\nprobabilities that are generated at each\nstep of the particular output oi here\nwhich is maybe the first answer here\nthen we ask the language model again the\nsame question and the language model\nwill come up with another uh output and\nthis output will also have associated\nwith it a list of log probabilities and\nthese are the log probability that you\nsee here each log probability\nfurthermore is weighted by an advantage\nterm the advantage term is basically\ntelling me how better is choosing a\nparticular token given a particular\ninput over all the tokens that are\navailable for example imagine we have\nthe input where is Shanghai is it better\nto choose the word Shanghai or is it\nbetter to as the first uh word of the\nact the response or the word Pizza I\nbelieve it's better to choose the word\nShanghai so the advantage of choosing\nthe word Shanghai would result in a\nbetter long-term reward for the policy\nso for the language model because it\nwill result in a good answer so it will\nresult in a high reward from our reward\nmodel for now we just have model the\nreward model as a human being who tells\nthat the language model okay this is a\ngood answer this is a bad answer but\nactually later we will see that the\nreward is actually also a language model\nand in the case of grpo they actually\nused in the case of the dpse R1 they\nused actually a rule based reward\nmodel so let's go back we we have a list\nof questions we generate multiple\nanswers with each of these questions\nusing our language model and then we for\neach of these uh answers we have the log\nprobabilities associated with this\nanswer which is just the product of all\nthe probabilities of choosing that\nparticular token given that particular\ninput we wait each of this log\nprobability by an advantage term uh\nwhich basically tells us how good is\nchoosing this part ular to token over\nall the other that are available for\nthis particular input and we train our\nlanguage model to maximize this\nobjective let's see what does it mean to\nmaximize this objective and now let's\nalso see what is this old here uh the\nlanguage model that we will be training\nis our um deep seek base right so at the\nbeginning suppose that this P old is the\nbase version of\ndsek uh what we do is basically we want\nto refine it iteratively by generate\nkeep generating um output from it and\nthen through the reward model we want to\ntell it okay this was a good um this was\na good output so do more of this or this\nwas a bad output so do less of this this\nis one of the advantage of using refer\nlearning because if you do supervis fine\ntuning you're just telling the language\nmodel to I want this so generate this if\nyou're doing a reinforcement learning\nyou have the ability to tell the\nlanguage model I want more of this and I\nwant less of this what we are doing is\nthe the language model at each\niteration um the language model should\nbe optimized in the following way if the\nlanguage model at the current interation\nis giving more probability more\nlikelihood to generate a a response that\nresulted in a good reward and the\nadvantage term will be high in that\ncase uh then this policy is this\nobjective is telling the language model\ndo more of this however if right now the\nlanguage model is giving less\nprobability to an action that also\nresulted in a low than bad reward and\nthe advantage term will be negative in\nthat case then by optimizing this\nobjective here by maximizing the\nobjective here the language model will\nlearn to do less of that uh I know that\nI didn't explain very well well the pi\nTheta and the pi old Theta I believe I\ncan do that later when if we have\ntime by talking about uh offline um\noffline learning in the case of uh in\nthe case of reinforcement learning\nmoreover in the grpo you find this kale\nDivergence term now for people who\nalready know what is the kale Divergence\nthat would be super easy but for people\nwho don't know basically the K\ndivergency is a way of measuring how to\ndistributions are different so how far\nthey\nare what we want is we want the language\nmodel to be fine-tuned to generate more\nof things that lead to a better reward\nto do less of things that result in low\nreward but at the same time we don't\nwant the language model to change too\nmuch Its Behavior for example imagine we\nhave a reward model that tells the\nlanguage model to be more polite uh what\nthe language model could do and imagine\nthat by saying being polite for example\nin case means that I always say thank\nyou right so what could happen is if we\ndon't enforce the K Divergence is that\nthe language model could just cheat and\nalways generates thank you thank you\nthank you thank you thank you thank you\nat every response like a list of thank\nyous because that results obviously in a\nhigh reward but the language model would\nstop doing its main job which is to\ngenerate something factual and useful so\nwe want the language model to change a\nlittle bit but to change so to be more\npolite but not just be just generate a\nbunch of thank you so change change but\nchange a little bit so that's why we add\nthe scale Divergence otherwise what the\nlanguage mod will do it will do what is\nknown as reward hacking which is it will\ntry to find a way to uh just learn what\nis a trick to get maximize its reward\nwithout actually being\nuseful um this is also for example if\nyou want a parallel like example it's\nlike you have a text code and it's very\ncomplex people will always find the way\nto cheat on it and if you have a text\ncode that is very simple made up of few\nrules then it's very unlikely that\npeople will be able to uh cheat on it so\num H so so you cannot do like uh in that\ncase you cannot\ncheat okay\num what do we miss here so first of all\nuh I didn't explain\nthe I didn't explain\nthe uh offline uh policy\nlearning and I didn't explain the clip\npart why we are clipping\nhere uh the clipping part basically we\nare saying if the language model is\ntrying to change its log\nprobabilities if the language model is\ntrying to change his log probabilities\nby being too confident about its change\nthen we don't want we we don't want to\nuh let the model be overly confident\nbasically what means that if the this is\nthe language model at the current\niteration and this you can think of it\nat the previous iteration in the\ntraining process uh if the language\nmodel at the current iteration is very\nconfident that by saying the word\nShanghai will result in a better\nreward even if it's a good choice We\ndon't want the land whe model to be\noverly confident so we clip this uh\nratio between the log probabilities up\nto 1 plus Epsilon or or in the lower\ncase in 1 minus\nEpsilon uh because if the language model\nwill choose the next word as a Shanghai\ngiven this question then okay we are\nlucky and it's um it's good but imagine\nthe language model is overly confident\nthen the next word is I don't know\ncoffee then we don't want the language\nmodel to make two big step we want the\nlanguage model to learn as slow as\npossible uh by choosing the accordingly\nthis Epsilon term which is something\nthat we can choose and this beta term\nokay um I believe I have covered a\nlittle bit of this so last last review\nguys so we are trying to optimize the\nlanguage model iteratively by telling it\nto make more of something that we want\nand to do less of what we don't want how\ndoes the model know what we want and\nwhat we don't want it's the reward that\nwe give it according to our reward model\nnow let's talk about the reward model\nthe reward model historically in po uh\nlet's go to the other here\nwas a model which was of the same\nstructure as the language model that we\nare trying to optimize in which we add a\nlinear layer on top that gives um a\nreward to each answer how can we assign\na numeric reward to a particular um\nanswer so if you remember when we talk\nabout BP I said that usually we start\nwith some questions then we generate a\nfew answers then we we ask some\nannotators to choose which answers we\nlike and they and to also tell us which\nanswers they don't like how do we\nconvert this data set of preferences\ninto a\nnumber we do that by training a language\nmodel which has the same architecture as\nthe policy that we are trying to train\njust a different head on top that\ninstead of generating the log\nprobabilities of the next token\ngenerates a numeric\nreward and uh we do it with what is\nknown as the Brad lary model which is\nbasically this loss here uh you don't\nhave to understand this law doesn't\nmatter it basically means that we want\nthe if we train a language model on this\nloss it will generate a very high reward\nfor the quest broadly Terry\nmodel\num so um if we uh train our language\nmodel on this loss it will basically\nresult in this head here this linear\nhead on top of the language model to\ngenerate a very high value for the\nquestions that for the answers that were\nchosen in the data set of preferences\nand a very low or low basically uh uh\nvalue for the answers that were not\nchosen by the professional\nannotators this is how it was done with\npo so we have an objective which\nincreases the likelihood of the things\nthat give High reward we also have a\nreward model which generates a numeric\nreward as a signal for the language\nmodel to understand what it should do\nmore and what it should do l Less in\ndeeps R1 they do something\ndifferent uh they instead of using a\nmodel as a reward they use a rule-based\nmodel system so they don't train a\nlanguage neural network to generate a\nnumber that gives a signal to the model\nto understand what uh what we like and\nwhat we don't like they used a\nrule-based system you can see\nhere and this rule based system is you\ncan do it for all the task that you can\nkind of verify so for example for the\nlead code problems they ask the how can\nyou check if the answer generated by the\nmodel is good well you just run it and\nif it performs if it first compiles and\nsecondly it runs in a predefined time\nlimit then it is a good answer doesn't\nmatter how it came to be if it's if it\nworks it's a good answer just like also\nfor example math some for most of math\nmath problems we do have the answer that\nwe expect the model to generate so we\ncan compare what is the actual generated\nanswer and um what is the we expect the\nmodel to generate so they create a rule\nbased um reward system in a way that\nthey ask the language model to generate\nsome output for a given problem and then\nthey can assign by just following rules\nso check if okay if it's a leite code\nproblem just run it and check if it runs\nokay good reward doesn't run okay zero\nuh and if it's a math problem they just\ntake the the output of the the moduel\ncompare it with the expected result and\nassign reward based on that so if it's\nthe answer matches what we expected good\nreward otherwise zero etc etc they also\nassign a reward for the for the language\nmodel if it formats the output in a\ncertain way uh by for example forcing\nthe they give a reward for the language\nmodel if it uses for example um if it\nfollows the format of putting all the uh\nthought process in the tags think and SL\nthink Etc so basically just with this\nthey train the language model so they\ntrain the language model to generate\nanswers and they reward these answers\nthrough a reward system which is B rule\nbased and they keep training it um and\nthe language models basically\nautomatically by itself learns to\ngenerate the um thought process that is\nnecessary to perform the the the tasks\nthat it is being asked so the language\nmodel um learns by itself by uh just\nwith reinforcement learning by with this\nreward system to uh generate the thought\nPro process that leads to the generating\nthe right code for the Le code problems\nto generate the right um thought process\nthat is necessary for solving me me\nproblems etc etc\netc uh let me check what else we need to\nknow from\n[Music]\nhere okay here they show some results uh\nI think the results you can check by\nyourself what is very interesting I\nthink it's this during the training of\nR1 just with reinforcement learning so\nas I as I want to remind you is they\ntook deeps V3 Bas and added this\nreinforcement learning step on top of it\nwhich is a massive reinforcement\nlearning step usually the alignment part\nis not so big and the more they find you\nthey they they run this re enforcement\nlearning um Step the step they they saw\nthat the language models\nautomatically learns to generate longer\nresponses because to solve problems you\nneed to generate a longer Chain of\nThought So the language model because of\nthe reward system learns that in order\nto get reward it is it should generate\nlonger responses so they didn't tell the\nlanguage model with supervis fine tuning\nto generate that kind of data with that\nparticular format with that kind of\nthought process just with reinforcement\nlearning with the right incentives the\nlanguage model learned to do that how\ndid it do that it learned that at a\nparticular input it should generate that\nparticular token which in the long term\nresults in a good reward so the long\nmodel the beauty of reinforcement\nlearning is that you not only only learn\nto do something that based on the\nimmediate reward that you get but also\non the long-term reward that you will\nget because sometimes the reward as you\ncan see here the reward model only\napplies when the entire answer is\nproduced so the model will only know the\nsignal that the model gets is only for\nthe entire output actually okay through\nthe advance term the signal is\npropagated back to each single token but\nokay we can skip that uh in the in the\nreinforcement learning actually the\nbeauty is that sometimes the reward you\nget is not for the single action you\ntake so in the case of my cat for\nexample let's say uh here so let's go\nback to my\ncat the cat will receive reward only\nafter taking many step that lead it to\nthe meat so only when the cat is here it\nwill know okay all this sequence of\nactions was a good choice but this\nsignal is propagated back to each single\naction in such a way that the cat when\nit will be here it will very likely to\nchoose I need to go down and less likely\nto choose I need to go right this also\nhappens in the case of language models\nin this case through the advantage term\nhere uh where is it through this\nAdvantage term here uh which is actually\ndone for each token in the case of okay\nnow we can go a little bit more\ntechnical\ndetails uh in the case of why they\nchoose grpo over po first of all well\nwith po basically this Advantage term\nhere uh to compute requires another\nfunction that is called the value\nfunction and this value function\nbasically to be computed requires the\ntraining of another model by using the\nadvantage term in grpo this Advantage\nterm is calculated without the value\nfunction but by the following formula\nhere which is basically just based on on\nthe rewards which is already uh given by\nthe reward model which we have which in\nthe case of the dpse R1 is Rule based so\nthey don't need to train this other uh\nmodel to generate the the value function\nwhich is adds more complexity to to the\nto the\nsystem okay so now we have seen what is\nreinforcement learning we have seen what\nis the connection between language\nmodels and reinforcement learning we\nhave seen a little bit what is the the\ngrpo\nobjective uh\num okay I believe let's actually do a\npoll guys do you want me to go deeper in\nthe grpo like let's explore uh exactly\nthis loss because actually the the the\nrest of the paper is uh okay we tried\njust reinforcement learning okay so\npeople like deep so let's go deep okay\nall right first of all the most\ninteresting thing about uh reinforcement\nlearning especially uh in the case of\npeople PPO andfor learn from Human\nfeedback is that is this thing called\nthe gradient policy optimization so\nlet's go back to the other slide and\nthen we go back to the DPO because the\nrest of the paper in DPO deeps is\nbasically they they took this um R1 and\nthen said okay instead of just doing\nreinforcement learning let's do maybe\nmultiple step of reinforcement learning\nand supervise fine tuning and then\nreinforcement learning again then Super\nand it leads to better outcomes but that\nis not technically difficult to\nunderstand I think because most of you\nalready kind of have backgrounds in this\nso if you're here it's because you kind\nof understand what we are talking about\nuh so let's do all the things that maybe\nsome people have difficulties with which\nis I believe this this part okay so\nlet's go deeper so um okay so let's go\nback to my\ncat and imagine I want to train my cat\nso I want to train my cat to um follow\nto to reach the meat so as we saw before\nmy cat is just a an agent with a policy\na policy is what it tells the cat what\naction to take given the position of the\ncat inside of the house you need to\nthink of this house as a grid so like\nlike the following I didn't you cannot\nsee the the grid lines because I don't\nknow my it's not showing but okay it\ndoesn't\nmatter what is the policy the policy as\nwe saw before is tells the cat what\naction to take given a particular\nposition what is our goal in\nreinforcement learning is to select a\npolicy that maximizes the reward that\nthe agent gets when using this policy so\nthis is the um the the objetive that if\nwe have we want to select among all the\npossible policies that we can can can\nhave the one that maximizes an objective\nwhat is this objective is the expected\nreward that we can get when using this\npolicy means that if I apply in the side\nof in the brain of my cat this policy\nwhich is the decision-making stuff then\nmy cat if this best policy will tell my\ncat to move down here and move right\nhere and move right right down down down\nand until it arrives to the meet this\nshould be the um the policy uh how do we\nactually uh train this policy uh we do\nwhat is um known as a um policy gradient\noptimization uh but before we understand\npolicy devation we need to understand a\nfew terms so uh we want to\num we want to first of all learn what is\nthe trajectory the trajectory is\nbasically a list of state and actions so\n[Music]\num okay yeah okay the trajectories are a\nlist of states of section so if the cat\nis here it's in the State uh let me draw\nthe lines otherwise it's too bad for you\nguys if you cannot see it uh here here\nhere\nhere okay this is the the cat is\ninitially here so this let's call it the\nstate number zero and the cat can choose\nsome action and let's call it the action\nnumber zero when the cat takes the\naction number zero it will arrive in a\nnew state so maybe the cat will arrive\nhere and it will become the state number\none of the cat and then the cat here we\ncan take another action according to its\npolicy let's call it action number one\nand which will lead the cat into a new\nstate so we call it s S2 which in which\nit will um in which it can take another\naction and let's call it action number\ntwo etc etc so the trajectory is a list\nof states and action that the agent can\ntake inside of the\nenvironment uh what we want is if we\ntake sample a trajectory according to\nour policy we want to maximize the\nreward that we get from each of the\npossible trajectories that we can\ntake um how do we do that uh let's do it\nhere so basically what we do is the\nfollowing we this is our\nobjective so if we can find as you know\nwhen we have in a deep learning what we\nare doing we are trying to either\nmaximize something or we are going to\nminimize something in the case of\num model training we usually always\nminimize a cost function in this case we\nwant to maximize the expected rewards\nwhen the agent acts according to this\npolicy this policy however is not just\nany policy it is a particular policy\nmade up of some parameters that we will\ncall\nTheta to give you a parallel on what is\nhappening here is the\nfollowing imagine you have a company and\nyou are like the CEO of the company and\nuh this company is made up of many\nactors and many functions and many\ndepartments so each of these um uh\nthings are let's say they are parameters\nof your company they Define your P\nbecause you can tune them and the the\ncompany function will change so how\npeople uh talk to each other how people\nwork how the Departments work how the\nlogistics work how the office Works Etc\nthey are all parameters of your company\nwhich Define the outcome of your company\nand imagine you want to maximize the\nprofit of your company so what you do\nyou learn to tune all of these\nparameters so you learn to for example\ntell people to behave in a particular\nway or you tell the people to uh\ncollaborate in a particular way or to\nwork on some projects and not work on\nsome other projects this is what we do\nin gradient policy optimiz policy\ngradient\noptimization we calculate the gradient\nwith respect to the parameters of this\nparticular uh objective function which\nwhat is the gradient\nuh yes later we talk about the discount\nfactor in reward sum okay so what is the\ngradient the gradient basically tells us\num how the um how the objective will\nchange if we change the parameter a\nlittle\nbit uh the gradient always tell us how\nit will increase so the gradient tells\nus what is the the direction of Max the\nmaximum Ascent of a particular uh\nfunction with respect to uh the variable\nin to which you calculate so in this\ncase we are Computing the gradient with\nrespect to the\nparameters of the objective function\nwhich tells us um how should I change\nthe parameters to increase this\nobjective function which is exactly the\nexpected reward that we want we can get\nfrom this policy\nso because the the the gradient tells us\nhow we should change the parameters to\nincrease the objective then we change\nthe the parameters according to the\ndirection of the gradient and this is\nwhat we do here so we have an objective\nwhich is tells us the expected reward\nwhen acting according to this policy we\ncalculate its gradient with respect to\nthe parameters which tells us how we\nshould change these parameters to\nincrease this objective so to increase\nthe expected reward and then we change\nthe parameters in the same direction of\nthe gradient and we do it\niteratively this is called policy\ngradient optimization and actually this\nis beautiful result because it means\nthat I can just use my cat whatever\npolicy my cat has right now I can just\num sample some trajectories from this\npolicy so I ask my cat to move around\ncheck what kind of reward I get\ncalculate the gradi with respect to the\nexpected reward according to these\ntrajectories and then tell the cat hey\nyou should do more of this because this\nled to a better reward or you should do\nless of this because it LEDs to a bad\nreward this is policy gradient\noptimization now it has some problems\nbecause uh policy gradient optimization\nbasically okay as you can let's skip the\nmath because if you want the math uh I I\nmade a video it's on YouTube so you can\nwatch it tomorrow but it has some\nproblems\nbecause as you can see the search space\nof the cat is enormous because at the\npossible trajectory that the cat can\ntake to go from here to the meat there\nare a lot because the cat can go like\nthis it can go like this it can go like\nthis it can go here then come back then\ngo down etc etc so there is many many\nmany many many um trajectory that the\ncan cat can take to go to the\nmeat uh however\nto compute this\nobjective we should actually check all\nthe possible trajectories to get the the\nthe direction of the gradient however\nthis is intractable means that in the\ncase of language model we should ask the\nlanguage model to generate all the\npossible output ever possible given a\nparticular question which is intractable\nbecause at each token the can the the\nlanguage model can choose what uh let's\nsay the vocabulary size is 30,000 then\nthe language model can choose 30,000\npossibilities for the first token then\n30,000 for the second 30,000 for the\nthird etc etc and to check all of them\nit's computationally impossible so we\ncan always approximate this with um\nsample uh with a sample uh this is\ncalled Monte Carlo estimation um however\nthis results in because we are not\nchecking all the possible trajectories\nbut we are making the decision of\noptimizing our cap using only a few\ntrajectories of course as you can see\nit's a risky situation so it means that\nwe are making a hard decision on how we\nshould change a policy without checking\nall the possible search\nspace uh this basically means that we\nhave high variance and there are many\nways to reduce this variance uh so when\nyou read the term Bas line in the deeps\npaper this is one of the ways to reduce\nthis variance because we are trying to\noptimize the language model into\nchoosing certain patterns into choosing\ncertain chain of thoughts into choosing\ncertain sequence of tokens without\nexploring all the possible generation\nthat the language model can\nhave um so so let me see how can we\nsimplify this\none um we blah blah okay so in order to\nreduce this variance so in order to make\nsure that we optimize the language model\neven without checking all the possible\nGenerations but still making sure that\nwe make the the gradient that we get so\nthe direction that tells us what\nparameter we should change and in which\ndirection in order to increase the\nexpected reward we can introduce this\nAdvantage term here this Advantage term\nbasically for each token tells the\nlanguage\nmodel how better is choosing this token\nover all the other token that I can\nchoose in this position for example in\nthe sample in the example that we saw\nbefore so where is\nShanghai is should the language model\nchoose the word Shanghai or it should\nchoose the word coffee or should it\nchoose the word Pizza well it's very\nmore it's much more advantageous to to\nto to choose the word Shanghai because\nit's very likely that the language model\nwill then complete it as Shanghai is in\nChina or Shanghai is a city in China or\nShanghai is located in China etc etc so\nthe the choosing the language the the\nthe word Shanghai results on the long\nterm in a much better uh reward so the\nadvantage of choosing Shanghai is higher\ncompared to all the other tokens in that\nuh condition in the case of the cat it\nmeans that the cat when is here it\nshould very high it's very advantageous\nto choose go down because it will result\nin going to the meat and um over all the\nother uh uh all the over all the other\nactions this doesn't mean that choosing\nup will lead you to die or to get no\nreward because you can always go up and\nthen change direction and go down but\nit's much more advantageous to just go\ndown this is the this is the\num uh this is the meaning of the\nadvantage term and this is the same\nAdvantage term that you see in the um\ngrp o loss now what is the difference\nbetween the the advantage term that you\nsee in the pop and the grpo is that the\nadvantage term in the PO requires the\nwhat is known as the value function in\nthe grpo they just um they they comput\nthis advantage in a different\nway uh which still results in in um\nvariance reduction um but without having\nthis value function estimation so grpo\nis computationally more um uh Advan uh\nhow to say efficient in this case\nokay let me see what else we need we've\nskipped also the part of the let me see\num off policy learning right so offline\npolicy\nlearning so what is off policy learning\nuh imagine we have the cat uh let's go\nto the cat\nactually\nokay to compute this the the the the\nloss to to okay the the gradient policy\noptimization we saw that we need to\nsample some trajectories right and we\ndon't have to sample all the possible\ntrajectories right because we are trying\nto approximate\nit um so when we sample these\ntrajectories we are sampling from a\npolicy which is the current brain of the\ncat so we ask the current brain of the\ncat to choose some actions and generate\nsome trajectories means that we ask the\nlanguage the cat to just navigate the\nhouse and let's see what it\ndoes\noops okay\nso we ask the the cat to just navigate\nthe house and see what it does and then\nafter the cat has navigated the house we\nlook at what the are the trajectory that\nthe cat has taken and then we give\nreward to the cat based on the\ntrajectory it has taken and then we\noptimize the policy which means that we\noptimize the brain of the\ncat um with the whatever it has learned\nwith the with the direction of the\ngradient based on the reward it has\nreceived but now the of the cat is a new\nbrain because it has changed compared to\nthe past which means that the next step\nof iteration of optimiz optimizing the\nbrain of the cat or its decision-making\nskills we need to sample new trajectory\nso we need to ask the cat again to go to\nall the house make a few choices and\nthen we check these choices and again we\ntell the cat hey you did here you did\nwell here you didn't do well so the cat\nwill learn to um to to to optimize its\nuh\npolicy which will result in a new policy\nand then again we need to sample from\nthis policy but as you can see every\ntime we do an optimization step we need\nto sample again these trajectories and\nin the case of language modules this\nmeans that first you need to sample some\nresponses then you uh reward these\nresponses based on your reward model and\nthen you need to sample new uh responses\nbecause now the the policy has changed\nbecause we updated the language model in\norder to avoid this sampling process\nwhich is expensive we introduce off\npolicy learning in which let me show you\nhere where is it okay in which basically\nwe take the language model we ask it to\ngenerate a lot of trajectories and we do\nit once then we sample some of these\ntrajectories which means basically we\nask it to generate some responses then\nwe sample some of these responses and we\nfind youe the language medal based on\nthe reward we got on these responses\nthen we don't sample new we we don't\nsample new trajectories or responses we\njust take another mini batch of the\ntrajectories that we sampled initially\nand again we do another step of\noptimization and we keep doing it for n\nsteps only then we sample new\ntrajectories this results in a much more\nefficient\ntraining so it's uh not like we are um\nhow to say do change a little bit the\npolicy and then sample new trajectories\nfrom this policy no we sample a lot of\ntrajectories initially we keep them in\nsome database in the memory or whatever\nyou want then we do some continuously\noptimizing the policy using the\ntrajectories that we have sampled\ninitially ER and this basically um\nresults in a much efficient uh\ntraining and now we can go back to the\num DPO sorry the DPS R1 uh paper here to\nunderstand finally the the the loss in\nits\nentirety so what we are doing here is we\nhave a policy at the current step of\nsampling and then we have the policy\nfrom which we sampled the trajectories\nso as you can see it's written here so\nwe sample first a question from our\ndatabase of questions then for each\nquestion we generate a list of outputs\nof responses so we prompted the model\nbasically with the question and then we\nask it to generate multiple responses\nand you can generate multiple responses\nlike we saw here like we ask language\nmodel with Shanghai and the language\nmodel will generate one response then we\nask it again very shangai and maybe this\ntime language model will say something\nelse and we ask it again etc etc so we\ngenerate list of outputs for the same\nquestion then we comput the following\nloss which is basically the ratio of the\nlog probability the leg the ratio of the\nprobabilities or the difference of the\nlog probabilities so the ratio of the\nprobabilities of the um assigned to the\noutput by the current iteration so the\ncurrent iteration at which we are um uh\noptimizing the language model with\nrespect to the language model from which\nwe sampled so the language model\nbasically when we initially sample the\ntrajectories we already precompute the\nprobabilities so we we can compute the\nprobabilities while while sampling we\ncan save them so we have always\navailable this Pi Pi of\nold now this ratio means what means that\nif at the current iteration the language\nimagine this ratio is more than one it\nmeans that at the current iteration the\nlanguage model is telling me that I want\nto choose this output more because I am\nmore likely to choose this one now what\nwe want is if the advantage of doing\nthis choosing this action is good\nbecause it leads Advantage positive\nmeans that it's good to choose it's\nadvantageous to choose this action\nbecause it results in a good reward so\nif the language model is more likely to\nchoose something and at the same time\nthis something also results in good\nAdvantage then this stuff here will be\npositive and it will be big and because\nwe are maximizing this will contribute\npositively to our objective because we\nare trying to maximize it so we want to\ndo more of this however if the language\nmodel on the other hand is trying to do\nless of something so means that this\nratio will be less than one and at the\nsame time this results in a something\nthat is disadvantageous so it means that\nit's not advantageous to do this then\nthe uh the language model is also\num okay uh let me check what so I got a\nlittle L if I am if this ratio is a\npositive and\nI am um I get a good Advantage it means\nthat the language model will be more\nlikely to do it if something that\nresults in bad advantage and the\nlanguage model is still doing it then it\nwill be a big negative reward which will\ncontribute negatively to our objective\nso the language model will be less\nlikely to do it at the same time we\ndon't want the language model to make\nbig decisions at every step we want to\nclip\nlimit the decision making of the uh\nlanguage model at each step so it means\nthat at each step of it of optimization\neven if the language model is very\nconfident that something is good then we\ndon't care how confident language is we\nwant to limit its confidence by clipping\nthis uh ratio here between 1 minus\nEpsilon and 1 plus\nEpsilon uh at the same time we also\ndon't want the language model to change\ntoo much so we have initial Frozen model\nmodel here which is pyref pyref\nbasically means the original model so in\nthe case of R1 it is the uh deepi V3\nbase so which is the language model that\nhas never been trained with\nreinforcement learning so we want\nbecause we want reinforcement learning\nframework we want to change the language\nModel A little bit to learn reasoning\nbut we don't want the language model to\nforget everything or to just not behave\nlike a language model anymore uh\notherwise language model could just do\nlike reward hacking so we want this is\nbasically pyre is the um uh the uh deep\nseek V3\nbase so the current policy so the\ncurrent language model should try to be\nas close as possible to the original\nmodel but at the same time it should try\nto change according to the reward it\ngets the advantages of the rewards it\ngets now how is the advantage term\ncomputed here basically it is each\nreward um normalized so it's basically\nit's like each reward coming out from a\ndistribution centered on mean of zero\nand standard division of\none um why do we want this because it\nmeans uh first of all uh we don't want\nthe numeric value of the reward to\naffect the training process but how\nbetter it is compared to the other uh\nrewards to affect the the the training\nprocess not the magnitude of its\nvalue okay now that we have seen this I\nbelieve you should\nhave most of the knowledge I guess to\nunderstand all of the\npaper uh because the other uh part that\nthey do is okay instead of just taking\nthe language model and just training\nwith the reinforcement learning let's\ntry to introduce some first of all um\nfirst of all let's try to introduce some\nvery high quality super uh uh fine\ntuning data and then we do another step\nof reinforcement learning and then we do\nanother step of fine tuning and then do\nanother step of reinforcement learning\nuh and this actually leads to a better\num better model uh another actually\ninteresting part of the the paper is the\ndistillation so the distillation I don't\nknow if most people are familiar with\nwhat is distillation and how it works so\nif you want I can talk about it a little\nbit otherwise I think let's go to sleep\nguys let's\nsee so yes means let's talk about\nit okay okay okay okay no\nproblem please give big picture overview\nI mean bro you can just read the paper\nyourself okay um yes okay more on\ndistillation uh okay so distillation\nbasically means this imagine you have um\nimagine you are trying to\nH\nH imagine you are trying to teach\nyourself\n[Music]\nuh okay imagine you are trying to teach\nyourself uh graduate maths uh that's one\nthing right imagine you don't have any\nbackground on math and imagine you learn\nit from a university teacher it's two\ndifferent thing right because if you try\nto learn it by yourself you need to come\nup with all the strategies to learn math\nbut if you learn it from a professor\nthen the professor can also give you\nhints on how to learn it faster so in\nthe case of uh models we have usually a\nbig model so let's call it Big\nBrother and then we have a smaller\nmodel what we want is and then we have a\na data set let's call it a data set so\ndata\nset what we do is we prompt the big\nbrother so the big model let's call it\nuh big model actually I don't want to\nconfuse people so this is the big\nmodel and this is the small model\nso now when we train language model so\nwhat how do we train language models\nfirst of all when we train language\nmodels we do that that in a\nself-supervised way what does it mean it\nmeans that there is no kind uh it means\nthat the the language model is trained\num without annotating the data means\nthat we sample a lot of text and we\nforce the language model to learn to\npredict the next token given the the\ncontext that comes before it it means\nthat imagine you want to train the\nlanguage model on the following\ntext for example uh the following\nsentence uh for distilled model we uh\nreport representative results blah blah\nblah imagine we want to train the\nlanguage model only on this green text\nhere what we will do is we will\ngive it the word for and we ask it to\nlearn to predict the word distilled when\nit is for then we give it the word for\ndistilled and we force it to learn to\npredict the word models when it is for\ndistilled the beauty of the Transformer\nis that all this process can be done in\nparallel so we don't have to do it one\nword at a\ntime uh if you do the job of training a\nsmall model on reasoning just by itself\nit will have much more difficulties\nhowever if you have a big model that has\nalready been trained on a specific task\nthen you can use the big model to help\nthe small model learn faster how when we\ntrain a language model on row data so in\nthis case for example we said to the\nlanguage model when you see four you\nshould choose\ndistilled uh when you see four distilled\nyou should models this basically is\ncalled the next token prediction task uh\nand the way we do it is we force the\ndistribution that the language model\noutputs and this distribution is the\ngoal of the language model so as we saw\nbefore the language model generates a\ndistribution over all the possible next\nwords so it will assign a probability to\nthe first word uh next word and to the\nsecond next word and to the third next\nword and the for next word and we ask it\nokay when you see for distilled you\nshould exactly this particular word\nwhich is the word\nmodels so you should choose the word mod\nTOs and all the other words should not\nbe chosen so 0 0 0 0 0 this one should\nbe chosen with 100% uh uh score this is\nhow we force the language model by doing\nit on many many many many text language\nmodel learns to generate a distribution\nthat very likely will generate this word\nmodels when it is for distilled and less\nlikely to generate the others if we do\nthe same job with the small model we\nwill see that the small model will have\na lot of difficulties learning um at the\nsame Pace as the big model why because\nthe big model has more parameters so it\nhas more flexibility in learning complex\ntask while the small model does not have\nthis flexibility so it will much be much\nslow learner so how do we how can we\ndistill the the the the knowledge of the\nbig model into the small model we do it\nas follows we take a a database set of\nprompts so a list of prompts\nwe feed it to the big model and then we\nask the big model to generate an answer\nnot only the answer we also ask it to\ngenerate the log probabilities at each\nstep of the generation process so uh\nbefore we use the sentence um where\nShanghai Shanghai is in China right so\nimagine we ask Lang with the big model\nwhere is Shanghai the model will\ngenerate Shanghai is in China uh but\nwith the because we have the word\nshanhai we also have the log probability\nof\nnot only the word Shanghai but also all\nthe other words that could we could have\nchosen in that\nposition we forc the small model to\nlearn not only to generate Shanghai but\nwe force it to learn the same\ndistribution that the big model\ngenerated for the same position so let\nme do it again so imagine now we do a\nconcrete example imagine we have a\nsentence where is\nShanghai Shanghai we put it give it to\nthe big model the big model will\ngenerate an answer what does it mean to\ngenerate an answer it means that it will\ngenerate first of all a distribution\nover what is the next likely next token\nwhich will be a list of probabilities\nwhere the word Shanghai will have a very\nhigh score so maybe\n0.6 and maybe the word pizza will be 0.1\nand the word the cat will be 0.05 Etc\n0.05 etc etc then we choose the word\nShanghai and we ask the language model\nagain where is Shanghai question mark\nShanghai then it will choose the word is\nbut it will not just choose the word is\nit will actually give us a distribution\nand we choose the word is it will give\nus a distribution that looks like this\nfor example it will say okay the word is\nis very likely so it's maybe 70%\nprobability the word I don't know uh Ro\nis unlikely because it's doesn't make\nsense and the word uh I don't know hell\nalso doesn't make sense or it is even\nless likely so for each position we can\ngenerate the log probabilities from the\nbig model and we force the small model\nnot to learn to just generate Shanghai\nbut to learn this entire distribution\nwhy because uh this gives a much\nstronger signal to the small model on\nwhat it should do what it can do and\nwhat it must never do instead of just\ntelling you should do this it's a bigger\nsignal for the small model to learn\nfaster and what they say in the um in\nthe deeps R1 paper is that by distilling\nyou can uh create much stronger model\nthan just by um training them from\nscratch a bit more context on the\nprobabilities for each forward pass\nwell\nuh no distillation is used on all the\ntokens so it's you telling for each\npossible token you learn the entire\ndistribution for each position not only\nthe\nlast so how we assign reward for each\naction okay\nin the case of\num in the case of\nPo by the way if you watch my video on\nPO I I explain all of that so I don't\nwant to kind of uh I don't know why\nnobody ever watched that video it's my\none of my masterpieces and nobody ever\ncared about it but all these slides come\nfrom my video that I done in\n2023 so um the way we generate the\nrewards is basically we the reward are\nusually only generated for the okay\nthere are two kind of reward that you\ncan generate one is called the outcome\nbased reward and one is called the\nprocess based reward in the case of Po\nwe generate the um outcome based reward\nand then through the advantage term is\nit is kind of distributed on all the pr\nprevious tokens because we sample a\nresponse and then we judge this response\naccording to our reward model then with\nthe advantage terms that you see in the\nloss this um reward is distributed on\nall the pre previous tokens so each\ntoken carries information on How likely\nit is going to lead to the final\nreward okay there is another part of\nokay I think we forgot one part guys um\nthe unsuccessful attempts this is\ninteresting also so uh they tried what\nis known as the process reward model\nbasically um okay the reward model that\nwe saw before is a rule based model that\nis outcome driven means that the model\nhave to generate the entire process and\nthe for example in the case of lead code\nproblems the model has to generate the\nfinal code to make it runnable then we\nrun the code we see how it performs and\nthen we have signal on the reward\nhowever there are other ways of\ngenerating reward and one of them is\ncalled the process reward model where\nyou divide the problem in sub problems\nand then you have a a model which is\ncalled the process reward model that\nassigns a reward to each single step\nhowever they in the paper they say that\nassigning first of all dividing a\nproblem into substeps it's difficult\nbecause every problem is different and\nwe don't want to kind of tell the model\nyou should follow this pattern it should\nthe model that should come up try to\ncome up with this pattern and actually\ndoes and secondly they say that it's\ndifficult to judge each single uh step\nso we sometimes we don't even know if\nthat step is good or not like when\nyou're trying to solve a met proof\nsometimes you do some intermittent steps\nthat may not lead to the um to the you\nmay not immediately see that they will\nlead to the uh final uh conclusion but\nthey are necessary right so it's\ndifficult to give them a reward uh until\nyou arrive to the end so it's difficult\nto kind of um give a reward to each\nsingle step and there is another\ntechnique called multic Carlo Tre search\nwhich basically it's a Tre search in\nwhich each intermediate nodes is given\neach each intermediate node is also a\nstep and each of these steps each of\nthese nodes actually has kind of a score\nassociated with it which increases the\nmore time that particular ular step\nleads to a successful uh solution so\nimagine for example you are doing a\nleite code and if you lead code problem\nso if you start your code for example\nwith a typo of course the code will not\ncompile so any sty anything that starts\nwith that type typo will never lead to a\nsuccessful solution so that node will\nnever be explored further so mon Carlo\nsearch basically forces the language\nmodel to explore more of the solutions\nthat lead to successful um attempts and\nless to the one that don't\nsucceed however even by doing Monte\ncarot research they they get like um sub\nuh results that are not as good as the\nuh ref forcement learning driven one and\nthe the beauty of this actually is also\nin this sentence which is rather than\nexplicitly teaching the model on how to\nfirst of all divide the problem into sub\nproblems and how to solve the problem or\ntell the model what is the format it\nshould follow to solve a problem the\nmodel just learns to solve to just\nlearns to come up with the right format\nwith the right Chain of Thought to solve\nthe problem uh of course they apply they\nplay with the reward because they they\ntell the reward that the the language\nmodel should um follow a particular\nformat and the language module should\nalso uh what is it uh should be accurate\netc etc so you can always play with a\nlittle bit reward and another thing that\nthey notice is actually if you train the\nlanguage model only on the reinforcement\nlearning then it leads to kind of the\nlanguage model with some side\neffects uh for example the fact that the\nlanguage model mixes languages because\nnobody told the language model that it\nhas to only speak English to solve a\nproblem that is uh stated in English so\nimagine I ask you to solve a problem\nlike a math problem and I say it in\nEnglish uh and imagine you are like a\nmixed kid and you can speak Chinese and\nEnglish or you are just Chinese who\nknows also English Etc\nmaybe in your head you you speak English\nand Chinese you think in English and in\nChinese and then you come up with this\nsolution and that's totally correct\nright so unless I tell you explicitly\nthat you should never think in Chinese\nthen you you why you should not take the\nfreedom of doing it right so if you\nnever tell the model to not do something\nthe model probably will do it and\nbecause in the reward here they never\ntold the language model to never think\nin other languages the language actually\nstarted thinking in other language\nanything to get the job done that's the\nbeauty of reinforcement learning so you\ngive the right incentive and the\nlanguage model will come up with the\nright way of reaching that goal as long\nas this signal is strong\nenough\num yes you need a massive cluster of\ngpus I think I believe so because you\nstill have a like you still running\ngradient descent on the language model\nitself so every time you are kind of\nfine tuning the language model you are\nchanging its weights\nso okay guys um before we talk about\nother questions okay uh all this lecture\nwas totally uh kind of um improvised so\nI didn't prepare it um so I hope I\ndidn't make big mistakes\nuh because the problem is reinforcement\nlearning from Human feedback is quite a\ncomplicated topic and you need to kind\nof derive everything step by step this\nis what I do in my video on refor\nlearning from Human feedback here I kind\nof sometimes skipped some step then went\nback Etc I hope I didn't create\nconfusion um so if there is kind of some\nparts about the RL that you didn't\nunderstand then I think we can talk\nabout it otherwise let's call it a day\nguys uh maybe you could record for\nYouTube more prepared I could butth I\nhave a daughter guys now I'm super busy\nand I also have a full-time job and I\nalso have a wife so have some mercy on\nme\nany thank you\nguys yeah I think everyone now should\nhave at least the basic and\nunderstanding on how to read the paper\nlike uh when you read the paper you\nshould have a clear idea of what's\nhappening and if you kind of need more\nbackground you\nare uh you can check like my previous\nworks I think let's stop the recording\nand have a good night\nguys and now if you want you can\nalso uh how to stop the\nrecording for",
  "transcript_chars": 68114,
  "ingested_at": "2026-05-15T16:37:42.713596+00:00",
  "source": "channel",
  "yt_meta": {
    "view_count": 62062,
    "like_count": 2061,
    "channel_id": "UCtAcpQcYerN8xxZJYTfWBMw",
    "categories": [
      "Science & Technology"
    ],
    "tags": []
  }
}