{
  "video_id": "hvGa5Mba4c8",
  "channel_slug": "umarjamilai",
  "channel_handle": "umarjamilai",
  "title": "Direct Preference Optimization (DPO) explained: Bradley-Terry model, log probabilities, math",
  "duration_seconds": 2926.0,
  "url": "https://www.youtube.com/watch?v=hvGa5Mba4c8",
  "upload_date": "",
  "transcript": "hello guys welcome back to my Channel\ntoday we are going to talk about DPO\nwhich stands for direct preference\noptimization it's a new technique that\ncame out in the middle of last year in\n2023 uh to align language\nmodels let's review the topics of today\nI will start with a short introduction\nto language models as usual so we can\nreview how language models work then we\nwill introduce the topic of AI alignment\nso what we mean by AI alignment and then\nwe will review reinforcement learning\nnow may be wondering why are we\nreviewing reinforcement learning if the\nwhole point of TPU is to remove REM\nreinforcement learning from language\nmodels well the reason is that actually\neven if DPO is does not use\nreinforcement learning algorithms they\nare still interconnected especially when\nwe talk about the reward model and the\nbread literary model so in order to\nunderstand the reward model and the\nbread lary model we need to review\nreinforcement learning and how the\nreward model affected the the process in\nreinforcement learning from Human\nfeedback in in the last part of the\nvideo we will see we will derive the DPO\nloss so we will understand how the where\ndoes it come from uh I will also give\nyou uh the show you the code on how to\ncompute the log probabilities so how we\nactually can use this log uh this this\nloss in practice now what are the\nprerequisite for watching this video\nwell for sure that you're familiar with\na little bit of probability and\nstatistics not much uh for example\nconditional\nprobability uh we are familiar with deep\nlearnings so what we mean by gradient\ndescent and loss functions you it's\nreally great if you have watched my\nprevious video on reinforcement learning\nfrom Human feedback in which I explain\nall the aspects of the reward model and\nthe reinforcement learning framework and\nthe p uh but it's not necessary for this\nvideo how because I will review most of\nthe part that are needed to understand\nthe DPO but it's really great if you\nhave already watched that video so you\ncan compare the two uh methods and also\nthat you're familiar with the transform\nmodel because we will be using it in\npractice when we want to compute the log\nprobabilities otherwise we don't know\nhow to use the loss of the\nDPO let's start our journey so what is a\nlanguage model well a language model is\na probabilistic model that assigns\nprobabilities to sequence of words in\npractice given a prompt for example a\nlanguage model allow us let me use the\nlaser so given a prompt for example\nShanghai is a city in a language model\ntells us the probability of what is\nmaybe the next token or word now in my\nvideos I always make the simplification\nthat a token is a word and a word is a\ntoken this is actually not the case in\nmost language models but it's useful for\nexplanation purposes so what is the\nprobability that the next token is China\nor the next token is Beijing or the next\ntoken is cat or pizza given a particular\nprompt this is the only thing that a\nlanguage model does and um the language\nmodel gives us this probability now you\nmay be wondering how can we use this\nlanguage model to generate text well we\ndo it with an iterative uh process so we\ntake a prompt for example where is\nShanghai we give it to the language\nmodel the language model will give us a\nlist of probabilities over what is the\npossible next word or token suppose that\nwe choose the token with the most with\nthe highest probability score so suppose\nit's Shanghai we take this token we\nselect it and we put it back into the\nprompt and we ask again the language\nmodel what is the next token then the\nlanguage model again will give us a list\nof probabilities over what is the\npossible next token we select the one\nthat we think is the most uh relevant\nusually we select the one with that is\nmost probable uh and we put it back into\nthe um prompt and we ask again the\nlanguage model etc etc until we reach a\nspe specified the number of generated\ntokens or we reach the end of sentence\ntoken which is a special token in this\ncase after four tokens generated the\nlanguage model will probably say\nShanghai is in China which is the answer\nto our\nquestion what do we mean by AI alignment\nnow when we train a language model we\ntrain it on a massive amount of data for\nexample thousands of books billions of\nweb pages and the entire Wikipedia Etc\nthis gives the language model a vast\nknowledge um in to to to complete any\nprompt in a reasonable way however this\ndoes not teach the language model to\nbehave in a particular way so for\nexample we can this does not the\npre-training does not teach the language\nmodel to be polite or to not use any\noffensive language or to not use any\nracist Expressions Etc because the\nlanguage model will just behave based on\nthe data that it has seen and if you\nfeed the internet data the language\nmodel will behave very very very badly\nactually so we need to kind of align the\nlanguage model to a desired Behavior so\nwe don't want the language model to use\nany offensive language we don't want it\nto be racist we want the language model\nto be helpful to the user to so to\nanswer questions like an assistant etc\netc and this is the goal of AI\nalignment now let's talk about\nreinforcement learning so reinforcement\nlearning is an area of AI that is\nconcerned with training intelligent\nagents to perform actions in an\nenvironment in order to maximize a\nreward that they receive from this\nenvironment let me show you with a very\nconcrete example I usually always use my\ncat oo for examples so let's talk about\noo oo is the agent in this case in this\nreinforcement learning scenario and he\nlives in a very simple word let's call\nit a greed word that is made of cells in\nwhich the cat's position is indicated by\ntwo coordinates the X position and the Y\nposition uh this can also be treated as\nthe state of the agent because at every\nevery positions corresponds to a\nparticular state the agent when it is in\na particular state it can take some\nactions in the case of the cat it can go\nright left up or down for every action\nthat the agent takes it will receive\nsome reward from the environment it will\nfor sure change its state to a new one\nso for example when the cat moves down\nit will change to a new state to a new\nposition and it will receive some reward\naccording to a reward model that we\nspecify in my case I have specified the\nfollowing reward model so when the cat\nmoves to an empty cell it receives a\nreward of zero If It Moves towards the\nbroom it receives a reward of minus one\nif somehow it arrives to the btub it\nwill receive a reinforce reward of minus\n10 because my cat is very scared of\nwater and if it arrives to the meat\nwhich is the cat's dream it will re\nreceive a reward of plus\n100 now what dictates what action the\nagent will take given a part particular\nstate or position well it is the policy\nthe policy indicates what is the\nprobability of the next action among all\nthe actions that are available that the\nagent can take can take given a\nparticular State and we usually write it\nlike this so that next action at time\nstep T is distributed like the\ndistribution uh induced by the policy\naccording to the state the um the agent\nis in now what is the goal in\nreinforcement learning the goal in\nreinforcement learning is to select a\npolicy or to optimize a policy in order\nfor the agent to maximize the expected\nreturn when it acts according to this\npolicy so imagine we have such a policy\nthat uh is optimized while our policy\nfor sure if the goal is to maximize the\nexpected reward when using this policy\nfor sure we will have a policy that will\ntake us on average to the meet because\nthat's one way to maximize the expected\nreward\nand for sure it will be a policy that\nwill allow us to minimize the chance of\nending up in the water here or to the\nbroom\nhere now you may be wondering okay the\ncat can be seen as a reinforcement\nlearning agent as a physical agent that\ntakes some action spot what is the\nconnection between reinforcement\nlearning and language models well as we\nsaw before in reinforcement learning we\nhave this thing called policy in which\ngiven State the policy tells us what is\nthe the probability over the action\nspace of the next action so what\npossible next action we can take and the\nprobability of each action this is also\nsomething similar to what we do with\nlanguage models because in language\nmodels we also have some kind of state\nwhich is our prompt and we ask the\nlanguage model to give us the\nprobability of the next token or we can\nconsider it the next action that the\nlanguage model can take and we want to\nreward this language model for selecting\ntokens in such a way that that they end\nup generating good responses and we\ndon't want to reward the language model\nfor selecting sequence of tokens that\nend up giving us bad responses now\nimagine we are trying to uh um train a\nlanguage model that needs to act like an\nAI assistant so for sure we want the\nlanguage model to be helpful to answer\nquestions in a meaningful way so not\njust output garbage we want the language\nmodel to not be racist or not use any\noffensive language right so this is all\ngood behaviors that we want from this\nlanguage model so we may want to build a\nreward model that will treat good\nresponses for example responses that\nactually answer the question asked by\nthe user and we we will reward them with\na high reward and we give a maybe zero\nreward or negative reward to all those\nanswers that are not coherent with what\nwe want so for example if the language\nuh model generates um uh dirty jokes or\num racist jokes for example we want we\ncan give zero reward to those um to\nthose responses so the language model\nacts also as a policy because the policy\nis something that given a prompt tells\nyou what is the pro probability over the\naction space or in this case the\nprobability over the token space so we\nwant to optimize this policy so we want\nto optimize the language model to\nmaximize the probability to maximize the\nexpected return or the expected reward\nthat it receives receives from our\nreward model so we want to optimize our\nlanguage model to generate good response\nbecause that's one way to obtain uh High\nreward from the reward model now you may\nbe wondering okay but how to define the\nreward model for a language model well\none way would be okay we can have a list\nof questions and answers generated by\nthe language model and then we can give\ngive a numeric reward to each one of\nthem and then we can use some\nreinforcement learning algorithm to uh\nfeed this reward model to the language\nmodel to optimize it but the problem is\nwhat kind of reward can we give to each\nof these pairs of questions and answer\nbecause for example let's look at the\nfirst question where is Shanghai the\nanswer suppose it is generated by the\nlanguage model is Shanghai is a city in\nChina now in in my opinion this is a\ngood response because it's short and up\nto the point but some other people maybe\nthink that only the word China is enough\nbecause this the user just asked where\nis Shanghai so there is no need to\nrepeat the word Shanghai but someone\nelse maybe think that this response is\ntoo short and the assistant should say\nhello I think that the your qu the\nanswer to your question is Shanghai is a\nc in China so different people will have\ndifferent opinions on what reward to\nassign to this particular pair of\nquestion and answer because we humans\nare not very good at finding a common\nground for agreement but unfortunately\nwe are very good at comparing and we\nwill exploit this fact so instead of\nbuilding a data set set that is made of\nquestions and answers and the rewards\nbecause we do not know what kind of\nreward to assign we will build a data\nset of questions and multiple answers\nand then we ask people to choose an\nanswer that they that they like\nAccording to some preference that we\nhave so we want generate for sure a\nlanguage model that is helpful so we\nwant the gener language model to give\nresponses that are correct and we want\nthe language model to be polite for\nexample so for example imagine we have a\nlist of questions and then we ask the\nlanguage model by by using for example a\nhigh temperature to generate multiple\nanswers and then we ask people to choose\nwhich one they like in this case for\nexample where is shanai for sure the\npeople most people will choose the\nanswer number one because Shanghai is a\ncity in China is the correct one uh for\nthis question here for example also\npeople will probably choose the this\nquestion here even if it's very short\nbecause the other one is probably wrong\nso using a data set like this we can\nactually train a model to trans\ntransform a pair of question and answer\ninto a numeric reward let's see how it\nis done if you have a pet you probably\nknow that to teach a particular Behavior\nto your cat or to your dog you need to\nuse biscuits or some treats so you ask\nthe cat to do something and if the cat\ndoes it then you give it a treat so it\nwill reinforce this memory in your cat\nand then the next time the cat is more\nlikely to do it because it will remember\nthat it received some treat and so it\nwill again perform that action again so\nit can probably receive another treat\nthis is exactly what we do in\nreinforcement learning we want to give\nsome digital biscuits to our\nreinforcement learning agent so that it\nuh is it is more likely to perform that\naction or that series of actions again\nin order to receive more reward however\nthe data set that we have built so far\nis made up made up of preferences so we\nhave a question multiple answers and\nthen we ask people to choose which\nanswer they\nlike we need to convert this data set of\npreferences into a numeric score that we\ncan give as a reward to our language\nmodel to choose more likely the answer\nthat was chosen by the people and to\nmake it less likely to choose the answer\nthat was not liked by the People by our\nannotators and this is can be done\nthrough a preference model in DPO and\nalso in reinforcement learning from\nHuman feedback we make use of the\nbreadly model so the bread L model is a\nway of converting a data set of\npreferences into a numeric score called\nreward that is given for each pair of\nquestions and answers our goal is to\ntrain a model that given a question and\nanswer or a prompt and the text\ngenerated text to give a score that\nresembles the preferences that have been\nchosen by our not\nators um this is the expression of the\nbread lary model so it is a model\nmeaning that we choose to model our\npreferences like this and actually it\nmakes sense because it is a probability\nso that's why for example we use\nexponentials because we want the\nprobabil the probability has to be uh\nnon- negative and also the probability\nthat the assigned to the correct\npreference so the probability of\nchoosing the correct answer of the wrong\nanswer so the one that is chosen by the\nannotators over the one that was not\nchosen by the annotators here I call it\nwinner and loser because also in the DPO\npaper they call it winner and loser it\nis modeled like this so it is\nproportional to the reward to the\nexponential of the reward that was\nassigned to the winning answer now how\nto train a model to convert a data set\nof preferences into a numeric reward we\ntake this expression and we can use a\nmaximum likelihood estimation now it\ndoesn't matter if you don't know what is\nmaximum likelihood estimation the point\nis we want to maximize the probability\nof assigning the correct ordering in our\npreferences so we want to maximize the\nprobability of choosing the correct uh\nanswer over the wrong answer and suppose\nthat we are maximizing this expression\nhere let's see how we can derive the\nloss to maximize this expression\nhere if you look at the DPO paper you\nwill see that they go from the Brad lary\nmodel which is this one directly to the\nloss here but they don't show you the\nderivation so I will I will show you how\nto derive the loss that maximizes this\nuh probability\nhere uh the uh derivation is very simple\nactually so uh first of all as you can\nsee in the loss you can see this\nfunction here it's a sigmoid function\nthe expression of the sigmoid function\nis this one and this is the graph of the\nsigmoid so the expression of the sigmoid\nfunction is 1 / 1 + e to the^ of minus\nx uh the first step of the derivation is\nto prove that um two uh exponentials so\na fraction of the this expression here\nso exponential divide divided by the sum\nof two exponentials can be written as a\nsigmoid of a minus B so here I call all\nthis part here so let me use the pen I\nthink it's\neasier so uh this part here so the\nreward assigned to the let's say the\nwinning answer is we call it a and the\nreward assigned to the losing answer we\ncall it B so this uh expression can be\nwritten as e to the power of a / e to\nthe power of a plus e to the power of B\nand we will prove that it can be written\nas the sigmoid of a minus B through the\nfollowing through the following step so\nfirst we can divide uh we take this\nexpression which is basically this one\nwe just replace the rewards with A and B\nbecause it makes it simpler to\nvisualize uh we divide the numerator and\ndenominator by the same quantity e to\nthe power of a we can do it uh then uh\nwe can um uh on at the numerator e to\nthe power of a cancels out with e to the\npower over and becomes a one then in the\ndenominator we add and subtract one we\nwe we can do it because it's like adding\nzero and then we collect the minus one\nso we don't change anything we just put\nthe parentheses this is possible through\nthe associative\nproperty uh then we do the uh common\ndenominator for these two expressions\nfor these two expressions and we arrive\nto this one we can simplify e to the\npower of a with minus E to the^ of a so\nit becomes e power of bide by e to the\npower of a which thanks to the property\nof the exponentials can be written as e\nto ^ of B minus a then we can take a\nminus sign outside and this expression\nhere is exactly the expression of the\nsigmoid function you can see here so\nit's 1 over 1 + e to the^ of minus\nsomething so it becomes the sigmoid of\nthat something here A minus B and this\nis exactly the loss that you see here so\nit is the sigmoid of the reward assigned\nto the winning answer minus the reward\nassigned to the losing answer here we\nalso see a log because usually we do not\nlog model the probability directly but\nwe model the log probability so we have\nalso this log because we want to model\nthe log probabilities it is something\nthat we can do because it's the\nlogarithm is a monotonic\nfunction and also you may be wondering\nwhy do we have this minus sign here uh\nthis is because we want to maximize this\nexpression but as you know in deep\nlearning uh Frameworks like pytorch we\nhave an Optimizer that is always\nminimizing a loss so instead of\nmaximizing something we can minimize the\nnegative expression of the objective\nfunction which is the same thing so\nbasically we take this loss function and\nif we apply it to a reward model which\nis a neural network it will be trained\nto uh maximize the probability of giving\nthe correct ordering to our preferences\nwhich can only happen when it assigns a\nhigh reward to the winning answer and a\nlow reward to the losing answer because\nif you look at this expression here as\nyou can see the probability is maximized\nwhen the in the numerator we have the uh\nreward assigned to the winning answer so\nthe reward assigned to the winning\nanswer is higher than the one assigned\nto the losing answer and um if you are\nwondering how to read an expression like\nthis so let me cancel because we will\nuse it a lot this kind of\nconvention uh this one this basically\nmeans that we have a data set of\npreferences uh where we have a prompt a\nwinning answer and a losing answer and\nthey belong to our data set of\npreferences and we train a module with\nthe gradient descent for each of these\npreferences we calculate this loss here\nthis expression here and if we minimize\nthis loss with the gradient descent we\nwill have a neural network that is\ntrained for the following the the bread\nliterary model\nbasically okay now that we have built a\nreward model which means that we have a\nmodel that given a question and answer\ncan assign a numeric reward to the\nlanguage model if the response is\ncorrect or looks good according to the\nbehavior that we want from our language\nmodel or looks bad according to the\nbehavior that we want from our language\nmodel now we can um we can train our\nlanguage model so what as you recall\nwhat is the goal in reinforcement\nlearning in reinforcement learning the\ngoal is to optimize a language model\nwhich is also the policy of our\nreinforcement learning agent in order to\nmaximize the cumulative reward when the\nagent acts according to this policy in\nother words if we let me use the pen so\nlet's ignore for now this green part\nhere Suppose there is no green part here\nso this doesn't exist\nimagine we have a language model let's\ncall it Pi Theta because it's a policy\nand we want to uh optimize this policy\nso we want to optimize this language\nmodel in order to maximize the reward\nthat it receives from the reward model\nit means that the language model will\ngenerate answers that give good reward\nand how they get good reward if the\nanswers are looks good they for example\nare not racist they are not using any\nsexual jokes and they are actually\nanswering the question that was\nasked um however and this is the goal in\nreinforcement learning from Human\nfeedback for example uh that's why there\nis it's called a reinforcement Le from\nHuman\nfeedback now uh if we use a model if we\nuse an\nobjective um like this that is we only\nwant to maximize the reward then the\nlanguage model may become greedy and\njust output garbage that gives it good\nreward so imagine we have a reward model\nthat rewards the language model for\nbeing polite the language model may just\nstart saying a list of thank you thank\nyou thank you or please please please\nand a lot of please or a lot of thank\nyous just to get high reward because\nprobably the word the word thank you and\nplease are highly rewarded by the reward\nmodel but we don't want the language\nmodel to just output garbage to get\nreward we want the language model to\nalso um uh output something that was\naccording to its training data so so\nit's a pre-training but we want to\nchange it a little bit so that it also\nacts according to our reward model so to\nour data set of preferences so it is\nmore polite but without forgetting what\nit has learned from the pre-training and\nthis is why we add this uh KL Divergence\nin the objective so let me use the pen\nagain so we we change the objective a\nlittle bit so we want the language model\nto maximize the reward it gets from the\nreward model but at the same time we add\na constraint to the language model\nthrough a KL Divergence now the K\nDivergence can be thought of as a\ndistance metric it is not a distance\nmetric but can be thought of as a\ndistance metric between two um\ndistributions um in which we have a\npre-train model so a language model that\nwas not fine-tuned through reinforcement\nlearning from Human feedback or DPO so\nit's just the language model that has\nbeen pre-trained on the Wikipedia on the\nbooks and on the internet web pages and\nthen we have the language model that we\nare optimizing so this Pi Theta and we\nwant them to be very similar so we want\nthe language model to not change much\ncompared to what it was before the\nreinforcement learning from Human\nfeedback or before the DPO training\ntring and this is why we add this KL\nDivergence so we want the language model\nto maximize its reward but at the same\ntime not forget or not change too much\nits output in getting this\nreward now what that now that we know\nthe reinforcement learning objective uh\nwhich is basically also the same\nobjective that we have in DPO because\nalso in DPO we want to train a language\nmodel that maximizes a reward but at the\nsame time does not for get its training\ndata let's look at what does it mean to\nactually maximize an objective function\nbecause this is an objective function\nthat we have and we want to maximize it\nbut what does it mean to maximize an\nobjective let's see um maximizing a\nfunction means to find the values of\nsome variable such that the value of the\nfunction is maximized for example if I\ngive you the following function f ofx is\nequal to Min - x - 3 + 4\nwhose graph is very simple it's just a\nparabola facing down uh to maximize\nthese functions means to find the value\nof the X variable such that the\nfunction the y basically the Y of this\nfunction is\nmaximized how to do that analytically\nwell we calculate the derivative of this\nfunction here we set the derivative\nequal to zero and we find the values of\nX for which this derivative is Z and\nthat is also the value for which the\nfunction will be maximized so the\nderivative of this simple function is -\n2x + 6 and the value of x that makes\nthis derivative zero is the value 3 x\nequal to 3 which is also the value of\nthe uh as you can see in the graph that\nmaximizes the function now the the\nobjective function that we saw before so\nthis one so in which we want to maximize\na reward but at the same time we want it\nwe want the the language model to not be\ntoo much different from the uh unaligned\nlanguage model so the the language model\nthat is not aligned through\nreinforcement learning from Human\nfeedback or DPO it is called a a\nconstrained optimization problem because\nwe want to maximize the reward but at\nthe same time we want to put some\nconstraint on this um objective function\nwe don't want the uh K Divergence to be\nuh too big we want it to be constrained\nin some limit\nnow the the point is okay there are many\ntechniques for constraint optimization\nand we will not be see them because\nthere are univers entire phds on\noptimization but one thing you may\nnotice is that okay this one here looks\nlike the objective function looks like a\nloss function so why cannot we just use\nfor example gradient decent to um\noptimize this objective function here\nsuch that we can train our language\nmodel to behave in a particular way to\nmaximize this reward well we could but\nas you know in deep learning and\nespecially with back propagation we need\nan objective function or a loss function\nthat is\ndifferentiable the following this uh\nobjective function is not differentiable\nwhy because as you can see from the\nexpression here this is an estimation of\nover the all the prompts in our data set\nand then a output that is generated by\nthe language model now to generate the\noutput of the language model as we saw\nbefore we need to use an iterative\nprocess in which we feed one token at a\ntime into the prompt uh we add uh we\nsample one token at a time from the\nlanguage model we take this token and we\nput it back into the prompt feed it\nagain to the language model Etc and we\nuse many strategies for selecting the\nnext token sometimes we use the gitty\nstrategy sometimes you use the beam\nsearch sometimes you use the top case\ntop P etc etc now this is sampling\noperation that we do on the language\nmodel to sample the answer of the\nlanguage model is not differentiable\nthat's why we cannot run reinforcement\nlearning to maximize this objective or\nto minimize the negative objective in\ncase we treat it as a loss and that's\nwhy in reinforcement learning we were\nforced to use algorithms like Po now\nlet's see how DPO handles\nthis in the DPO paper uh they start with\na very simple uh introduction to the\nreinforcement learning objective so as\nwe saw before the reinforcement learning\nobjective is to select a policy so a\npolicy that um maximizes the expected\nreward when using this policy so the\npolic is the language model and at the\nsame time puts a constraint on how much\nthis policy can change during this\ntraining this optimization and in the D\npaper they say okay there is an exact\nsolution to this optimization problem\nand it is the following it is the\nequation for in the DPO paper and exact\nsolution me I mean that there is an\nanalytical solution to the this uh\nconstrained optimization\nproblem just like we had a analytical\nsolution for the maximization problem of\nthis Parabola so we could find through\nthe derivative and setting the\nderivative equal to zero we could find\nthe value of x such that this function\nhere is maximized and for the using the\nsame reasoning but but different\ntechnique\num there we can we also have a\nanalytical solution for the constrainted\noptimization problem that we saw before\nand this is the solution now you may be\nwondering okay great we have an exact\nsolution just like the parabola so now\nwe are all set right yes the problem is\nwe we have an an exact solution but it's\nnot easily computable so it's not easy\nto compute so mathematically it it is it\nexists it makes sense but it's not easy\nto compute why because we have this Z of\nX term here now this Z of X term here if\nyou look at how it's defined it's the\nsummation of all possible\ny's um that are generated by the\nreference model so the as you know when\nwe do reinforcement learning from Human\nfeedback or DPO we have two models one\nthe one one is the language model that\nwe are trying to optimize and one is the\nFrozen model that we don't optimize but\nwe use it as a reference for the K\nDivergence so this is called the p ref\nso all the outputs generated by P ref\nmultiplied by the exponential of the\nreward now the problem is this summation\nis done over all possible WIS it means\nthat we need to sample all possible\noutputs through through um from our\nlanguage model given all the prompts\nthat we have in our data set of\npreferences now to generate all possible\noutputs is very very very expensive\nimagine you need to generate your\nlanguage model can generate 2,000 tokens\nfor each prompt it means that and you\nhave a vocabulary size of 30,000 it mean\nthat for the first position you have\n30,000 possibilities for the second\nposition you have 30,000 possibilities\nfor the third position you have 30,000\npossibilities and then you multiply all\nthese possibilities so it becomes a lot\na lot a lot of uh outputs that you need\nto generate to evaluate this Z of X term\nso the analytical solution to the\nconstraint optimization problem that we\nso before exists but it's not easy to\ncompute however one thing is interesting\nfrom this expression imagine that\nsomehow magically we have um access to\nan optimal policy so this um solution to\nthe optimization problem allow us to\ncompute what is the optimal policy given\nthe optimal reward model and the\nreference policy so the reference\nlanguage model but imagine that for some\nreason some magically we have access to\nuh we have this term here\nso if we have this term here we can\ncompute the optimal reward\nmodel with respect to the optimal\npolicy how well we can just isolate this\nR of X and Y term from this expression\nhere and it's very easy to compute\nbecause we can apply the logarithm on\nthe left and the right side of this\nexpression so let's do it step by\nstep we can apply the the log on the\nleft side and on the right side of this\nexpression here so this expression here\nand we will get that the uh the log of a\nproduct as you know is the sum of the\nlogs and the log of the ratio is the\ndifference of the logs so this Z term is\nin the denominator so it becomes a minus\nlog of Z of X this one is in the\nnumerator and this one is in the\nnumerator so they become sums of logs so\nthis one plus this log here then the log\nand exponential can cancel out because\nthey're inverse functions so this allow\nus to isolate this R of XY term with\nrespect to all the other terms and we\ncan write it like this so we can\ncalculate R of X and Y with respect to\nan optimal policy that we think we have\naccess to we do not have access to it\nbut we pretend we have access to it okay\nso there are two things that we do not\nhave in this expression we do not have\nthe reward model the optimal reward\nmodel and we do not not have the optimal\npolicy but we pretend that we have the\noptimal policy why let's see the next\nstep that they do in the DPO\npaper the next step is they say okay do\nyou remember the bread literary model as\nyou remember the bread lary model is a\nreward model right is the model that\ngiven a data set of preferences allow us\nto compute a numeric score a numeric\nreward well this um bread literary model\nreceives uh is based on a reward that we\nassign right so what if we plug the\nreward that we have uh computed uh from\nthe constraint optimization problem into\nthe bread literary model well we can do\nit so we have this reward that we obtain\nfrom the constraint optimization Problem\nby inverting the formula and we plug it\ninside the bread lary model so if you\nremember the bread literary model can\nalso be written as a sigmoid and and we\nprove it before in the previous\nslide so what we do is okay the bread\nlary model can be written as a\ndifference of rewards in the sigmoid\nfunction so if we plug the\nreward here so the reward uh obtained by\nthe constraint optimization uh problem\nsolution we will see that the two Z of X\nterms because this is a difference of\nrewards as you can see if we plug here\nfor the reward assigned to the winning\num response and here the reward assigned\nto the losing response we have these two\nZ of X terms so plus beta log of Z of X\nand minus beta log of Z ofx that will\ncancel out because they are one the\nopposite of the\nother this way we can obtain a formula\nthat does not contain the Z of X term\nand it's now computable\nso basically if we use the loss of the\npr\nlary model so as you remember the the pr\nliterary model is a model that allow us\nto to train a language model to uh model\nthe reward right and um if we use the\nloss of the bread literary model in\nwhich the reward is coming with respect\nto the um to the optimal uh policy we\ncan use it to optimiz the policy to\nadhere implicitly to the reward model\naccording to the bread ly model and this\nis the whole idea of the DPO paper so we\ncan plug the exact solution of the\nconstraint optimization of the\nreinforcement learning objective we can\ninvert it to get the reward we plug it\ninto the Brad literary model because the\nBrad literary model only depends on the\ndifference of rewards assigned to the\nlosing uh to the winning answer and to\nthe losing answer the uncomputable term\nZ of X cancels out and then it becomes\ncomputable and now we can use it to\ntrain a language model that will act\naccording to the reward model of the\nBrad literary model so to the U\npreference model um given by the bread\nliterary model so it will favor good\nresponses and at the same time it will\num uh you will be less likely to output\nthe preferences that were not chosen and\nat the same time it will put a\nconstraint onto the K Divergence so at\nthe same time it will put a restriction\non how much the language model can\nchange with respect to the uh reference\nmodel so the language model that was not\noptimized with reinforcement learning\nfrom Human feedback or\nDPO so basically with DPO we are doing\nkind of the same thing that we are doing\nin reinforcement learning but without\nusing the reinforcement learning um\nalgorithms so the goal in both of them\nis the same so we want to optimize a\npolicy we want to optimize a language\nmodel to maximize a cumulative reward\nbut at the same time we want to put a\nconstraint on how much it can change\nusing the K Divergence in the case of\nreinforcement learning from Human\nfeedback we are using the pop algorithm\nto optimize this um objective to\noptimize this policy but in the case of\nDPO we do not have to use reinforc\nlearning from Human feedback because we\nfound a loss that implicitly is already\num mapping this reward fun with this\nreward objective through this\nloss um let's see how to actually now uh\ncompute the um the log probabilities so\nhow to actually use this loss because\nfirst of all let's look at the\nexpression of this loss this law says\nthat if you have a data set of\npreferences in which X is the prompt the\nY YW is the chosen answer and Y L is the\nnot chosen answer because as you\nremember this data set is made up of\npreferences of question and two answers\nand then we asked some annotators to\ntell tell us which answer they prefer so\nif we have this data set we can run\ngradient descent using this data set\nover this loss here this loss here now\nto calculate this loss okay the\nlogarithm we can always calculate it's\njust a function the sigmoid is a\nfunction we can calculate but the\nlogarithm and the beta are the beta is a\nhyperparameter that indicates how much\nwe want the language model to change\nwith respect to the reference language\nmodel or how much we want to constraint\nit and then we have to compute these log\nprobabilities so the log of the\nprobability of generating this uh y w\nwhen the language moduel is prompted\nwith the prompt X and also for the P ref\nso also for the language model that is\nnot being optimized by DPO let's see how\nto practically uh compute this log\nprobabilities so um when you run DPO\nit's very simple so imagine for example\nyou are uh using a hugging phase it's\njust a matter of using this classes or\nDPO trainer in which you pass the\nlanguage model that you're optimizing\nthe Frozen version of the language model\nthat you don't want to optimize but it's\nis the reference language model um that\nis used to compute the log probabilities\nto calculate the K\nDivergence then you can give some other\ntraining arguments you can have the list\nof them on the website of hugging phas\nand then this beta parameter which\nindicates the strength on how much you\nwant the language model to change and\nalso in the website of DPO of hugging pH\nthey also give you what is the typical\nrange for this um for this uh hyper\nparameter now what will happen inside\nthe Library of the DPO trainer so inside\nthe hugging face Library when you use\nthe DPO trainer they compute this loss\nso they calculate this log probabilities\nyou can see here so for example this log\nprobability you can see here but how do\nthey actually compute well as you know a\nlanguage model usually most very most um\nin most cases it is a Transformer model\nand to compute these log probabilities\nas you know they use a prompt\na question the answer that is chosen and\nthe answer that is um not chosen so the\nwinning and the losing answer suppose\nthat we want to generate the log\nprobabilities for the winning answer so\nthis one so we have a language model we\ngive it a prompt and the answer that was\ngenerated and we want to calculate this\nlog probabilities what we can do is we\ncan combine the question and the answer\nin the same uh string in the same uh\ninput for the language model so imagine\nthe question is where is Shanghai oops\nimagine the question is where is\nShanghai question mark and the answer is\nShanghai is in China now let me use the\nlaser okay we can feed all of this to\nour language model so the pi Theta the\nlanguage model is a Transformer model\nmost of the cases and it will generate\nas you know the Transformer model\ngenerates some hidden States so it takes\nsome input which which are uh embeddings\nand it outputs some embeddings that are\ncontextualized also according to the\nself attention mask now if you don't\nknow how this works I highly recommend\nyou watch my previous video on the\ntransformer in which I show the self\nattention mechanism but basically the\nTransformer model is a model that takes\nsome embeddings and through the\nsortation mechanism output embeddings\nthen we take these embeddings and we can\nproject them into Logics using a linear\nlayer and we can do that for all the\ntokens that we give to the input usually\nwe only apply the linear layer to the\nlast token when generating the tokens\nbecause we are interested in generating\nthe next token but we we but we can do\nit for all the hidden States we are not\nforced to only use the last one because\nthe each hidden State encapsulates\ninformation about itself and all the\ntokens that come before it so we can\ntake this logits and then we can also\nconvert them into probabilities but we\ndo not want probabilities we want log\nprobabilities because as you can see\nhere we have this log function here so\ninstead of applying the soft Max we can\napply the log soft Max to each of these\nlogits now when we apply the soft Max it\nwill become a distribution over the\nentire vocabulary for one for each uh\ntoken in the vocabulary but we want only\nthe probability corresponding to the\nToken that was actually chosen to\ngenerate this particular answer and we\nalso know which token it was because we\nhave the answer so to the question where\nis shanai we know what is the answer\nbecause it's in our data set of\npreferences so we know that the answer\nis Shanghai is in China so how to\ncompute these log probabilities so we\ncan compute the log probabilities over\nthe entire dictionary over the entire\nvocabulary and then we only select the\nlog probability corresponding to the\nToken that was actually selected in the\nanswer so for this uh question for\nexample we select the for example the\nlast hidden State for the question which\ncorrespond to what should be the next\ntoken and we know what is the next token\nthe next token is Shanghai so we take\nthe Lo probability only corresponding to\nShanghai for this prompt here so where\nis Shanghai question mark Shanghai we\nknow that the next token should be is\nbecause it is already present so we take\nthe lock probability only corresponding\nto the Token is etc etc and we do it for\nall the token that are in the answer and\nthis gives us all the log probabilities\nof the tokens of the answer for this\ngiven\nquestion and then we can sum them up why\nwe need to sum them up because it's a\nlog probabilities usually if they are\nprobabilities we multiply them but\nbecause they are log probabilities uh we\nsum them up because the the logarithm\ntransforms transforms uh products into\nsubmissions and this is exactly what\nhappens inside the hugging face Library\nso inside the hugging face library to\ncompute the log probabilities to\ncalculate this loss they actually do\nwhat I described so they take the logits\nand they use the labels what are the\nlabels just the um the tokens\ncorresponding to the answer uh for\nexample to the winning answer or to the\nlosing answer depending on which term\nyou are Computing this one or this one\nand the model that we are using so the\nreference model or the model that we are\ntrying to optimize and then they select\nhere in this line here they check the\nthe log probabilities only corresponding\nto the uh labels so to the next token\nthat we that we already know what is it\nand then they sum them up as you can see\nhere and here they also apply a loss\nbecause we know we don't want all the\nlog probabilities but only the one\ncorresponding to the tokens that are\nbelonging to the answer not to the one\nthat belong to the\nquestion and this is how DPO works\nthank you guys for watching my video I\nhope you learned a lot I tried to\nsimplify as much as possible the math of\nDPO but the basic idea is that we want\nto remove reinforcement learning uh to\nalign language models this makes the\ntract much simpler because it just\nbecomes a simple loss in which you can\nrun a gradient descent and you don't\nhave to worry about training a separate\nreward model which is something that we\ndid in reinforcement learning from Human\nfeedback so if you watch my previous\nvideo as you remember the um the math is\nmuch more hard and much more topics to\nintroduce on how to optimize the\nobjective that we saw and um please come\nback to my channel for more videos like\nthis I usually try to make videos that\nare very deep very um uh in depth for\nevery topic sometimes they can be a\nlittle hard but I try to simplify as\nmuch as possible also depending on my\nknowledge and also depending on how much\nit is possible to simplify a difficult\ntopic and and um if you have any\nquestions please leave it in the comment\nand I will probably um keep publishing\nuh more videos like this but if you want\nvideos that are more simpler please let\nme know and also let me know in the\ncomments what kind of topics you would\nlike me to explore for next thank you\nguys and have a nice day",
  "transcript_chars": 44082,
  "ingested_at": "2026-05-15T16:38:05.622943+00:00",
  "source": "channel",
  "yt_meta": {
    "view_count": 36118,
    "like_count": 1208,
    "channel_id": "UCtAcpQcYerN8xxZJYTfWBMw",
    "categories": [
      "Science & Technology"
    ],
    "tags": [
      "rlhf",
      "reinforcement learning",
      "rl",
      "pytorch",
      "trl",
      "hugging face",
      "python",
      "tutorial",
      "math",
      "ai alignment",
      "language models",
      "large language models",
      "deep learning",
      "machine learning",
      "dpo",
      "direct preference optimization",
      "bradley terry model",
      "preference model",
      "reward model",
      "reinforcement learning from human feedback"
    ]
  }
}