{
  "video_id": "DVGmsnxB2UQ",
  "channel_slug": "statquest",
  "channel_handle": "statquest",
  "title": "Reinforcement Learning with Neural Networks: Mathematical Details",
  "duration_seconds": 1501.0,
  "url": "https://www.youtube.com/watch?v=DVGmsnxB2UQ",
  "upload_date": "",
  "transcript": "if you make a guess and you make the\nwrong guess then multiply it by a\nnegative number and correct the mistake\nstack Quest\nbam hello I'm Josh starmer and welcome\nto stack Quest today we're going to talk\nabout reinforcement learning with neural\nnetworks and this time we're going to\nfocus on the mathematical\ndetails this stag Quest is brought to\nyou by the letters a b and c a always b\nb c curious always be curious note this\nstack Quest assumes that you are already\nfamiliar with gradient descent and the\nessential concepts of how reinforcement\nlearning can be added to a neural\nnetwork if not check out the\nquests in the STA Quest reinforcement\nlearning with neural networks essential\nConcepts there were two places we could\ngo to get a french fry snack squatch's\nFred Shack and Norm's fry Hut and we\nused a neural network to help us decide\nwhere to go based on how hungry we were\nhowever since we didn't know in advance\nif Squatch or Norm was going to give us\na large or small order of fries we\ncouldn't train the neural network with\nstandard back\npropagation instead we trained the\nneural network with reinforcement\nlearning specifically we showed how the\nPO policy gradients method can train the\nneural network now we're going to go\nthrough the mathematical details and\nshow every single step required to train\nthis neural\nnetwork specifically we're going to show\nall of the details required to train\nthis bias B with policy\ngradients so the first thing we do is\nrun an input value through the neural\nnetwork to calculate an output value\nwe'll start with 0.0 as the input which\nmeans we're not\nhungry now we just do the\nmath and the probability that we will go\nto Norms P Norm is\n0.5 and that means that the\ncorresponding probability that we will\ngo to squatches P Squatch is 1 minus P\nNorm which equals\n0.5 in other words we can draw a line\nthat is 0.5 units long to represent the\nprobability of going to\nsquatches and then append another line\nthat is 0.5 units long to represent the\nprobability of going to\nNorms and we end up with a line that\ngoes from 0 to\n1 now to decide which place we will\nvisit for a snack we pick a random\nnumber between 0o and one in this\nexample we randomly pick the number\n0.2 and because 0.2 is is in the region\nthat represents squatch's fry Shack\nwe'll go there now comes the part where\nwe make a\nguess if we make a guess that the\ncorrect thing to do is to go to\nsquatches then ideally we would want the\nprobability of going to squatches P\nSquatch to be one and that means we can\nquantify the difference between the\nideal value for p Squatch 1.0 that is\nbased on the guess that we should go to\nsquatches\nand\n0.5 the value for p Squatch based on the\noutput from the neural\nnetwork now because the values for p\nSquatch are probabilities we will use\ncross entropy to quantify their\ndifferences note there's a lot to be\nsaid about cross entropy and if you're\ninterested in the details you can check\nout these quests or my book on\nAI however for this example the only\nthing you really need to know is that\ncross entropy is the negative log base e\nof the probability we go to either Norms\nor\nsquatches now remember the goal of this\nexample is to optimize the\nbias so just like we do for normal back\npropagation the first thing we do is\ncalculate the derivative of the Cross\nentropy with respect to the bias we want\nto\noptimize note before we get started I\nwant to be clear that we are calculating\nin the cross entropy for going to\nsquatch's fry\nShack so to help us remember what's\ngoing on we'll let C Squatch refer to\nthe Cross entropy for visiting\nSquatch now the cross entropy for\nvisiting Squatch C Squatch is the\nnegative log of P\nSquatch because we went to\nsquatches however because the output\nfrom the neural network is the\nprobability of going to Norms P Norm we\nrewrite the cross entropy equation in\nterms of P Norm because P Squatch = 1\nminus P\nNorm P Norm in turn is the Y AIS output\nvalue from the sigmoid activation\nfunction and the output from the sigmoid\nactivation function depends on its input\nthe x-axis coordinate x and x comes from\nmultiplying the input value hunger by\nthe weight and then adding the\nbias now because the cross entropy is\nconnected to the sigmoid activation\nfunction by P norm and the sigmoid\nactivation function is connected to the\nbias by\nX we can solve for the derivative of the\nCross entropy for visiting Squatch with\nrespect to the bias by using the chain\nrule the chain rule tells us that the\nderivative of the Cross entropy for\nvisiting Squatch with respect to the\nbias is the derivative of the Cross\nentropy for visiting Squatch with\nrespect to the output from the neural\nnetwork P\nNorm times the derivative of P Norm with\nrespect to X the xais input value to the\nsigmoid activation function times the\nderivative of x with respect to the\nbias we'll start by working out the\nderivative of the cross entropy for\nvisiting Squatch with respect to the\noutput from the neural network P\nNorm now because the cross entropy for\nvisiting Squatch is equal to the\nnegative log of 1us P\nNorm the first thing we do is plug in\nthe negative log of 1us P Norm for the\ncross\nentropy now to solve for this derivative\nwe have to use the chain\nrule because p Norm the thing we want to\ntake the derivative with respect to is\ninside the log function so to apply the\nchain rule we create a new variable\ncalled stuff and let stuff equal the\nstuff inside the log function oneus P\nnorm and that means we are solving for\nthe derivative of the negative log of\nstuff with respect to P\nNorm the chain rule tells us that the\nderivative of the negative log of Stu\nwith respect to P Norm is the derivative\nof the negative log of stuff with\nrespect to stuff times the derivative of\nstuff with respect to P\nNorm now because the derivative of the\nlog of x with respect to X is 1 /\nX the derivative of the negative log of\nstuff with respect to stuff is -1 /\nstuff\nthe next term the derivative of stuff\nwith respect to P Norm is\n-1 so we multiply the first term by\n-1 multiplying -1 by -1 gives us 1 ided\nby\nstuff lastly we plug in 1 minus P Norm\nfor\nstuff and that means the derivative of\nthe Cross entropy for visiting Squatch\nwith respect to the output from the\nneural network P Norm\nis equal to 1 / 1 - P Norm so we plug\nthat into the equation\nbam now we need to figure out the\nderivative of P Norm with respect to X\nthe xais coordinate that is used as\ninput to the sigmoid activation\nfunction so the first thing we do is\nplug in the equation for the sigmoid\nactivation function then we rewrite the\nfraction as the denominator to the -1\npower now because X the thing we want to\ntake the derivative with respect to is\ninside parentheses we can use the chain\nrule to solve for the\nderivative so just like before we Define\nstuff to be the stuff inside the\nparentheses then we rewrite the equation\nwith\nstuff the chain rule then converts this\nrelatively hard to solve\nderivative into the product of these two\neasier to solve\nderivatives for the first term we use\nthe power rule to get -1 * stuff raised\nto the -2 power and the derivative of\nthe second term is e raised to thex\npower now we just multiply both terms\ntogether and plug in 1 + e raised to\nthex power power for\nstuff now technically this is the\nderivative of the output of the sigmoid\nactivation function P Norm with respect\nto its input\nX however you never see it like that\never instead we split the square in the\ndenominator into two terms multiplied\ntogether then we put + one and minus one\nwhich add up to zero in the numerator of\nthe second\nterm we then rewrite the second term as\nthe difference of two\nfractions lastly the first term is the\nsame as the equation for the sigmoid\nactivation function so we swap the\nfraction with the\nfunction and the first part of the\nsecond term is equal to\none and the second part is the same as\nthe sigmoid activation\nfunction and this is the derivative of\nthe sigmoid activation function that\nyou'll see pretty much\neverywhere so we plug that into the\nequation\nbam now we need to figure out the\nderivative of x the input to the sigmoid\nactivation function with respect to the\nbias so the first thing we do is plug in\nthe equation for\nx and the derivative with respect to the\nbias is one so we plug that into the\nequation\nbam lastly because P Norm is the output\nfrom the sigmoid activation function we\ncan replace sigmoid X in the derivative\nwith P\nNorm hooray we finally calculated the\nderivative of the Cross entropy for\nvisiting Squatch with respect to the\nbias\nbam now let's calculate the value of the\nderivative\nso we plug in P Norm the output from the\nneural network\n0.5 and when we do the math we get\n0.5 and thus the slope of the tangent\nline for the current bias value is\n0.5 note if we plugged this derivative\ndirectly into gradient descent it would\nshift the value of the bias to the\nleft however if you remember this\nderivative was based on the guess that\ngoing to squatches for fries was a good\nidea and our guess might not be\ncorrect and if our guess is not correct\nthen instead of Shifting the value for\nthe bias to the left we actually need to\nshift the value for the bias to the\nright in other words depending on\nwhether our guess is correct or not we\neither need to shift the value for the\nbias to the left or the right\nthe good news is that all we need to do\nto determine how to update the bias is\norder some fries from\nSquatch in this example when we order\nsome fries we get this relatively small\nserving however this is okay because our\nhunger level is 0.0 meaning we're not\nhungry at all so it was actually a good\nthing that Squatch ate some of our\nfries in other words getting a small\norder of fries when we are not hungry\nmeans we made the correct guess and\nsince we made the correct guess we set\nthe reward to\n1.0 in contrast if Squatch had given us\na large order of\nfra then we wouldn't be hungry enough to\neat all of the fries and that means we\nmade the wrong\nguess so in that case we would set the\nreward to\n-1 note we can use different values for\nthe reward and we talked about that in\nthe stat Quest reinforcement learning\nwith neural networks essential\nConcepts anyway going back to when we\nmade the correct guess and the reward\nwas\n1.0 now we multiply the derivative by\nthe\nreward to get an updated derivative that\nwill point us in the correct\ndirection in this example we were not\nhungry and Squatch gave us a small order\nof fries so we made the correct guess\nand the reward equals\n1.0 and multiplying the derivative 0.5\nby the reward 1.0 means the updated\nderivative is the same as the original\nderivative and the updated derivative\ntells us to do the same thing as before\nand shift the value for the bias to the\nleft in other words when we make the\ncorrect guess to begin with then the\nderivative that is based on that guess\npoints Us in the correct\nDirection note if we guessed wrong and\nSquatch gave us a ton of fries even\nthough we were not\nhungry then multiplying the derivative\nby a negative reward makes it so that\nthe updated derivative\n0.5 points in the opposite direction of\nthe original\nderivative and instead of Shifting the\nvalue for the bias to the left we will\nshift it to the\nright in other words when we make an\nincorrect guess to begin with then the\nderivative that is based on that guess\npoints Us in the wrong direction and\nmultiplying it by a negative reward\ncorrects that error now going back to\nthe original correct guess we then plug\nthe updated derivative into gradient\ndescent to calculate the step\nsize now in this Example The Learning\nrate is set to\n1.0 so the equation ends up being 1 0 *\nthe updated derivative\n0.5 and so the step size is equal to\n0.5 then we calculate the new value for\nthe bias by subtracting the step size\n0.5 from the old bias value\n0.0 to get the new bias value\n0.5 now we plug the new bias term into\nour neural\nnetwork now that we have updated the\nneural\nnetwork we have to get some more fries\nso we can keep training the\nbias so we plug in our hunger level\n0.0 and when we do the math the\nprobability that we will go to Norms is\nnow\n0.4 that means that the probability we\ngo to squatches 1 minus P Norm has\nincreased to\n0.6 and the probability that we go to\nNorms has decreased to 0.4\nremember the last time our hunger was\n0.0 we went to\nsquatches and we got what we wanted so\nit makes sense that given the same\nhunger as before we now have a higher\nprobability of going to\nsquatches now we pick a random number\nbetween 0 and 1 and we get\n0.9 and because 0.9 is in the region\nthat represents Norm's fry Hut that's\nwhere we'll go now we make the guess\nthat going to Norms when we are not\nhungry is the best thing to do and if\ngoing to Norms is the right thing to do\nthen ideally the probability of visiting\nNorms should be\n1.0 and that means we can quantify the\ndifference between the output and our\nguas with cross\nentropy thus the first part of updating\nthe bias is to calculate the derivative\nof the Cross entropy with respect to the\nbias note before we get started I want\nto be clear that we are calculating the\ncross entropy for going to Norms fry Hut\nso to help us remember what's going on\nwe'll let C Norm refer to the Cross\nentropy for visiting\nNorm now the good news is calculating\nthe derivative of the Cross entropy for\nvisiting Norms is almost exactly the\nsame as the derivative we calculated\nearlier for the cross entropy for\nvisiting Squad\nwatches specifically the only difference\nis that the cross entropy for visiting\nNorm C Norm is the negative log of P\nNorm instead of the negative log of P\nSquatch and remember P Norm is just the\noutput from the neural\nnetwork and just like we saw before P\nNorm is the Y access coordinate that\ncomes from the output of the sigmoid\nactivation function and the out output\nfrom the sigmoid activation function\ndepends on its input the x-axis\ncoordinate x and x comes from\nmultiplying the input value hunger by\nthe weight and then adding the\nbias so just like before because the\ncross entropy is connected to the\nsigmoid activation function by P norm\nand the sigmoid activation function is\nconnected to the bias by\nX the chain rule tells us that the\nderivative of the Cross entropy for\nvisiting Norm with respect to the bias\nis the derivative of the Cross entropy\nfor visiting Norm with respect to the\noutput from the neural network P\nNorm times the derivative of P Norm with\nrespect to x times the derivative of x\nwith respect to the\nbias first we'll calculate the\nderivative of the Cross entropy for\ngoing to Norms with respect to P norm\nand we get -1 / P\nNorm so we plug that into the equation\nbam everything else is the exact same as\nwhen we calculated the derivative of the\nCross entropy for going to\nsquatches and thus this is the\nderivative of the Cross entropy for\nvisiting Norms with respect to the\nbias now we just plug in the value for p\nNorm\n0.4 and when we do the math we get\n0.6 and thus the slope of the tangent\nline for the current bias value is\n0.6 now we need to determine if we made\nthe correct guess so that we know if we\nshould shift the value for the bias to\nthe left or the right so we order some\nfries and Norm gives us a huge pile of\nthem normally this would be\nawesome unfortunately we we aren't\nhungry so that means we made the wrong\nguess and the reward equals\n-1 we then multiply the derivative by\nthe reward to get the updated derivative\n0.6 note because we made the wrong guess\nthe original derivative which was based\non that guess was pointing in the wrong\ndirection so by multiplying the original\nderivative by a negative number we flip\nthe direction and correct the\nmistake this is super important so let\nme repeat it when we multiply a\nderivative that is based on an incorrect\nguess by a negative reward we flip the\ndirection and correct the\nmistake\nbam we then plug the updated derivative\ninto gradient descent to calculate the\nstep\nsize and remember in this Example The\nLearning rate is set to\n1.0 so the step size ends up being\n0.6 then we calculate the new value for\nthe bias by subtracting the step size\n0.6 from the old bias value\n0.5 to get the new bias value\n-1.1 now we plug the new bias term into\nour neural\nnetwork now if we plug 0 .0 which means\nwe are not hungry into the neural\nnetwork the probability that we will go\nto Norms is even smaller than it was\nbefore now it's\n0.2 and it makes sense that the\nprobability of going to Norms is getting\nsmaller because so far each time we've\nplugged 0.0 into the input we should\nhave gone to squatches instead of\nnorms and that means that the\nreinforcement learning algorithm we're\nusing ordinary policy gradients is\nworking double\nbam note so far we've only plugged 0.0\nfor Hunger into the neural\nnetwork however in order to fully train\nthe model we have to use values from 0\nto one as inputs to the neural\nnetwork and after doing lots of updates\njust like we did earlier using the same\nderivative equations and updating them\nwith the rewards\nbut now using all kinds of input values\nbetween 0er and\none eventually the value for the bias\nstarts to hover around\n-10 and that means we're done\ntraining triple\nbam note in this stat Quest We simply\nDefine the rewards to be -1 and one\nhowever out in the wild you'll probably\nsee much more elaborate ways to\ndetermine the\nreward that that said regardless of how\nfancy the reward function is its purpose\nis the same to update and possibly\ncorrect the derivative that we\ncalculated based on our\nguess now it's time for some Shameless\nself-promotion if you want to review\nstatistics machine learning and AI\noffline check out the stack Quest PDF\nstudy guides and my books the stat Quest\nIllustrated guide to machine learning\nand the stat Quest Illustrated guide to\nneural networks and AI at stack\nquest.org there's something for\neveryone hooray we've made it to the end\nof another exciting stat Quest if you\nlike this stat Quest and want to see\nmore please subscribe and if you want to\nsupport stat Quest consider contributing\nto my patreon campaign becoming a\nchannel member buying one or two of my\noriginal songs or a t-shirt or a hoodie\nor just donate the links are in the\ndescription below all right until next\ntime Quest on",
  "transcript_chars": 18133,
  "ingested_at": "2026-05-15T10:52:59.627916+00:00",
  "source": "channel",
  "yt_meta": {
    "view_count": 29996,
    "like_count": 754,
    "channel_id": "UCtYLUTtgS3k1Fg4y5tAhLbw",
    "categories": [
      "Education"
    ],
    "tags": [
      "Josh Starmer",
      "StatQuest",
      "Machine Learning",
      "Statistics",
      "Data Science"
    ]
  }
}