{
  "video_id": "Z-T0iJEXiwM",
  "channel_slug": "statquest",
  "channel_handle": "statquest",
  "title": "Reinforcement Learning: Essential Concepts",
  "duration_seconds": 1093.0,
  "url": "https://www.youtube.com/watch?v=Z-T0iJEXiwM",
  "upload_date": "",
  "transcript": "reinforcement learning it's just like me\nand you we learn and adapt from\nexperience yes we do stat Quest bam\nhello I'm Josh starmer and welcome to\nstack Quest today we're going to talk\nabout reinforcement learning and we're\ngoing to cover the essential Concepts\nthis stag Quest is brought to you by the\nletters a b and c a all always B Be C\ncurious always be\ncurious imagine we were hungry and\nwanted to eat a snack and had to choose\nbetween going to squatch's fry Shack and\nNorm's fry\nHut the problem is that we've never been\nto either place so how do we decide\nwhich one to go\nto specifically we'd like to know which\nplace will do a better job giving us a\nsatisfying order of\nfries to solve this problem we're going\nto use reinforcement\nlearning reinforcement learning is a\nmethodology that basically lets\ncomputers learn and adapt based on\nexperience and it's used in all kinds of\nsituations for example reinforcement\nlearning has been used to help computers\nbecome better at playing games like\nCheckers and go and to help cars drive\nby themselves and it's even been used to\nmake chat gbt sound more human when it\nresponds to your\nprompts that said in this stat Quest\nwe're going to learn how it can help us\ndecide where to eat\nfries note what follows is just one\nexample of how reinforcement learning\ncan be done and the purpose is to focus\non the main ideas of how this method\nworks and for now we're going to ignore\nthe copious amounts of terminology of\nassociated with this field but don't\nworry we'll dive into the terminology\nafter this\nexample so with that said now let's see\none way that reinforcement learning can\nhelp us decide where we should go eat\nfries since we've never been to either\nsquatch's fry Shack or Norm's fry Hut\nwe'll start with each Place having an\nequal probability that we will go there\nin other words we'll start with the\nprobability of picking watches fry Shack\nfor a delicious snack P Squatch set to\n0.5 and the probability of picking\nNorm's fry Hut for a tasty treat P Norm\nset to\n0.5 now to visualize how we will choose\na restaurant let's draw a line that is\n0.5 units long to represent the\nprobability of going to\nsquatches and then appin to that another\nline that is 0.5 units long that\nrepresents presents the probability of\ngoing to\nNorms combined we have a line that goes\nfrom 0 to\none now to decide which restaurant we\nwant to visit we pick a random number\nbetween 0 and one in this example we\nrandomly pick the number\n0.7 and because 0.7 is in the region\nthat represents Norms fry Hut we go\nthere and when we go to Norms we are\nserved a large order of fries\nbam getting served a satisfying order of\nfries at norms is awesome but so far\nwe've only been there one\ntime and that means that we don't yet\nhave a lot of confidence that most of\nthe time will be served a satisfying\norder of fries at\nnorms and it's possible that squatch's\nfry Shack also serves satisfying orders\nof fries so we still might want to try\nout squatch's\nplace because we were just served an\norder of delicious and satisfying fries\nat Norm's fry Hut but it was just our\nvery first visit and we're not sure if\nthat will happen most of the\ntime it makes sense that we should\nincrease the probability that we will\nvisit Norm's fry Hut in the future but\njust by a little bit so here's one way\nwe can increase the probability that we\nwill visit Norms in the future but just\nby a little bit\nbit first because these fries are so\nsatisfying on a scale from 0 to 1 we\ngive them a fry score equal to one and\nthen we can use this equation to\ncalculate the new probability of\nvisiting Norm's fry Hut new P\nNorm this equation uses the existing\nprobability that we will visit Norms P\nnorm and combines it with the fry score\nthat we assigned to our order of fries\nand a learning rate which helps prevent\nchanging the probability too much too\nsoon to calculate a new probability for\nvisiting Norms fry Hut new P\nNorm so we plug in the current value for\np Norm\n0.5 plug in the fry score\none and then plug in a learning rate\nwhich in this example will set to\n0.1 note we'll show what happens when we\nhave different learning rates in just a\nbit for now when we do the math we get\n0.5 +\n0.05 which is\n0.55 and that means we increase the\nprobability of going to Norm's fry Hut\nto\n0.55 we then calculate the new\nprobability of visiting squatch's fry\nShack new P\nSquatch as one minus new new P\nNorm so we plug in the value for new P\nNorm\n0.55 and we get\n0.45 and that means we decrease the\nprobability of going to squatch's fry\nShack to\n0.45 in other words we increased the\nprobability that we will visit Norms\nagain in the future just a little\nbit and we decreased the probability\nwe'll visit squatches is just a little\nbit increasing the probability of\nvisiting Norms the next time we want to\nsnack just a little bit makes sense\nbecause even though we got a satisfying\norder of fries at norms so far we've\nonly been there one\ntime and it's possible that Norm's fries\nwill not always be awesome and it's also\npossible that squatch's fries could be\nawesome so we still want to have a\nchance to visit\nsquatches bam now before we move on and\nget hungry for another snack let's reset\nthe probabilities to what they were at\nthe start 0.5 for both Squatch and norm\nand talk about the learning rate\nlearning rates control how big or little\na change we will make to the\nprobabilities each time we update them\nfor this equation learning rates are set\nto values between 0 and\none if we set the learning rate to to\n0 then the whole second term is equal to\nzero and goes\naway and we end up with the new P Norm\nequal to P Norm in other words when the\nlearning rate is set to zero we don't\nmake any changes at all in contrast if\nwe set the learning rate to\none and then plug in the current value\nfor p Norm\n0.5 and plug in the fry score\none then we will end up setting new P\nNorm to\none and that means that the next time we\nget hungry we will go to Norms without\nleaving any probability to visit\nsquatches and that's not really what we\nwant to do right now since we've never\nbeen to\nsquatches however when we set the\nlearning rate to something larger than\nzero and less than one like\n0.1 we end up setting new P Norm to\n0.55 which is just a little larger than\nthe original value for p Norm\n0.5 and that small change does a better\njob reflecting the satisfying order of\nfries we got from Norms while also\npreserving a chance that we'll visit\nsquatches note in practice the learning\nrate is something that we usually set to\nsome default value like 0.1 and if it\ndoesn't work we test other\nvalues small\nbam understanding the purpose of the\nlearning rate and updating the\nprobabilities that we will visit\nsquatch's fry Shack and Norm's fry Hut\nis hungry work so now it's time for\nanother\nsnack like before we can draw lines to\nrepresent the probabilities that we will\nvisit squatch's fry Shack or Norms fry\nHut but now the line for squatches is a\nlittle shorter\nand the line for Norms is a little\nlonger next we pick another random\nnumber between 0er and one and this time\nwe get\n0.2 and because 0.2 is in the region\nthat represents squatch's fry Shack\nwe'll go there and when we go to\nsquatches we get served an\nunsatisfyingly small order of\nfries hey Squatch why are your fries so\nlame I needed a snack too so I ate some\nof them are you going to eat our fries\nevery\ntime I don't know I guess it depends on\nhow hungry I\nam okay just know that it's bad for\nbusiness anyway because squatch's fries\nare unsatisfying we give them a fry\nscore equal to\nzero and now we can use this equation to\ncalculate a new probability for visiting\nsquatch's fry Shack new P\nSquatch this equation is like the one we\nused earlier but now we use P Squatch\nthe most recent probability for visiting\nsquatch's fry Shack instead of P\nNorm so we plug in the numbers using the\nsame learning rate we used before\n0.1 and we get\n0.41 so because the fries at squatches\nwere unsatisfying we reduce the\nprobability we will go there from\n0.45 to\n0.41 with the new probability for\nvisiting squatch's fry Shack new P\nSquatch we can update our probability\nfor visiting Norms fry Hut new P norm\nand we increase the probability that we\nwill visit Norms from\n0.55 to\n0.59 now when we draw out the\nprobabilities the line for squatches is\na little shorter than\nbefore and the line for Norms is a\nlittle\nlonger so we pick another random number\nand this time we get 0.1 so we go back\nto\nsquatches and again when we go to\nsquatches the fries are lame\ndouble I couldn't help it I was\nhungry anyway like before squatch's fry\nscore is equal to\nzero and we use that to update the\nprobability we'll visit squatches in the\nfuture to\n0.37 then we update the probability that\nwe'll visit Norms in the future to\n0.63 using this approach we can continue\nto visit norms and squatches to update\nthe probabilities we will go to each\nPlace most of the time Norm gives us a\nsatisfying order of\nfries so the fry score for Norms is\nusually\none and as a result we usually increase\nthe probability that we will visit\nNorms however every now and then Norm\njust can't resist and eats most of our\nfries so the fry score is equal to\nzero and we end up decreasing the\nprobability that we will visit\nnorms and on the increase inre inly\nfewer occasions that we go to squatches\nwe usually get a lame order of\nfries and as a result we reduce the\nprobability that we will visit squatches\neven\nmore however once in a blue moon Squatch\nsomehow resists eating our fries and we\nget a satisfying order and we end up\nincreasing the probability that we will\ngo to squatches in the\nfuture anyway now we just repeat these\nsteps we use the probabilities we will\nvisit squatches and Norms to determine\nwhere we will go to get\nfries and then based on how satisfying\nthe fries were we update the\nprobabilities we will visit each place\nafter repeating those steps a bunch of\ntimes the probability of going to\nsquatches ends up hovering around\n0.19 and the probability of going to\nNorms ends up hovering around\n0.81 in other words every time we go out\nfor fries we change the probabilities\nbut after a while those changes stay\nclose to 0.19 for Squatch and 0.81 for\nnorm and now we know that when we get\nhungry most of the time we should go to\nNorms because Squatch gets hungry too\noften and eats our\nfries\nyum double bam\nnow that we've seen an example of how\nreinforcement learning can work it's\ntime for a terminology\nalert for starters using reinforcement\nlearning lingo we would call squatch's\nFry Shack and Norms fry Hut the\nenvironment the environment is something\nthat we want to explore and can interact\nwith in this example we explore and\ninteract with the environment by\nvisiting each place and ordering\nfries and because we are the ones\nexploring the environment we are called\nthe\nagent in this example we the agent\nexplore the environment by using the\nprobabilities that we will visit\nsquatches and\nnorms and these probabilities are called\nthe\npolicy in this example we use the policy\nin a very simple way we simply pick a\nrandom number between 0er and one and if\nit lands in the area that represents\nNorms we go to\nNorms otherwise if the random number\nFalls in an area that represents\nsquatches we go\nthere however our policy could be more\ncomplicated by taking into account how\nmany times we visited Norms or squatches\nin the past week or factor in how hungry\nwe are lastly the pry score which we use\nto update the policy is called the\nreward\nin reinforcement learning the goal is to\nmodify the policy so that we can\nmaximize the\nreward and in this example that means\nupdating the probabilities that we visit\nsquatches and Norms so that we maximize\nthe delicious fries we are served note\nin the example given here most of the\ntime Norm gave us a great order of fries\nand the reward was relatively\nHigh however every now and then Norm ate\nsome of our fries and gave us a small\nserving and the reward was low so the\nreward can change and doesn't have to\nalways be the same to\nsummarize we the agent needed to explore\nthe environment which in this example\nmeans we needed to decide if we should\nvisit squatches or\nNorms so we used the policy the\nprobabilities that we should visit each\nplace to help us decide where to go\nand depending on the order of fries we\neither got a large reward or fry score\nor a small\nreward and because our goal is to\nmaximize our reward we update the policy\nso that we will visit the place that is\nmore likely to give us a satisfying\norder of fries more\nfrequently triple\nbam now it's time for some Shameless\nself-promotion if you want to review\nstatistics\nmachine learning and AI offline check\nout the stack Quest PDF study guides and\nmy books the stack Quest Illustrated\nguide to machine learning and The\nstatquest Illustrated guide to neural\nnetworks and AI at stat quest.org\nthere's something for everyone hooray\nwe've made it to the end of another\nexciting stack Quest if you like this\nstack Quest and want to see more please\nsubscribe and if you want to support\nstack Quest consider contributing to my\npatreon campaign becoming a channel\nmember buying one or two of my original\nsongs or a t-shirt or a hoodie or just\ndonate the links are in the description\nbelow all right until next time Quest on",
  "transcript_chars": 13227,
  "ingested_at": "2026-05-15T10:53:13.639923+00:00",
  "source": "channel",
  "yt_meta": {
    "view_count": 94745,
    "like_count": 3194,
    "channel_id": "UCtYLUTtgS3k1Fg4y5tAhLbw",
    "categories": [
      "Education"
    ],
    "tags": [
      "Josh Starmer",
      "StatQuest",
      "Machine Learning",
      "Statistics",
      "Data Science"
    ]
  }
}