{
  "video_id": "bQ5BoolX9Ag",
  "channel_slug": "statquest",
  "channel_handle": "statquest",
  "title": "Decoder-Only Transformers, ChatGPTs specific Transformer, Clearly Explained!!!",
  "duration_seconds": 2205.0,
  "url": "https://www.youtube.com/watch?v=bQ5BoolX9Ag",
  "upload_date": "",
  "transcript": "decoding is all that you need stat\nQuest hello I'm Josh starmer and welcome\nto stack Quest today we're going to talk\nabout decoder only Transformers and\nthey're going to be clearly explained\ntrust me whatever Transformer you want\nto use it's better with\nlightning\nbam right now people are going going\ntotally bananas about chat\nGPT for example stat Squatch might type\nsomething into chat GPT like what is\nstat Quest stat Quest is awesome\nbam now in order to learn more about how\nchat GPT Works let's learn about decoder\nonly Transformers which is the specific\ntype of Transformer used for chat\nGPT note if you're not already familiar\nwith basic Transformers don't worry\nwe'll go through each Concept in a\ndecoder only Transformer one step at a\ntime that said if you are familiar with\nbasic Transformers then you might want\nto just skip to the last chapter normal\nTransformers versus decoder only\nTransformers anyway in this stack Quest\nwe're going to show how a decoder only\nTransformer can take a simple input\nprompt what is stat Quest and generate a\nsimple response awesome now since a\nTransformer is a type of neural network\nand neural networks usually only have\nnumbers for input values the first thing\nwe need to do is find a way to turn the\ninput and output words into numbers\nthere are a lot of ways to convert words\ninto numbers but for neural networks one\nof the most commonly used methods is\ncalled word embedding the main idea of\nword embedding is to use a relatively\nsimple neural network that has one input\nfor every word and symbol in the\nvocabulary that we want to use in this\ncase we have a super simple vocabulary\nthat allows us to input a short phrase\nlike what is stat Quest and we also have\nan input for a potential response\nawesome lastly we have an input for the\nEO symbol which stands for end of\nsentence or end of sequence because the\nvocabulary can be a mix of words word\nfragments and symbols we call each input\na token the inputs are then connected to\nsomething called an activation function\nand in this example we have two\nactivation functions and each connection\nmultiplies the input value by something\ncalled a weight hey Josh where do these\nnumbers come from great question Squatch\nand we'll answer it in just a bit for\nnow let's just see how we convert the\nword what into to numbers first we put a\none into the input for what and then Zer\ninto all of the other inputs now we\nmultiply the inputs by the weights on\nthe connections to the activation\nfunctions for example the input for what\nis 1 so we multiply\n-2.3 by 1 to get - 2.38 going to the\nactivation function on the left and we\nmultiply\n0.10 by 1 to get 0 .10 going to the\nactivation function on the right in\ncontrast if the input value for the word\nis is 0 then we multiply\n0.61 by 0 to get 0 going to the\nactivation function on the left and we\nmultiply\n0.17 by 0 to get 0 going to the\nactivation function on the right in\nother words when an input value is zero\nthen it only sends zeros to the\nactivation functions\nand that means is stat Quest awesome and\nthe EOS symbol all just send zeros to\nthe activation functions and only the\nweight values for what end up at the\nactivation functions because its input\nvalue is one so in this case - 2.38 goes\nto the activation function on the left\nand 0.10 goes to the activation function\non the right in this example the\nactivation functions themselves are just\nidentity functions meaning the output\nvalues are the same as the input values\nin other words if the input value or\nxaxis coordinate for the activation\nfunction on the left is\n-2.3 then the output value the Y AIS\ncoordinate will also be -\n2.38 likewise because the input to the\nactivation function on the right is\n0.10 the output is also 0.10\nthus these output values - 2.38 and\n0.10 are the numbers that represent the\nword what now before we move on I want\nto mention that all of these weights and\nall of the other weights we're going to\ntalk about in this Quest are determined\nusing something called back propagation\nto get a sense of what back propagation\ndoes let's imagine we had this data and\nwe wanted to fit a line to it back\npropagation would start with a line that\nhas a random value for the Y AIS\nintercept and a random value for the\nslope and then using an iterative\nprocess back propagation would change\nthe Y AIS intercept and slope one step\nat a time until it found the optimal\nvalues likewise in the context of neural\nnetworks each weight starts out as a\nrandom number but when we train the\nTransformer by getting it to predict\nwords in known documents back\npropagation opt optimizes these values\none step at a time and results in these\nfinal weights also just to be clear the\nprocess of optimizing the weights is\nalso called training bam note there's a\nlot more to be said about training and\nback propagation so if you're interested\ncheck out the quests anyway we now know\nhow to calculate the word embedding\nvalues for the first word in our phrase\nwhat so we'll keep track of those values\nwith this diagram in the upper right\nhand corner corner now we reuse the\nexact same word embedding Network to\nconvert the remaining words in the\nprompt what is stat Quest into numbers\nthus the word embedding values for is\nare 0.61 and\n0.17 and the word embedding values for\nstat Quest are - 2.38 and\n0.10 and now we have converted all of\nthe words in the prompt what is stack\nQuest into word embedding values note\nreusing the exact same word embedding\nNetwork for each word in the prompt\nallows the decoder only Transformer to\nhandle prompts that have different\nlengths because we can just copy the\nnetwork as many times as we need to\nconvert all the words in the prompt into\nnumbers also note there's a lot more to\nsay about word embedding so if you're\ninterested check out the quest anyway\nnow that we know how to convert words\ninto numbers let's talk about word order\nfor example if Norm said Squatch eats\npizza then Squatch might say\nyum in contrast if Norm said Pizza eats\nSquatch then Squatch might say yikes so\nthese two phrases Squatch eats pizza and\npizza eats Squatch use the exact same\nwords but have very different meanings\nso keeping track of word order is super\nimportant so let's talk about positional\nencoding which is a technique that\nTransformers use to keep track of word\norder although there are several ways to\nkeep track of word order with positional\nencoding one of the most commonly used\nmethods uses a sequence of alternating s\nand cosine squiggles each squiggle gives\nus specific position values for each\nwords embeddings so now let's see how we\ncan add positional en coding values to\nthe word embeddings we created for the\nprompt what is stat Quest the first word\nwhich in this case is what has an xais\ncoordinate all the way to the left on\nthe green squiggle and the position\nvalue for its first embedding is the Y\nAIS coordinate zero the position value\nfor the second embedding comes from the\norange squiggle and the y- AIS\ncoordinate on the orange squiggle that\ncorresponds to the first word is one now\nto get the position ition values for the\nsecond word is we simply use the Y AIS\ncoordinates on the squiggles that\ncorrespond to the x-axis coordinate for\nthe second word likewise we use the\ncorresponding y AIS coordinates for the\nthird word stat Quest note because the S\nand cosine squiggles are repetitive it's\npossible that two words might get the\nsame position or Y AIS values however\nbecause the squiggles get wider for\nlarger embedding position\nand the more embedding values we have\nthen the wider the squiggles get then\neven with a repeat value here and there\nwe end up with a unique sequence of\nposition values for each word anyway now\nwe just do the math to get the\npositional encoding for all three input\nwords and we end up with the word\nembeddings plus positional encoding for\nthe prompt what is\nstatquest now because we're going to\nneed all the space we can get let's\nconsolidate the math in the diagrams and\nlet the S and cosine and plus symbols\nrepresent the positional encoding now\nthat we know how to keep track of each\nword's position let's talk about how a\ndecoder only Transformer keeps track of\nthe relationships among words for\nexample if the prompt was the pizza came\nout of the oven and it tasted good then\nthis word it could refer to pizza or or\npotentially it could refer to the word\noven Josh I've heard of good tasting\npizza but never a good tasting oven I\nknow Squatch that's why it's super\nimportant that the decoder only\nTransformer correctly Associates the\nword it with pizza the good news is that\ndecoder only Transformers have something\ncalled masked self- attention which can\nhelp correctly associate the word it\nwith the word Pizza in general terms\nmasks self attention works by seeing how\nsimilar each word is to itself and all\nof the preceding words in the sentence\nfor example masked self attention starts\nby calculating the similarity between\nthe first word the and itself then\nmasked self- attention calculates the\nsimilarity between pizza and itself and\nthe preceding\nwordthe and then masked self attention\njust keeps calculating similarities like\nthis allowing each word to look at\nitself and the words that came before it\nbut not after until it gets to the end\nof the input once the similarities are\ncalculated they are used to determine\nhow the decoder only Transformer encodes\neach word for example if you looked at a\nlot of sentences about pizza and the\nword it was more commonly associated\nwith pizza than oven then the similarity\nscore for pizza will cause it to have a\nlarger impact on how the word it is\nencoded by the decoder only Transformer\noh no it's the dreaded terminology alert\nbecause masked self attention only\nallows access to the words that come\nbefore it and not the words that come\nafter it is sometimes called an auto\nregressive method anyway now that we\nknow the main ideas of how masked self\nattention Works let's look at the\ndetails so let's go back to our simple\nexample where we had just added\npositional encoding to The Prompt what\nis stat Quest now since what is the\nfirst word in the prompt its masked self\nattention values will only reflect its\nsimilarity to itself and ignore\neverything else in contrast the masked\nself attention values for the second\nword is reflect the similarity to itself\nas well as a similarity with the first\nword what and the last word stat Quest\ntakes into account its similarity with\nitself and everything that came before\nit now I know that what is the first\nword in the input but it will make it\neasier to understand how masked self\nattention works if we start with the\nsecond word is so let's move the\nposition encoded is over a little bit to\ngive us some room the first thing we do\nto calculate the masked self attention\nfor the word is is multiply its position\nencoded values by a pair of weights and\nthen add those products together to get\nNegative\n2.4 then we do the same thing with a\ndifferent pair of weights to get\n2.6 we do this twice because we started\nout with two position encoded values\nthat represent the word is and after\ndoing the math two times we still have\ntwo values representing the word is Josh\nI don't get it if we want two values to\nrepresent is why don't we just use the\ntwo values we started with that's a\ngreat question Squatch and we'll answer\nit in a little bit\nG anyway for now just know that we have\ntwo new values to represent the word is\nand in Transformer terminology we call\nthem query numbers and now we're going\nto use the query numbers for is to\ncalculate the similarities with itself\nand the first word what and we do that\nby creating two new numbers like we did\nbefore to represent the word is\nand creating two new numbers to\nrepresent the word what in Transformer\nterminology both sets of new numbers are\ncalled key values and we use them to\ncalculate similarities with the query\nfor is one way to calculate similarities\nbetween the query and the keys is to\ncalculate something called a dotproduct\nfor example in order to calculate the\ndot product similarity between the query\nand key for is we simply multiply each\npair of numbers together and add the\nproducts to get\n5.9 likewise we can calculate the dot\nproduct similarity between the query for\nis and the key for what by multiplying\nthe pairs of numbers together and adding\nthe products to get -\n25.7 the small similarity value for what\nrelative to is -\n25.7 compared to the large similarity\nvalue for is relative to itself\n5.9 tells us that is is much more\nsimilar to itself than it is to the word\nwhat that said if you remember the\nexample where the word it could relate\nto pizza or oven then the word it should\nhave a relatively large similarity value\nwith respect to the word Pizza since it\nrefers to pizza and not oven note\nthere's a lot to be said about\ncalculating similarities in this context\nand the dot product so if you're\ninterested check out the quests anyway\nsince is is much more similar to itself\nthan it is to the word what then we want\nis to have more influence on its\nencoding than the word what and we do\nthis by first running the similarity\nscores through something called a\nsoftmax function the main idea of a\nsoftmax function is that it preserves\nthe order of the input values from low\nto high and translates them into numbers\nbetween 0o and one that add up to one so\nwe can think of the output of the soft s\nMax function as a way to determine what\npercentage of each input word we should\nuse to encode the word is in this case\nbecause is is so much more similar to\nitself than the word what we'll use 100%\nof the word is to encode is and 0% of\nthe word what to encode the word is note\nthere's a lot more to be said about the\nsoft Max function so if you're\ninterested check out the quest anyway\nbecause we want 100% of the word is to\nencode is and 0% of the word what to\nencode is we create two more numbers\nthat will cleverly call values to\nrepresent the word what and scale them\nby\n0.0 then we create two value numbers to\nrepresent the word is and scale them by\n1.0 lastly we add the scaled values\ntogether and these sums which combine\nseparate encodings for both input words\nwhat and is relative to their similarity\nto is are the masked self- attention\nvalues for is Bam now that we have the\nmasked self attention values for is we\ncan go back and calculate them for the\nfirst word what remember because what is\nthe first word it only needs to know how\nsimilar it is to itself so we can get\nrid of most of what we did for the word\nis but keep the key and value numbers\nthat we calculated for the word what now\nso we have a little more room to work\nlet's move everything over a bit and\njust like we did before we create a\nquery for the word what now we use the\nquery and key for what to calculate the\nsimilarity with itself and we get\n-12.4 so we plug 12.4 into the soft Max\nfunction and we get\n1.0 last ly we scale the value numbers\nfor what by\n1.0 and -2.9 and -1.3 are the mask self\nattention values for the first word what\nit might seem a little silly to do all\nthis math just to end up using the value\nnumbers for what as the masked self\nattention numbers for what however doing\nit this way gives us a unified method\nfor calculating masked self attention so\nso far we've calculated the masked self\nattention values for what which only\nrequired taking its own value numbers\ninto account and we calculated the mass\nself attention for is which required\ntaking the value numbers from what and\nis into account now we need to calculate\nthe masked self attention for the third\nword stat Quest and that means we need\nto take the value numbers for stat Quest\ninto account which we calculate like\nbefore and we need to take the value\nnumbers for For What and is into account\nso the first thing we do is calculate\nthe query numbers for the word stat\nQuest and then calculate its key numbers\nand then we bring back the keys for what\nand is and calculate the similarities\nbetween the query for stat Quest and the\nkeys for stat Quest is and what now we\nrun all three similarities into the soft\nMax function and the output from the\nsoft Max tells us what percentage of\neach word's value numbers to use when\ncalculating the massed self attention\nfor stat Quest now we bring back the\nvalue numbers that we calculated earlier\nfor what is and Stat Quest and scale\nthem based on the similarity scores\nlastly we add the pairs of scaled values\ntogether to get the masked self\nattention numbers for the word stat\nQuest\nbam note before we move on I want to\npoint out that we read use one set of\nweights to create query numbers for each\nword in other words the set of Weights\nwe use to create the query numbers for\nwhat is the same set of Weights we use\nfor the query numbers for is in stat\nQuest likewise the key numbers are\ncalculated with a different set of\nWeights that are shared for each word\nand the value numbers are also\ncalculated with another set of Weights\nthat are reused for each word reusing\nthe sets of weights for the the query\nkey and value numbers lets the decoder\nonly Transformer handle prompts that\nhave different lengths because we can\njust keep reusing the weights as many\ntimes as we need now that we understand\nthe details of how masked self attention\nWorks let's shrink the diagram so that\nwe can keep building our decoder only\nTransformer bam Josh you forgot\nsomething if we want two values to\nrepresent what why don't we just use the\ntwo positioning encoded values we\nstarted with first the new masked self\nattention values for each word contain\ninput from all of the other words that\ncame earlier and this helps give each\nword context and this can help establish\nhow each word in the input prompt is\nrelated to the others also if we can\nthink of this unit with its three sets\nof weights for calculating queries keys\nand values as a masked self attention\ncell then in order to correctly\nestablish how words are related in\ncomplicated sentences and paragraphs we\ncan create a stack of masked self\nattention cells each with its own sets\nof Weights that we apply to the position\nencoded values for each word to capture\ndifferent relationships among the words\nin the manuscript that first described\nthe original GPT they stacked 12 masked\nself attention cells\nbam okay going back to our simple\nexample with only one masked self self\nattention cell there are a few more\nthings we need to do before we start\ngenerating a response to the prompt what\nis stat Quest First We Take the position\nencoded values and add them to the self\nattention values these bypasses are\ncalled residual connections and they\nmake it easier to train complex neural\nnetworks by allowing the masked self-\nattention layer to establish\nrelationships among the input words\nwithout having to also preserve the word\nembed and position encoding information\nbam lastly we need a way to use the\nencodings we have for each word in the\nprompt to generate the word that follows\nit and then generate a response in other\nwords we want to use these two numbers\nthat represent the word what to generate\nthe word that comes after what and we\nwant these two numbers that represent\nthe word is to generate the word that\ncomes after\nis lastly we want these two numbers that\nrepresent the word stat quest to\ngenerate the word that comes after stat\nQuest hey Josh I don't get it why do we\nwant to generate the word that comes\nafter what when we already know the next\nword is because this is a decoder only\nTransformer we need one thing that can\nboth encode The Prompt and generate the\noutput thus even though we are not yet\ngenerating a response we need to include\nthe parts that will do it\nalso we can compare the known input to\nwhat the model generates when we train\nthe model so now let's see what the\nmodel generates given the first word in\nthe prompt what and that means taking\nthese two values - 5.28 and\n0.2 and plugging them into something\ncalled a fully connected layer this\nfully connected layer has one input for\neach value that represents the current\ntoken so in this case we have two in\ninputs and one output for each token in\nthe vocabulary which in this case means\nfive outputs note a fully connected\nlayer is just a simple neural network\nwith weights numbers we multiply the\ninputs by and biases numbers we add to\nthe sums of the products also note in\nthe original GPT manuscript instead of a\nnew fully connected layer they use the\nword embedding Network that we started\nwith but in Reverse in other words they\njust reuse the same set of Weights that\nwe use to encode the words into numbers\nand then flip them to help decode the\nnumbers however not all decoder only\nTransformers do it this way and a very\ncommon alternative is a fully connected\nlayer small bam now when we do the math\nwe get five output values which we run\nthrough a final soft Max function to\ngenerate the next word is and since the\nprompt was what is stat Quest the model\ngenerated the correct word\nbam the second word in the prompt is is\nand it generates the word what which is\nnot correct w\nw however the correct word stat Quest\nwas almost correctly generated so we\nhope our model doesn't feel too much\nshame note if we were training the\ndecoder only Transformer then we would\nuse the fact that we made a mistake to\nmodify the weights and biases in\ncontrast when we are just using the\nmodel to generate responses then it\nreally doesn't matter what words come\nout right now so in this case we'll just\nnote that we made a mistake and move on\nlastly the word stat Quest generates the\nEOS token which is correct since we are\nat the end of The Prompt\nbam now let's review what we've done so\nfar we started with an input prompt what\nis stat Quest and we used word embedding\nto convert each word into numbers and we\nused positional encoding to keep track\nof word order in the prompt then we\nadded masked self attention to determine\nthe relationships among the words in the\nprompt then we added residual\nconnections to make it easier to train\nthe model lastly we added a fully\nconnected layer and a soft Max to\ngenerate the next word\neach part of the decoder only\nTransformer is reused so that it can\nhandle prompts of different lengths and\nthe encoding for each word in the prompt\ncan happen at the same time rather than\nsequentially and thus can be done\nquickly when multiple Computing cores\nare available lastly generating the\noutput uses the exact same steps that we\nuse to encode The Prompt thus generating\nthe output starts with word embedding\nand it uses the exact same word EMB set\nnetwork that we use to encode The Prompt\nand because we just finished encoding\nThe Prompt we start generating the\noutput from the EOS token in this case\nwe're using the EOS token to start\ngenerating the output because that is a\ncommon way to initialize this process\nhowever sometimes you'll see people use\nSOS for start of sentence or start of\nsequence to initialize the process Josh\nstarting with SOS makes more sense to me\nthen you can do it that way Squatch I'm\njust saying that a lot of people start\nwith\nEOS also note we will start with the EOS\ntoken regardless of whether or not it\nwas the last token generated when we\nencoded The Prompt anyway we plug in one\nfor Eos and zero for everything else and\nwe end up with the numbers that\nrepresent the EOS token now let's shrink\nthe word embedding down to make more\nspace so that we can add positional\nencoding note note the EOS token comes\nafter the three tokens that represent\nthe input so it's in the fourth position\nand since the EOS token is in the fourth\nposition with two embeddings we just add\nthose two position values and before we\nmove on we need to say a few more words\nabout masked self attention so far we've\ntalked about how masked self- attention\nhelps the decoder only Transformer keep\ntrack of how words are related within\nthe\ninput however it's also important to\nkeep track of the relationships between\nthe input sentence and the output for\nexample if the input sentence was don't\neat the delicious looking and smelling\npizza then when generating new output it\nis super important to keep track of the\nvery first word don't if we focus on\nother parts of the sentence and omit the\ndon't then we'll end up with eat the\ndelicious looking and smelling pizza and\nthese two sentences have completely\nopposite meanings so it's super\nimportant that when generating the\noutput we keep track of the significant\nwords in the input the nice thing is\nthat all we have to do to add this\nability to our decoder only Transformer\nis just include the prompt when we do\nmasked self attention while generating\nthe output so the first thing we do is\ncalculate the query numbers for the EOS\ntoken and then we calculate the key\nnumbers using the same sets of Weights\nwe used for the prompt and then we bring\nback the key values for the prompt what\nis stat Quest then we calculate the\nsimilarities between the query and the\nkeys and run everything through the\nsoftmax function now we calculate the\nvalue numbers for the EOS token and\nbring back all of the value numbers we\ncalculated earlier and scale all of them\nbased on the similarity scores lastly we\nadd the pairs of scaled values together\nto get the masked self attention values\nfor the EOS token bam anyway now that we\nhave the masked self attention values\nfor the EOS token we add the residual\nconnections Now we move the diagram to\nthe left and run the numbers that\nrepresent the EOS token through the same\nfully connected layer we used earlier\nand the same soft Max function we used\nbefore and the first word generated by\nour decoder only Transformer is awesome\nwhich is totally awesome however even\nthough the output is awesome we're not\ndone yet because the decoder only\nTransformer will keep generating output\nuntil it generates the EOS token so we\nplug the word we just generated awesome\ninto another copy of the word embedding\nlayer then we add the positional\nencoding using the values for the fifth\nposition because now the sequence is\nwhat what is stat Quest EOS awesome now\nwe calculate the masked self attention\n[Music]\nvalues\nbam now that we have the masked self\nattention values for awesome we add the\nresidual\nconnections Now we move the diagram to\nthe left and run the numbers that\nrepresent awesome through the same fully\nconnected layer we used earlier and the\nsame softmax function we used before and\nour decoder only Transformer generates\nthe EOS token which means we are done\ngenerating output double\nbam now let's talk about the differences\nbetween the decoder only Transformer\nthat we just learned about and a basic\nTransformer as we just saw a decoder\nonly Transformer uses the exact same\ncomponents to encode The Prompt that it\nuses to generate the output and it uses\nmasked self attention which is\ncalculated with only the current word\nand everything that preceded it and the\nmasked self attention is applied equally\nto the input prompt and to the output\nthat is generated masked self attention\nallows a decoder only Transformer to\ndetermine how words in the prompt are\nrelated and make sure that it keeps\ntrack of important input words when\ngenerating the output in contrast a\nregular Transformer uses one type of\nunit called the encoder to encode The\nPrompt and a different type of unit\ncalled the decoder to generate the\noutput when encoding the input prompt\ninstead of using masked self attention a\nnormal Transformer uses self attention\nwhich includes all of the words in the\ninput not just the ones that came before\nto determine how the words are related\nto each other and a normal Transformer\nuses encoder decoder attention to let\nthe decoder keep track of important\nwords in the input encoder decoder\nattention uses queries from the decoder\nbut only keys and values from the\nencoder now so far we've only talked\nabout how attension is used in a normal\nTransformer during inference when it\nencodes the input and generates new\noutput however during training a normal\nTransformer uses masked self attention\nin the decoder for example during\ntraining we know the output should be\nawesome\nEOS which means we don't have to decode\nthe initial EOS before we decode awesome\nlike we do when we are generating new\noutput instead since we know we will be\ndecoding awesome we can do its math at\nthe same time we do the math for the EOS\ntoken doing the math at the same time\nmeans we can train faster and thus\nduring training a normal Transformer\nwill use masked self attention on the\ntokens in the known output this allows\nthe Transformer to learn how to generate\nthe correct output without cheating and\nlooking ahead note when we are training\na normal Transformer the masked self\nattention only includes the output\ntokens in contrast a decoder only\nTransformer uses massed self attention\nall of the time not just during training\nand it includes the input and the output\nso the three big differences between a\nnormal Transformer and a decoder only\nTransformer are a normal Transformer\nuses one unit to encode the input called\nthe encoder and a separate unit to\ngenerate the output called the decoder\nand a normal Transformer uses two types\nof attention during inference self\nattention and encoder decoder attention\nlastly during training a normal\nTransformer uses masked self attention\nbut only on the output in contrast a\ndecoder only Transformer has a single\nunit for both encoding the input and\ngenerating the output and a decoder only\nTransformer uses a single type of\nattention masked self attention and a\ndecoder only Transformer uses masked\nself attention all the time on\neverything the input and the output\ntriple\nbam note there's a lot more to say about\nnormal encoder decoder Transformers so\nif you're interested check out the quest\nnow it's time for some Shameless\nself-promotion if you want to review\nstatistics and machine learning offline\ncheck out the stat Quest PDF study\nguides in my book the stat Quest\nIllustrated guide to machine learning at\nstat quest.org there's something for\neveryone hooray we've made it to the end\nof another exciting stack Quest if you\nlike this stack Quest and want to see\nmore please subscribe\nand if you want to support stat Quest\nconsider contributing to my patreon\ncampaign becoming a channel member\nbuying one or two of my original songs\nor a t-shirt or a hoodie or just donate\nthe links are in the description below\nall right until next time Quest on",
  "transcript_chars": 30301,
  "ingested_at": "2026-05-15T10:55:12.532682+00:00",
  "source": "channel",
  "yt_meta": {
    "view_count": 225059,
    "like_count": 5143,
    "channel_id": "UCtYLUTtgS3k1Fg4y5tAhLbw",
    "categories": [
      "Education"
    ],
    "tags": [
      "Josh Starmer",
      "StatQuest",
      "Machine Learning",
      "Statistics",
      "Data Science",
      "ChatGTP",
      "Transformers",
      "AI",
      "LLM",
      "Large Language Models"
    ]
  }
}