{
  "video_id": "C9QSpl5nmrY",
  "channel_slug": "statquest",
  "channel_handle": "statquest",
  "title": "Coding a ChatGPT Like Transformer From Scratch in PyTorch",
  "duration_seconds": 1871.0,
  "url": "https://www.youtube.com/watch?v=C9QSpl5nmrY",
  "upload_date": "",
  "transcript": "we're going to code a Transformer from\nscratch\nhooray stat\nQuest hello I'm Josh ster and welcome to\nstat Quest today we're going to talk\nabout coding Transformers from scratch\nin pi torch this stack Quest is brought\nto you by the letters a b and c a always\nBB C curious always be\ncurious I also want to use this\nopportunity to increase awareness of an\nawesome charity called give internet.org\na platform that makes it simple and\ntransparent for anyone to sponsor\ninternet access laptops and education\nfor underprivileged\nstudents note the code in this stack\nlist is available for free so use the\nlink and the pinned comment below to get\nyour own copy and follow along lastly in\nthis stack Quest we will be building a\nDE coder only Transformer which is the\nfoundation for chat GPT thus if you are\nnot already familiar with the concepts\nand Matrix math behind decoder only\nTransformers check out the\nquests the first thing we do is import\ntorch to create the tensors we will use\nto store the raw data and to provide a\nfew helper functions then we import\ntorch.nn for the module linear and\nembedding classes and a bunch of other\nhelper\nfunctions then we import torch. nn.\nfunctional to access the softmax\nfunction that we will use when\ncalculating attention then we import\natom to fit the neural network to the\ndata with back\npropagation and to give us the tools to\ncreate a large scale Transformer network\nwith lots of training data we import\ntensor data set and data loader from\ntorch. yous. data lastly we'll import\nlightning as L to make it way easier to\nwrite our code and for Auto automatic\ncode optimization and scaling in the\ncloud bam now let's create the training\ndata set that we will use to train our\ntransformer for this example all we want\nis for the Transformer to respond to two\ndifferent\nprompts what is stat Quest and Stat\nQuest is what and in both cases we want\nthe answer to be\nawesome thus the vocabulary consists of\nthe following tokens\nwhat is stack Quest awesome and\nEOS and we map the tokens to ID numbers\nbecause the p torch word embedding\nfunction that we will use in in.\nembedding only accepts numbers as\ninput and we save everything in a\ndictionary called token to\nID we then make a dictionary ID to token\nthat can go from ID numbers back to the\noriginal\ntokens these dictionaries will make it\neasy to format the input to the\nTransformer and interpret the output\nfrom the\nTransformer now let's talk about how to\nconvert the prompts and the responses\ninto a data set for example if the\nprompt is what is stat Quest and the\nresponse is\nawesome then what will be the first\ninput\ntoken and ideally since we want each\ntoken to generate what comes next the\nfirst output token will be\nis then we want to use is as the next\ninput token and we want that to Output\nstack\nQuest then we want stack quest to Output\nEOS to signify that we are done\nprocessing the input so these three\nsteps process the\nprompt however we're not yet done\ndetermining what the inputs and outputs\nshould be for the Transformer\nbecause we want the EOS token to Output\nawesome and we want awesome to generate\na second EOS to indicate that we are\ndone generating\noutput thus the tokens to use as input\nduring training come from processing The\nPrompt as well as from generating the\noutput likewise the outputs that we use\nduring training come from both\nstages thus if these are our inputs for\ntraining\nthen we can code an input tensor using\ntoken to ID like this what is stack\nQuest EOS\nawesome likewise if the prompt is stack\nQuest is what then we can code an input\ntensor using token to ID like this stack\nQuest is what EOS\nawesome now going back to the first\nprompt what is stack Quest when we\ngenerate the output from each decoder\nunit is stack Quest EOS awesome\nEOS we see we should code the label for\nthe first prompt like this is stat Quest\nEOS awesome\nEOS and we see we should code the label\nfor the second prompt like this is what\nEOS awesome EOS\ns ultimately we have the input and label\nfor the first prompt and\nresponse what is stack Quest\nawesome and the input and label for the\nsecond prompt and\nresponse stack Quest is what\nawesome now we just pass inputs and\nlabels to tensor data set to create a\ntensor data set object called data set\nand lastly we pass data set to data load\nto create a data loader object called\nData loader\nbam now we know how to create the inputs\nand labels for the training data set the\nnext part is word\nembedding and we're just going to let\nnn. edding take care of that for\nus so the next thing we need to talk\nabout is positioning\ncoding position en coding commonly uses\na sequence of alternating s and cosine\nsquiggles to calculate values for each\ntoken and embedding\nvalue this is the equation for the first\nsign\nsquiggle pause refers to the position or\nx-axis coordinate of the token in the\ninput this is the equation for the first\ncosine\nsquiggle this is the equation for the\nsecond s\nsquiggle the two indicates that this is\nthe second sign\nsquiggle and D model refers to the\nnumber of values we are using per\ntoken this is the equation for the\nsecond cosine\nsquiggle in general these are the\nequations we can use for as many tokens\nas we want with each token specified\nwith\npause and each embedding position\nspecified with\nI note this + one simply means that the\ncosine comes after the sign and doesn't\nchange the formula within the cosine\nfunction in in other words the formula\ninside the S function is identical to\nthe formula inside the cosine\nfunction now let's work through an\nexample so that we can see these\nequations in\naction if we had two\ntokens then for the first token pause\nequals\nzero and for the second token pause\nequals\n1 now if each token had four word\nembeddings then D model =\n4 then for the first token we calculate\nthe position and coding values by\nstarting with I equal 0 so we plug pause\nequals 0 and I equals 0 into the two\nequations we'll start with the sign\nfunction and we get\nzero so zero is the first position\nencoding\nvalue now let's evaluate the cosine\nfunction Bing and we get\none so one is the second position\nencoding\nvalue now because we still have two more\nword embedding values to add positional\nencoding to we increment I the word\nembedding index by one and then plug\npause equal 0 and I = 1 into the sign\nfunction and 0 is the third position\nencoding value then we plug pause equals\n0 and I = 1 into the cosine function and\none is the fourth position encoding\nvalue so I equals 0 is used for the\nfirst two position encoding values since\nwe have two functions s and cosine and I\nequal 1 is used for the second two\nposition encoding\nvalues for the next token we increment\nPause by one and reset I so that I\nequals z and and then calculate the\nfirst two position en coding values just\nlike\nbefore\nbam now that we have the first two\nposition encoding values for the second\ntoken we increment I by one and then\ncalculate the second two position\nencoding values\nbeoop boop boop\nbam and anyway rather than use these\nequations each time we want to add\npositional encoding to a\ntoken we precompute the y-axis values\nand store them in a matrix this makes\nadding position and coding values super\nfast here's the code that we'll use to\nprecompute and add position and coding\nvalues to the tokens we start by\ndefining a new class position and coding\nthat inherence from inn. module then\nlike all always we Define an init method\nD model which is short for dimension of\nthe model is the number of word\nembedding values per token and Max Len\nis the maximum number of tokens our\nTransformer can process input and output\ncombined note for this super simple\nexample we're setting D model to two and\nMax Len to six but in practice you would\nset them to much larger\nvalues then we call nn. modules andit\nmethod and now we start the code that\nwill create a matrix of position\nencoding\nvalues we start by creating a matrix\nthat we call PE for position en codings\nthat is full of\nzeros PE will have Max Len rows and D\nmodel\ncolumns for example if Max Len equals 3\nand D model equals 2 PE will start out\nlooking like this\nnow we create a column Matrix position\nthat represents the positions pause for\neach\ntoken we're using torch. a range to\ncreate a sequence of numbers between\nzero and Max Len and Float ensures that\nthe numbers are\nfloats and UNS squeeze one turns the\nsequence of numbers into a column Matrix\nfor example if Max Len equals 3 then we\nwill get this column Matrix\nnow we create a row Matrix embedding\nposition that represents the index I * 2\nfor each word\nembedding just like before we use torch.\na range to create a sequence of numbers\nbut this time they are between zero and\nD model note setting step equals to two\nresults in the same sequence of numbers\nthat we would get if we multiplied I by\ntwo so by setting step equal to two we\nsave ourselves a little math and just\nlike before we use float to ensure that\nthe numbers are\nfloats thus when D model equals 2 then\nembedding index is just a single value\nzero but when D model equals 6 then we\nend up with three\nvalues now each value in position is\ndivided by this term so we create div\nterm to represent the\ndivisor now we just do the math the\nfirst line assigns values from the sign\nfunction to The Matrix\nP starting with the First Column column\nzero and then this two means every other\ncolumn after that the second line\nassigns values from the cosine function\nto The Matrix\nPE starting with the second column\ncolumn one and then this two means every\nother column after that ultimately if we\nuse the default values Max Len equals 3\nand D models equals 2 PE will end up\nlooking like this where the First Column\nhas values from the sign\nfunction and the second column has\nvalues from the cosine\nfunction lastly we use register buffer\nto ensure that PE gets moved to a GPU if\nwe use one now we create a forward\nmethod that takes in word embedding\nvalues and adds the position and coding\nvalues to the word embedding\nvalues and that's all we have to do to\ncode positional\nencoding\nbam now that we have the position\nencoding taken care of let's talk about\nmasked self\nattention the first thing we need to do\nis calculate the query key and values\nfor each\ntoken and that means we need to code all\nof this math and in pi torch that means\nwe need to use Matrix\nnotation for example if we have the word\nembeddings plus position and coding\nvalues for what is and\nEOS then we can do all of the math\nrequired to create query values the\nmultiplication by these\nweights and the\nsummations by multiplying the encoded\nvalues for the tokens by a matrix that\ncontains the weights associated with\ncreating the\nqueries that that matrix multiplication\nwill give us a matrix of query values\nthat we'll call Q that has one row per\nencoded\ntoken likewise we can multiply the\nencoded tokens by a matrix containing\nthe weights associated with creating key\nvalues to create a matrix called K that\ncontains key values for each\ntoken lastly we can multiply the encoded\ntokens by a matrix containing the\nweights associated with creating value\nnumbers\nto create a matrix of values that we\nwill call\nV thus when we code an attention class\nin pi torch we replace this treel likee\ndiagram with matrix multiplication that\ncreates the queries keys and\nvalues so we start by defining a class\ncalled\nattention that inherits from inn.\nmodule then just like always we create\nan init method\nin this case we're passing the init\nmethod D model the dimension of the\nmodel or the number of word embedding\nvalues per token we need to know the\nnumber of word embedding values per\ntoken because that defines how large the\nweight matrices are that we use to\ncreate the queries keys and\nvalues in this specific example we're\nusing two word embedding values for each\ntoken so D model equals 2 and that means\neach weight Matrix needs D model equal\nto 2 rows and D model equal to 2\ncolumns so that we end up with D model\nequals to two query numbers per token\nbam the next thing we do is call the\nparents AIT\nmethod now in order to create the weight\nMatrix that we will use to calculate the\nquery values Q we will use nn. linear\nwhich will create the weight Matrix and\ndo the math for us n features defines\nhow many rows are in the weight Matrix\nso we set it to D model and out features\ndefines the number of columns in the\nweight Matrix so we set it to D model as\nwell lastly in the original Transformers\nmanuscript they don't add additional\nbias terms when calculating attention so\nwe won't either by setting bias equal to\nfalse as a result we end up with an\nobject we're calling WQ with the\ncurrently UNT trained weights needed to\ncalculate query values and because WQ is\na linear object it doesn't just store\nthe weights but it will also do the math\nfor us when the time comes then we do\nthe exact same thing to create a linear\nobject WK that contains the weights\nneeded to calculate the\nkeys lastly we create a linear object WV\nto calculate the\nvalues and just to give us flexibility\nto input put training data in\nsequentially or in batches we create\nsome variables to keep track of which\nindices are for rows and\ncolumns the forward method is where we\nactually calculate the masked self\nattention values for each\ntoken and another thing we're doing for\nthe sake of flexibility is allowing the\nquery key and values to be calculated\nfrom different token\nencodings for example encoder decoder\nTransformers have something called\nencoder decoder attention where the keys\nand values are calculated from the\nencoded tokens in the\nencoder and the queries are calculated\nfrom the encoded tokens in the\ndecoder so allowing the encodings to\ncome from different sources gives us the\nflexibility to do encoder decoder\nattention if we want to also since we\nwant to be able to do masked self\nattention we can pass in a mask now we\ncalculate the query key and values for\neach token by passing the encodings to\neach linear object and now we are ready\nto calculate a\ntension we start by using torch. matat\nmole to multiply Q by the transpose of K\nthis calculates the similarities between\nthe queries and the keys which we save\nin\nSims then we scale the similarities by\nthe square root of the number of values\nused in each key note this scaling is\nsomething that has been standard\npractice since the original Transformer\nmanuscript in\n2017 however it's not required and\nreally only helps out when the model is\nrelatively\nlarge the next thing we do is add the\nmask if we're using one to the scaled\nsimilarities masking is used to prevent\nearly tokens from cheating and looking\nahead at later\ntokens to understand how we add a mask\nusing the mask fill method let's imagine\nthat the mask is a matrix of trues and\nfalses and the trues correspond to\nattention values that we want to\nignore so the masted fill method\nreplaces the TRS with -1 * 10 to 9th\nwhich represents 1 billion an\napproximation of negative\ninfinity and it replaces the falses with\nzero to create the final mask that is\nadded to the scaled similarities in\nscaled Sims\nthe next thing we do to calculate\nattention is run the scaled similarities\nthrough a softmax\nfunction applying the softmax function\nto the scaled similarities determines\nthe percentages of influence that each\ntoken should have on the\nothers which is why we store the results\nin a variable called attention\npercents lastly we use torch. matat Mo\nto multiply the attention percentages by\nthe values in v and that gives us the\nfin final attention scores stored in\nattention\nscores which we return\nbam now that we have coded the attention\nclass we can create a class that puts\nthe first three steps\ntogether and then adds the residual\nconnections and then runs those values\nthrough a fully connected\nlayer and then runs them through a\nsoftmax to get the\noutputs and we do that by creating a\nclass called decoder only\nTransformer note unlike the position\nencoding and attention classes we\ncreated earlier this one inherits from\nlightning\nmodule doing it this way rather than\nhaving every class inherit from\nlightning module allows us to take\nadvantage of everything lightning offers\nwithout the overhead of inheriting it\nmultiple times anyway a first recreate\nthe init method which allows us to\nspecify numb tokens the number of tokens\nin the\nvocabulary D model the number of values\nwe want to represent each\ntoken and Max Len the maximum length of\nthe input plus\noutput then as always we call the\nparents a nit\nmethod then we create an embedding\nobject and we name it we for word\nembedding embedding needs to know how\nmany tokens are in the\nvocabulary and the number of values we\nwant to represent each token token then\nwe create a position en coding object\nusing the class we created earlier and\nname it\nPE and then we create an attention\nobject then we create the fully\nconnected layer with nn.\nlinear inn. linear needs to know how\nmany inputs there\nare and how many outputs there\nare then we create the loss function to\nquantify how well the model\nperforms in this case we're using cross\nentropy\nloss because our model has multiple\noutputs and cross entropy loss will\napply the softmax function for\nus now we put all the pieces together in\nthe forward method forward takes an\narray of token ID numbers that will be\nused as inputs to the\nTransformer first we convert the tokens\ninto word embedding\nvalues then we add the position encoding\nand then we create the mask that will\nprevent early tokens from looking at\nlate tokens when we calculate\nattention we start by creating a matrix\nof ones with torch.\nons for example if we pass four token\nIDs into this\nmethod then the call to torch. On's will\nmake a matrix with four rows and four\ncolumns full of\nones that Matrix of ones is then passed\nto torch. TRL\nand I believe that Tri L is pronounced\nTri L where L stands for lower\ntriangle because torch. Tri L leaves the\nvalues in the lower triangle as they are\nand turns everything else into\nzeros ultimately we save a matrix with\nones in the lower triangle and Zer in\nthe upper triangle in a variable called\nmask then we use mask equal equal 0 to\nconvert the Zer into truus and the ones\ninto\nfalses and in the end if we pass in four\ntoken IDs this is the mask we will use\nfor masked self\nattention once we have the mask we\ncalculate\nattention note because the query key and\nvalue matrices will all be calculated\nfrom the same token\nencodings we pass in the same set of\nposition encoded values three times for\nthe queries keys and\nvalues we also pass in The Mask so that\nearly tokens can't cheat and look ahead\nat later\ntokens and then we add the residual\nconnections and lastly run everything\nthrough a fully connected layer remember\nthe loss function we are using cross\nentropy loss does the soft Max for us so\nall we have to do is return the output\nfrom the fully connected layer\nbam now that we have coded a decoder\nonly\nTransformer we need to write the code\nrequired to train it so the next thing\nwe do is create a method to configure\nthe optimizer we're using in this case\nwe're using atom which is like\nstochastic gradient descent but a little\nless\nstochastic and we're passing all the\nweights and biases in the model that we\nwant to train which is all of them to\natom and in this example we're setting\nthe learning rate to 0.1 because it\nmakes training this specific model very\nfast however the default value\n0.001 is probably more commonly\nused then we code a training step method\nwhich takes a batch of training data and\nan index for that\nbatch we then split the training data\ninto inputs and\nlabels and then pass the input tokens\ninto the forward method that we just\nwrote to compute the\noutput and then we compare the output\nfrom the Transformer to the known labels\nusing the loss function and remember the\nloss function does the soft Max for us\nlastly we return the loss\nbam now let's run the model before\ntraining it just to see what it\ndoes so the first thing we do is create\na model from the decoder only\nTransformer class that we just\ncreated then we create an input\nprompt in this case we're using what is\nstat Quest\nEOS then we figure out how many tokens\nwe are using as\ninput we do this because our super\nsimple model can only handle a total of\nsix tokens so keeping track of how many\ntokens are in the input will tell us how\nmany we can create as\noutput then we run that through the\nTransformer which generates predictions\nfor each token in the\ninput this means that the model\ngenerates a prediction for what should\ncome after the first token\nwhat and for all of the other input\ntokens however we're really just\ninterested in what the model predicts\nwill come after the EOS token so we use\n-1 to index the outputs generated by the\nEOS token the outputs generated by the\nEOS token are an array of output values\none per possible output token so we use\nthe ARG Max function to identif the\noutput token with the largest\nvalue thus the token with the largest\noutput value will be the first token\ngenerated as a response to the\ninput note we don't have to take the\ntoken with the largest value if we don't\nwant and this is something that is often\nconfigured in more complicated\nmodels anyway we save that token so that\nwe can print it out later then we use a\nloop to keep generating output\ntokens until we reach the maximum number\nof tokens that our model can\ngenerate or the model generates the EOS\ntoken note each time we generate a new\noutput token we add it to the input so\nthat each prediction is made with the\nfull\ncontext and then the model predicts the\nnext output token using the full context\nwhich is the input plus the output token\nso\nfar lastly we print out the generated\ntokens after converting them from ID\nnumbers to\ntext now when we run this code the\nresponse to the input what is stack\nQuest EOS\nis\neos and that is not what we\nwanted ideally we would have gotten\nawesome\nEOS so that means we need to train the\nmodel the good news is that the code for\ntraining is super\nsimple first we create a lightning\ntrainer and tell it to only do 30 Epoch\nwhich is enough for our simple model and\ndata set and then we pass our model and\nthe data loader we created earlier to\nthe trainer using the fit\nmethod and after we train the model we\nrerun the code we wrote earlier that\ngenerates a response to the prompt what\nis stack Quest\nEOS however this time we don't create a\nnew model at the start we use the one\nthat we just trained anyway the output\nis awesome\nEOS\nbam and now let's see what happens when\nthe input prompt is stat Quest is what\nEOS using the same code we used earlier\nand the output is awesome\nEOS and that means our decoder only\nTransformer works exactly as expected\ntriple\nbam now with time for some Shameless\nself-promotion if you want to review\nstatistics and machine learning offline\ncheck out the stat Quest PDF study\nguides and my book the stat Quest\nIllustrated guide to machine learning at\nstack quest.org there's something for\neveryone hooray we've made it to the end\nof another exciting stack Quest if you\nlike this stack Quest and want to see\nmore please subscribe and if you want to\nsupport stack Quest consider\ncontributing to my patreon campaign\nbecoming a channel member buying one or\ntwo of my original songs or a t-shirt or\na hoodie or just donate the links are in\nthe description below all right until\nnext time Quest on",
  "transcript_chars": 23251,
  "ingested_at": "2026-05-15T10:53:41.408528+00:00",
  "source": "channel",
  "yt_meta": {
    "view_count": 113635,
    "like_count": 3114,
    "channel_id": "UCtYLUTtgS3k1Fg4y5tAhLbw",
    "categories": [
      "Education"
    ],
    "tags": [
      "Josh Starmer",
      "StatQuest",
      "Machine Learning",
      "Statistics",
      "Data Science"
    ]
  }
}