{
  "video_id": "bCz4OMemCcA",
  "channel_slug": "umarjamilai",
  "channel_handle": "umarjamilai",
  "title": "Attention is all you need (Transformer) - Model explanation (including math), Inference and Training",
  "duration_seconds": 3484.0,
  "url": "https://www.youtube.com/watch?v=bCz4OMemCcA",
  "upload_date": "",
  "transcript": "hello guys welcome to my video about the\nTransformer and this is actually the\nperson 2.0 of my series on the\nTransformer I had a previous video in\nwhich I talked about the Transformer but\nthe audio quality was not good and as\nsuggested by my viewers as the video was\nreally uh had a huge success the viewers\nsuggested me to to improve their audio\nquality so this this is why I'm doing\nthis video\nuh you don't have to watch the previous\nseries because I would be doing\nbasically the same things but with some\nimprovements so I'm actually\ncompensating from some mistakes I made\nor from some improvements that I could\nadd\nafter watching this video I suggest\nwatch my watching my other video about\nor how to code a Transformer model from\nscratch so how to code the model itself\nhow to train it online data and how to\ninference it stick it with me because\nit's gonna be a little long journey but\nfor sure what\nnow before we talk about the Transformer\nI want to first talk about recurrent\nneural networks so the networks that\nwere used before they introduced the\ntransformer for most of the sequence to\nsequence jobs tasks so let's review them\nrecurring neural networks existed a long\ntime before the Transformer and they\nallowed to map one sequence of input to\nanother sequence of output in this case\nour input is X and we want an input\nsequence Y what we did before is that we\nsplit the sequence into single items so\nwe gave the recurrent neural network the\nfirst item as input so X1 along with an\ninitial State usually made up of only\nzeros and the recurrent normal Network\nproduced an output let's call it y1\nand this happened at the first time step\nthen we took the hidden State this is\ncalled the hidden state of the network\nof the previous time step along with the\nnext input token so X2 and the network\nhad to produce the SEC the second output\ntoken Y2 and then we did it the same\nprocedure at the third time step in\nwhich we took the hidden state of the\nprevious time step along with the input\nState the input token at the time steps\n3 and the network has to produce the\nnext output token which is Y3 if you\nhave enter n tokens you need n time\nsteps to map a end sequence input into\nan end sequence output\nthis worked fine for a lot of tasks but\nhad some problems let's review them\nthe problems with recurring neural\nnetworks first of all are that they are\nslow for long sequences because think of\nthe process we did before we have kind\nof like a for Loop in which we do the\nsame operation for every token in the\ninput so if you have the longer the\nsequence the longer this computation and\nthis made the the network not easy to\ntrain for long sequences the second\nproblem was the vanishing or the\nexploding gradients now you may have\nheard these terms or expression on the\nInternet or from other videos but I will\ntry to give you a brief Insight on what\ndoes what do they mean on a practical\nlevel so as you know\nFrameworks like Pi torch they convert\nour networks into a computation graph so\nbasically suppose we have a computation\ngraph I this is not an error network I\nwill making I will be making a\ncomputational graph that is very simple\nhas nothing to do with the neural\nnetworks but will show you the problems\nthat we have so imagine we have two\ninputs X and another input let's call it\ny\nour computational graph first let's say\nmultiplies these two numbers so we have\na first a function let's call it f of x\nand y\nthat is X multiplied by y\nlet me multiplied and the result let's\ncall it Z\nis map is given to another function\nlet's call this function G of Z is equal\nto let's say Z squared\nwhat our phytorch for example does it's\nthat pytorch want to calculate the\nusually we have a loss function by torch\ncalculates the derivative of the loss\nfunction with respects to its each\nweight in this case we just calculate\nthe derivative of the G function so the\noutput function with respect to all of\nits inputs so derivative of G\nwith respect to X let's say is equal to\nthe derivative of G with respect to f\nand multiplied by the derivative of f\nwith respect to X\nthese two should kind of cancel out this\nis called the chain Rule now as you can\nsee\nthe longer the chain of computation so\nif we have many nodes one after another\nthe longer this multiplication chain so\nhere we have two because the distance\nfrom this node and this is two but\nimagine you have 100 or 1000\nnow imagine this number is 0.5 and this\nnumber is 0.5 also the resulting numbers\nwhen multiplied together is a number\nthat is smaller than the two initial\nnumbers it's gone up 0.25 because it's\none to one half multiplied by one half\nis one fourth\nso if we have two numbers that are\nsmaller than one and we multiply them\ntogether they will produce an even\nsmaller number and if we have two\nnumbers that are bigger than one and we\nmultiply them together they will produce\na number that is bigger than both of\nthem so if we have a very long chain of\ncomputation it eventually will either\nbecome a very big number or a very small\nnumber\nand this is not desirable first of all\nbecause our CPU of our GPU can only\nrepresent numbers up to a certain\nPrecision let's say 32-bit or 64-bit and\nif the number becomes too small the\ncontribution of this number to the\noutput will become very small so when\nthe pi torch or our automatic let's say\nour framework will calculate how to\nadjust the weights the weight will move\nvery very very slowly because the\ncontribution of this product is will be\na very small number\nand this means that we have the gradient\nis Vanishing or in the other case it can\nexplode become very big numbers\nand this is a problem the next problem\nis difficulty in accessing information\nfrom long time ago\nwhat does it mean it means that as you\nremember from the previous slide we saw\nthat the first input token is given to\nthe recurrent neural network to with\nalong with the first state\nnow we need to think that the recurrent\nneural network is a long graph of\ncomputation it will produce a new hidden\nState then we will use the the new\nhidden State along with the next token\nto produce the next output if we have a\nvery long sequence\num of input sequence the last token will\nhave a hidden state whose contribution\nfrom the first token has nearly gone\nbecause of this long chain of\nmultiplication so actually the last\ntoken will not depend much on the first\ntoken and this is also not good because\nfor example we know as humans that in a\ntext in a quite long text the context\nthat we saw let's say 200 words before\nstill relevant to the context of the\ncurrent words and this is something that\nthe RNN could not map\nand this is why we have the Transformer\nso the Transformer solves these problems\nwith the recurrent neural networks and\nwe will see how\nthe structure of the Transformer we can\ndivide into two macro blocks the first\nmacro block is called encoder and it's\nthis part here\nthe second macro block is called a\ndecoder and it's the second part here\nthe third part here you see on the top\nit's just a linear layer and we will see\nwhy it's there and what it is function\nso and the two layers so the encoder and\nthe decoder are connected by this\nconnection you can see here\nin which some output of the encoder is\nsent as input to the decoder and we will\nalso see how let's start first of all\nwith some notations that I will be using\nduring my explanation and you should be\nfamiliar with this notation also to\nreview some maths so the first thing we\nshould be familiar with is matrix\nmultiplication so imagine we have a\ninput Matrix\nwhich is a sequence of let's say words\nso sequence by D model and we will see\nwhy it's called sequence by the model so\nimagine we have a matrix that is a 6 by\n512 in which each row is a word\nand this word is not made of characters\nbut by 512 numbers so each word is\nrepresented by\n512 numbers okay like this imagine you\nhave 512 of them along this row 512\nalong this other row etc etc one two\nthree four five so we need another one\nhere okay the first word we will call it\na the second B the C D E and F\nif we multiply this matrix by another\nMatrix let's say the transpose of this\nMatrix so it's a matrix where the rows\nbecomes columns\nso three\nfour\nfive and\nsix\nthis word will be here B C D E and F and\nthen we have\num\n512 numbers along each column because\nbefore we had them on the rows now they\nwill become on the column so here we\nhave the 512 number\netc etc\nthis is a matrix that is\n512 by 6 so let me add some brackets\nhere if we multiply them we will get a\nnew Matrix that is we cancel the inner\ndimensions and we get the outer\nDimension so it will become six by six\nso it will be 6 rows by 6 rows so let's\ndraw it\nhow do we calculate the values of this\noutput Matrix this is six by six\nthis is the dot product of the first row\nwith the First Column so this is a\nmultiplied by a this the second value is\nthe first row with the second column the\nthird value is the first row with the\nthird column\nuntil the last column so a multiplied by\nF Etc what is the dot product is\nbasically you take the first number of\nthe first row so here we have 512\nnumbers here we have 512 numbers so you\ntake the first number of the first row\nand the first number of the First Column\nyou multiply them together\nsecond value of the first row second\nvalue of the First Column you multiply\nthem together and then you add all these\nnumbers together so it will be let's say\nuh this number multiplied by this plus\nthis number multiplied by this plus this\nnumber multiplied by this plus this\nnumber multiplied by this plus you sum\nall this number together and this is the\na DOT product a so we should be familiar\nwith this notation because I will be\nusing it a lot in the next slides let's\nstart our journey with of the\nTransformer uh by looking at the encoder\nso the encoder\nstarts with the input embeddings so what\nis an input embedding\nfirst of all let's start with our\nsentence we have a sentence of in this\ncase six words what we do is we tokenize\nit we transform the sentence into tokens\nwhat does it mean to tokenize we split\nthem into single words\nit is not necessary to always split the\nsentence using single words we can even\nsplit the sentence in part in smaller\nparts that are even smaller than a\nsingle word so we could even split this\na sentence into let's say 20 tokens by\nusing the each by splitting each word\ninto multiple words this is usually done\nin most modern\nTransformer models but we will not be\ndoing it otherwise it's really difficult\nto visualize so let's suppose we have\nthis input sentence and we split into\ntokens and each token is a single word\nthe next step we do is we map these\nwords into numbers\nand these numbers represent the position\nof these words in our vocabulary so\nimagine we have a vocabulary of all the\npossible words that appear in our\ntraining set each word will occupy a\nposition in this vocabulary so for\nexample the word will occupy the\nposition 105 the word the cat will\noccupy the position 6500\nEtc\nand as you can see this cat here has the\nsame number as this cat here because\nthey occupy the same position in the\nvocabulary\nwe take these numbers which are called\ninput IDs and we map them into a vector\nof size 512.\nthis Vector is a vector made of 512\nnumbers\nand we always map the same word to\nalways the same embedding\nhowever this number is not fixed it's a\nparameter for our model so our model\nwill learn to change these numbers in\nsuch a way that it represents the\nmeaning of the word so the input ID is\nnever change because our vocabulary is\nfixed but the embedding will change\nalong with the training process of the\nmodel so the embeddings numbers will\nchange according to the needs of the\nloss function so the input embedding are\nbasically mapping our single word into\nan embedding of size 512 and we call\nthis quantity 512 D model because it's\nthe same name that it's also used in the\npaper attention is all you need\nlet's look at the next layer of the\nencoder which is the positional encoding\nso what is positional encoding\nwhat we want is that each word should\ncarry some information about its\nposition in the sentence because now we\nbuilt a matrix of words that are\nembeddings but they don't convey any\ninformation about how where that\nparticular word is inside the sentence\nand this is the job of the positional\nencoding so what we do\nwe want the model to treat words that\nappear close to each other as close and\nwords that are distant as distant so we\nwant the model to see this information\nabout the special information that we\nsee with our eyes so for example when we\nsee this sentence what is positional\nencoding we know that the word what is\nmore far from the word\num is compared to encoding because we we\nhave this partial information given by\nour eyes but the model cannot see this\nso we need to give some information to\nthe model about how the words are\nspecially distributed inside of the\nsentence\nand we want the positional encoding to\nrepresent a pattern that the model can\nlearn and we will see how\nimagine we have our original sentence\nyour cat is a lovely cat what we do is\nwe first convert into embeddings using\nthe previous layer so the input\nembeddings and these are embeddings of\nsize 512 then we create some special\nvectors called the positional encoding\nvectors that we add to these embeddings\nso this Vector we see here in red\nis a vector of size 512 which is not\nlearned it's computed once and not\nlearned along with the training process\nit's fixed and this word this Vector\nrepresents the position of the word\ninside of the sentence\nand this should give us a output that is\na vector of size again 512 because we\nare summing this number with this number\nthis number with this number so the\nFirst Dimension with the First Dimension\nthe second dimension with that so we\nwill get a new Vector of the same size\nof the input vectors or how are these\nposition in both embedding calculated\nlet's see\nimagine we have a smaller sentence let's\nsay your cat is and you may have seen\nthe following expressions from the paper\nwhat we do is we create a vector of five\nof size D model so 512 and for each\nposition in this Vector we calculate the\nvalue using these two expressions\nusing these arguments so the first\nargument indicates the position of the\nword inside of the sentence so the word\nyour occupies the position zero and we\nuse them for the even Dimension so the\nzero the two the four the 510 Etc we use\nthe first expression so the sine and for\nthe other positions of this Vector we\nuse the second expression\nand we do this for all the words inside\nof the sentence so this particular\nembedding is calculated p e of 1 0\nbecause it's the first word embedding\nzero so this one represents the argument\npause and this 0 represents the argument\n2 I and p e of 1 1 means that the first\nword uh Dimension one so we will use the\ncosine\ngiving the position one and the two I\nwill be equal to 2i plus 1 will be equal\nto 1.\nand we do this for this third word Etc\nif we have another sentence we will not\nhave different positional encodings\nwe will have the same vectors even for\ndifferent sentences because the\npositional encoding are computed once\nand reused for every sentence that our\nmodel will see\nduring inference or training so we only\ncompute the positional encoding once\nwhen we create the model we save them\nand then we reuse them we don't need to\ncompute it every time we feed the feed a\nsentence to the model\nso why the authors chose the cosine and\nthe sine functions to represent\npositional encodings because let's watch\nthe plot of these two functions uh the\nyou can see the plot is by position so\nthe position of the word inside of the\nsentence and this depth is the dimension\nalong the vector so the two I that you\nsee saw before in the previous\nexpressions\nand if we plot them we can see as humans\na pattern here and we hope that the\nmodel can also see this path okay the\nnext layer of the encoder is the\nmulti-head attention\nwe will not go inside of the multi-head\nattention first we will first visualize\nthe single head attention so the\nself-attention with a single head and\nlet's do it\nso what is self-attention self attention\nis a mechanism that existed before they\nintroduced the Transformer the Alters of\nthe Transformer just changed it into a\nmulti-head attention so how did the\nself-attention work\nthe self-attention allows the model to\nrelate words to each other\nokay so we had the input embeddings that\ncapture the meaning of the word then we\nhave the positional encoding that give\nthe information about the position of\nthe word inside of the sentence now we\nwant this self-attention to relate words\nto each other\nnow imagine we have uh in an input\nsequence of six word with the D model of\nsize 512.\nwhich can be represented as a matrix\nthat we will call Q K and V so our q k\nand V is a same Matrix are the same\nMatrix representing the input so the\ninput of six words with the dimension of\n512 so each word is represented by a\nvector of size 512. we basically apply\nthis formula we saw here from the paper\nto calculate the attention the self\nattention in this case why\nself-attention because it's the each\nword in the sentence related to other\nwords in the same sentence so it's\nself-attention\nso we start with our Q Matrix which is\nuh the input sentence so let's visualize\nit for example so we have six rows and\non this uh on the columns we have 512\ncolumn now they are really difficult to\ndraw but let's say we have 512 columns\nand here we have six okay now what we do\naccording to this formula we multiply it\nby the same sentence but transposed so\nthe transpose of the K which is again\nthe same input sequence\nwe divide it by the square root of 512\nand then we apply this soft Max\nthe output of this as we saw before in\nin the initial Matrix and notations we\nsaw that when we multiply 6 by 512 with\nanother Matrix that is 512 by 6 we\nobtain a new Matrix that is six by six\nand each value in this Matrix represents\nthe dot product of the first row with\nthe First Column this represents the dot\nproduct of the first row with the second\ncolumn Etc\nthe values here are actually randomly\ngenerated so don't concentrate on the\nvalues what you should notice is that\nthe soft Max makes all these values in\nsuch a way that they sum up to one so\nthis Row for example here some sums up\nto one this other row also sums up to\none etc etc and this value we see here\nit's the dot product of the first word\nwith the embedding of the word itself\nthis value here is the dot product of\nthe embedding of the word your with the\nembedding of the word cat and this value\nhere is the dot product of the word the\nembedding of the word your with the\nembedding of the word is\nthe next thing we and this value\nrepresents somehow a score that how\nintense is the relationship between one\nword and another let's go uh ahead with\nthe formula so for now we just\nmultiplied Q by K divided by the square\nroot of Decay applied to the soft Max\nbut we didn't multiply by V\nso let's go forward we multiply this\nmatrix by V and we obtain a new Matrix\nwhich is 6 by 512 so if we multiply a\nmatrix that is 6 by 6 with another that\nis 6 by 512 we get a new Matrix that is\n6 by 512 and one thing you should notice\nis that with the dimension of this\nMatrix is exactly the dimension of the\ninitial Matrix from which we started\nthis what does it mean that we obtain a\nnew Matrix that is six rows so let's say\nsix rows\nwith 512 columns\nin which each these are our words so we\nhave six words and each word has an\nembedding of Dimension 512 so now this\nembedding here represents not only the\nmeaning of the word which was given by\nthe input embedding not only the\nposition of the word which was added by\nthe positional encoding but now somehow\nthis special embedding so these values\nrepresent a special embedding that also\ncaptures the relationship of this\nparticular word with all the other words\nand this particular embedding of this\nword here also captures not only its\nmeaning not only its position inside of\nthe sentence but also the relationship\nof this word with all the other words\nI want to remind you that this is not\nthe multi-head attention we are just\nwatching the self-attention so one head\nwe will we will see later how this\nbecomes the multi-head attention\nself-attention has some properties that\nare very desirable\nfirst of all it's permutation invariant\nwhat does it mean to be permutation\ninvariant it means that if we have a\nmatrix let's say\nfirst we had a matrix of six words in\nthis case the let's say just four words\nso a b c and d\nand suppose by applying the formula\nbefore this produces this particular\nMatrix in which the there is new special\nembedding\nfor the word a a new special embedding\nfor the word b a new special bedding for\nthe word c and d so let's call it a\nprime B Prime C Prime D Prime if we\nchange the position of these two rows\nthe values will not change the position\nof the output will change accordingly so\nthe values of B Prime will not change it\nwill just change in the the position and\nalso the C will also change position but\nthe values in each Vector will not\nchange and this is a desirable\nproperties self-attention as of now\nrequires no parameters I mean I didn't\nintroduce any parameter that is learned\nby the model I just took the initial\nsentence of in this case six words\nwe multiplied it by itself we divide it\nby a fixed quantity which is the square\nroot of 512 and then we apply the soft\nMax which is not introducing any\nparameters so for now the self-attention\nrate didn't require any parameter except\nfor the embedding of the words\nthis will change later when we introduce\nthe multi-head attention\nalso we expect because the each value in\nthe self-attention in the soft Max\nMatrix is a DOT product of the word\nembedding with itself and the other\nwords we expect the values along the\ndiagonal to be the maximum because it's\nthe dot product dot product of each word\nwith itself\nand\nthere is another property of this Matrix\nthat is before we apply the soft softmax\nif we replace the value in this Matrix\nsuppose we don't want the word your and\nCat to interact with each other or we\ndon't want the word let's say is and the\nlovely to interact with each other what\nwe can do is before we apply the softmax\nwe can replace this value with minus\ninfinity and also this value with minus\ninfinity\nand when we apply the soft Max the soft\nMax will replace minus infinity with 0.\nbecause as you remember the soft Max is\ne to the power of x if x is going to\nminus infinity e will be e to the power\nof minus infinity will become very very\nclose to zero so basically zero\nthis is a desirable property that we\nwill use in the decoder of the\nTransformer now let's have a look at\nwhat is a multi-head attention so what\nwe just saw was the self attention and\nwe want to convert it into a\nmulti-headed tension you may have seen\nthese expressions from the paper but\ndon't worry I will explain them one by\none so let's go\nimagine we have our encoder so we are on\nthe encoder side of of the Transformer\nand we have our input sentence which is\nlet's say 6 by 512 so Six Word by 512 is\nthe size of the embedding of each word\nin this case I call it sequence by D\nmodel so sequence is the sequence length\nas you can see on the legend in the\nbottom left of the slide and the D model\nis the size of the embedding Vector\nwhich is 512. what we do just like the\npicture shows\nand we take this input and we make four\ncopies of it one will be sent uh wait\none will be sent along this connection\nwe can see here and three will be sent\nto the multi-header attention with three\nrespective names so it's the same input\nthat becomes three matrices that are\nequal to input one is called the query\none is called key and one is called\nvalue so basically we are taking this\ninput and making three copies of it one\nwe call Q K and B they have of course\nthe same dimension\nwhat does the multihead attention do\nfirst of all it multiplies these three\nmatrices by three parameter matrices\ncalled WQ w k and WV\nthese matrices have Dimension D model by\nD model so if we multiply a matrix that\nis sequence by the model with another\none that is D model by D model we get a\nnew Matrix as output that is sequenced\nby D model so basically the same\nDimension as the starting Matrix\nand we will call them Q Prime K Prime\nand V Prime\nour next step is to split these matrices\ninto smaller matrices let's see how\nwe can split this Matrix Q Prime by the\nsequence Dimension or by the D model\ndimension\nin the multi-hat attention we always\nsplit by the D model Dimension so every\nhead will see the full sentence but a\nsmaller part of the embedding of each\nword\nso if we have an embedding of let's say\n512 it will become smaller embeddings of\n512 divided by four and we call this\nquantity d k so d k is D model divided\nby H where H is the number of heads in\nour case we have H equal to 4.\nwe can calculate the attention between\nthese smaller matrices so q1 K1 and V1\nusing the expression taken from the\npaper\nand this will result into a small Matrix\ncalled Head 1 head 2 head 3 and head\nfour the dimension of head 1 up to head\nfour is sequence by d v\nwhat is DV is basically it's equal to DK\nit's just called a DV because the last\nmultiplication is done by V and in the\npaper they call it DV so I am also\nsticking to the same names\nour next step is to multi combine these\nmatrices these small heads\nby concatenating them along the DV\nDimension just like the paper says so we\ncan cut all this head together and we\nget a new Matrix that is sequence by H\nmultiplied by DV\nwhere H multiplied by DV as we know DV\nis equal to d k so H multiplied by DV is\nequal to D model so we get back the\ninitial shape so it's sequence by D\nmodel here\nthe next step is to multiply the result\nof this concatenation by w o\nand W O is a matrix that is H multiplied\nby DV so D model multiple with the other\ndimension being T model and the result\nof this is a new Matrix that is the\nresult of the multi-head attention which\nis sequenced by D model\nso the multi had attention instead of\ncalculating the attention between these\nmatrices here so Q Prime K Prime and V\nPrime splits them along the D model\nDimension into smaller matrices and\ncalculates the attention between these\nsmaller matrices so each head is\nwatching the full sentence but as\ndifferent aspect of the embedding of\neach word why we want this because we\nwant the each head to watch different\naspects of the same word for example in\nthe Chinese language but also in other\nlanguages one word may be a noun in some\ncases maybe a verb in some other cases\nmaybe a adverb in some other cases\ndepending on the context\nso what we want is that one head maybe\nlearns to relate that word as a noun\nanother head maybe learns to relate that\nword as a verb and another head learn to\nrelease that verb as an objective or\nadverb\nso this is why we want a multi-head\nattention\nnow you may also have seen online that\nthe the attention can be visualized and\nI will show you how when we calculate\nthe attention between the Q and the K\nmatrices so when we do this operation so\nthe soft Max of Q multiplied by the K\ndivided by the square root of d k\nwe get a new Matrix just like we saw\nbefore which is sequenced by sequence\nand this represents a score that\nrepresents the intensity of the\nrelationship between the two words\nwe can visualize this\nand this will produce a visualization uh\nsimilar to this one which I took from\nthe paper in which we see how the all\nthe heads work so for example if we\nconcentrate on this work making this\nword here we can see that making is\nrelated to the word difficult so this\nword here by different heads so the blue\nhead the red head and the green head\nbut the wire let's say the Violet head\nis not relating this two word together\nso making and difficult is not related\nby the violet or the pink head\nThe Violet head or the pink head they\nare relating the word making to other\nwords for example to this word 2009\nwhy this is the case because maybe this\npink head could see the part of the\nembedding that these other heads could\nnot see that made this interaction\npossible between these two words\nyou may be also wondering why these\nthree mattresses are called query keys\nand values\nokay the terms come from the database\nterminology or from the python-like\ndictionaries but I would also like to\ngive my interpretation of my own making\na very simple example I think it's quite\neasy to\num\nunderstand\nso imagine we have a python-like\ndictionary or a database in which we\nhave keys and values\nthe keys are the category of movies and\nthe values are the movies belonging to\nthat category in my case I just put one\nvalue\nso we have Romantics category which\nincludes Titanic we have action movies\nthat include the Dark Knight Etc imagine\nwe also have a user that makes a query\nand the query is love\nbecause we are in the Transformer world\nall these words actually are represented\nby embeddings of size 512.\nso what our Transformer will do he will\nconvert this word love into an embedding\nof 512 all these queries and values are\nalready embeddings of 512 and it will\ncalculate the dot product between the\nquery and all the keys\njust like the formula so as you remember\nthe formula is a soft Max of query\nmultiplied by the transpose of the keys\ndivided by the square root of the model\nso we are doing the dot product of all\nthe queries with all the keys\nin this case the word love with all the\nkeys one by one\nand this will result in a score that\nwill amplify some values or not amplify\nother values\num in this case our embedding may be in\nsuch a way that the word love and\nromantic are inter are related to each\nother the word love and comedy are also\nrelated to each other but not so\nintensively like the word love and\nromantic so it's more how to say let's\nless strong relationship but maybe the\nword horror and love are not related at\nall so maybe their soft Max score is\nvery close to zero\nour next\num\nlayer in the encoder is the ADD and norm\nand to introduce the other Norm we need\nthe layer normalization so let's see\nwhat is the layer normalization\nlayer normalization is a layer that okay\nlet's make a practical example imagine\nwe have a batch of n items in this case\nn is equal to three\nitem one item two item three each of\nthese items will have some features it\ncould be an embedding so for example it\ncould be a feature of a vector of size\n512 but it could be a very big Matrix of\nthousands of features doesn't matter\nwhat we do is we calculate the mean and\nthe variance of each of these items\nindependently from each other\nand we replace each value with another\nvalue that is given by this expression\nso basically we are normalizing so that\nthe new values are all in the range 0 to\n1.\nactually we also multiply this new value\nwith a parameter called gamma and then\nwe add another parameter called beta and\nthis gamma and beta are learnable\nparameters\nand the model should learn to multiply\nand add these parameters so as to\namplify the value that it wants to be\nAmplified and not amplify that value\nthat it doesn't want to be Amplified\nuh so we don't just normalize we\nactually introduce some parameters\nand I found a really nice visualization\nfrom papers with code.com\nin which we see the difference between\nbatch norm and layer Norm so as we can\nsee in the layer normalization we are\ncalculating if n is the batch Dimension\nwe are calculating all the values\nbelonging to one item in the batch\nwhile in the batch Norm we are\ncalculating the same feature for all the\nbatch so for all the items in the batch\nso we are mixing let's say values from\ndifferent items of the batch while in\nthe layer normalization we are treating\neach item in the batch independently\nwhich will have its own mean and its own\nvariance\nlet's look at the decoder now\num in the encoder we saw the input\nembeddings in this call in this case\nthey are called output embeddings but\nthe underlying working is the same here\nalso we have the positional encoding and\nthey are also the same as the Imp as the\nencoder\nthe next layer is the musket multi-head\nattention and we will see it now we also\nhave the multi-head attention here with\nthe\nhere we should see that the\nthere is the encoder here that produces\nthe output and is sent to the decoder in\nthe forms of keys\nand values\nwhile the query so this connection here\nis the query coming from the decoder\nso in this multi-head attention it's not\na self-attention anymore it's a cross\nattention because we are taking two\nsentences one is sent from the encoder\nside so let's write encoder in which we\nprovide the output of the encoder and we\nuse it as a query as keys and values\nwhile the output of the masked\nmulti-head attention is used as the\nquery in this multi-head attention\nand the musket multi-head attention is\nthe self-attention of the input sentence\nof the decoder so we take the input\nsentence of the decoder we transform\ninto embeddings we add the depositional\nencoding we give it to this multi-head\nattention in which the query key and\nvalues are the same input sequence we do\nthe ADD and Norm then we send this as\nthe queries of the multi-head attention\nwhile the keys and the values are coming\nfrom the encoder then we do the add the\nnorm\nI will not be showing the feed forward\nwhich is just a fully connected layer\nwe then send the output of the feed\nforward to the ADD and norm and finally\nto the linear layer which we will see\nlater so let's have a look at the Muscat\nmulti-head attention and how it differs\nfrom a normal multi-head attention\nwhat we want our goal is that we want to\nmake the model causal it means that the\noutput at a certain position can only\ndepend on the words on the previous\nposition so the model must not be able\nto see future words how can we achieve\nthat\nas you saw the the output of the soft\nMax in the attention calculation formula\nis this Matrix sequence by sequence if\nwe want to hide the interaction of some\nwords with other words we delete this\nvalue and we replace it with minus\ninfinity before we apply the soft Max so\nthat the soft Max will replace this\nvalue with 0. and we do this for all the\ninteraction that we don't want so we\ndon't want your to watch future words so\nwe don't want your to watch cat is a\nlovely cat and we don't want the word\ncat to watch future words but only all\nthe words that come before it or the\nword itself so we don't want this this\nthis this also the same for the other\nwords Etc\nso we can see that we are replacing all\nthe word all this values here that are\nabove this diagonal here so this is the\nprincipal diagonal of the Matrix and we\nwant all the values that are above this\ndiagonal to be replaced with minus\ninfinity so that so that the soft Max\nwill replace them with zero let's see in\nwhich stage of the multi-head attention\nthis mechanism is introduced so when we\ncalculate the attention between these\nsmaller matrices so q1 K1 and V1\nbefore we apply this soft Max we replace\nthis values so this one this one this\none this one this one Etc with minus\ninfinity then we apply this soft Max and\nthen the soft Max will take care of\ntransforming these values into zeros so\nbasically we don't want these words to\ninteract with each other\nand if we don't want this interaction\nthe model will learn to not make them\ninteract because the model will not get\nany information from this interaction so\nit's like this word cannot interact now\nlet's look at how the inference and\ntraining works for a Transformer model\nas I saw said previously we are dealing\nwith it we will be dealing with the\ntranslation tasks so because it's easy\nto visualize and it's easy to understand\nall the steps let's start with the\ntraining of the model we will go from an\nEnglish sentence I love you very much\ninto an Italian sentence it's a very\nsimple sentence it's easy to describe\nlet's go\nwe start with a description of the\nof the Transformer model and we start\nwith our English sentence which is sent\nto the encoder so our English sentence\nhere\non which we prepared and append to\nspecial tokens one is called start of\nsentence and one is called end of\nsentence these two tokens are taken from\nthe vocabulary so they are special\ntokens in our vocabulary that tells the\nmodel what is the start position of a\nsentence and what is the end of a\nsentence we will see later why we need\nthem\nfor now just think that we take our\nsentence we prepend a special token and\nwe append a special token\nthen what we do as you can see from the\npicture we take our inputs we transform\ninto input embeddings we add the\npositional encoding and then we send it\nto the encoder\nso this is our encoder input sequence by\nthe model we send it to the encoder it\nwill produce an output which is encode a\nsequence by D model and it's called the\nencoder output so as I saw we saw\npreviously the output of the encoder is\nanother Matrix that has the same\nDimension as the input Matrix\nin which the embedding we can see it as\na sequence of embeddings in which this\nembedding is special because it captures\nnot only the meaning of the word which\nwas given by the input embedding we saw\nhere so by this not only the position\nwhich was given by the positional\nencoding but also the interaction of\nevery word with every other word in the\nsame sentence because this is the\nencoder so we are talking about\nself-attention so it's the interaction\nof each word in the sentence with all\nthe other words in the same sentence\nwe want to convert this sentence into\nItalian so we prepare the input of the\ndecoder which is a start of sentence\nas you can see from the picture of the\nthe Transformer the outputs here you can\nsee shifted right\nwhat does it mean to shift right\nbasically it means we prepared a special\ntoken called SOS start of sentence\nyou should also notice that these two\nsequences actually they in when we code\nthe Transformer so if you watch my other\nvideo on how to code a Transformer you\nwill see that we\nmake this sequence of fixed length so\nthat if we have a sentence that is te\namo multo or a very long sequence\nactually when we feed them to the\nTransformer they all becomes become of\nthe same length how to do this we add\npadding words to reach the length the\ndesired length so if our model can\nsupport let's say a sequence length of\n1000 in this case we have a fourth\ntokens we will add\n996 tokens of padding to make this\nsentence long enough to reach the\nsequence length of course I'm not doing\nit here because it's not easy to\nvisualize otherwise\nokay we prepared this input for the\ndecoder we add transform into embeddings\nwe add the positional encoding then we\nsend it first to the multi-head\nattentions to the musket\nmulti-haditation so along with the\ncausal mask\nand then we take the output of the\nencoder and we send it to the decoder as\nkeys and values while the queries are\ncoming from the musket so the queries\nare coming from this layer and the keys\nand the values are the output of the\nencoder\nthis the output of all this block here\nso all this big block here\nwill be a matrix that is sequence by the\nmodel just like for the encoder\nhowever we can see that this is still an\nembedding because it's a D model it's a\nvector of size 512 how can we relate\nthis\num embedding back into our dictionary\nhow can we understand what is this word\nin our vocabulary that's why we need a\nlinear layer that will map sequence by D\nmodel into another sequence by\nvocabulary size so it will tell for\nevery embedding that it sees what is the\nposition of that word in our vocabulary\nso that we can understand what is the\nactual token that is output by the model\nafter that we apply the softmax and\nthen we have our label what we expect\nthe model to Output given this English\nsentence\nwe expect the model to Output this te\namo multo end of sentence and this is\ncalled the label or the target what we\ndo when we have the output of the model\nand the corresponding label we calculate\nthe loss in this case is the cross\nentropy loss and then we back propagate\nthe loss to all the weights\nnow let's understand why we have these\nspecial tokens called SOS and EOS\nbasically you can see that here the\nsequence length is 4 actually is 1000\nbecause we have the padding but let's\nsay we don't have any padding so it's\nfour tokens start of sentence the ammo\nmulto and what we want is the T ammo\nmulto end of sentence so our model when\nit will see the start of sentence token\nit will output the first token as output\nT when it will see T it will output ammo\nwhen it will see armor it will output\nmolto and when it will see a multo it\nwill output end of sentence which will\nindicate that okay the translation is\ndone\nand we will see this mechanism in the\ninference\nah this all happens in one time step\njust like I promised at the beginning of\nthe video I said that with recurrental\nor neural networks we have end time\nsteps to map n input sequence into an\noutput sequence but this problem would\nbe solved with the Transformer yes it\nhas been solved because you can see here\nwe didn't do any for Loop we just did\nall in one pass we give an input\nsequence to the encoder an input\nsequence to the decoder we produced some\noutputs we calculated that cross entropy\nloss with the label and that's it it all\nhappens in one time step and this is the\npower of the Transformer because it made\nit very easy and very fast to train very\nlong sequences and with the very very\nnice performance that you can see in\ncharge GPD you can see GPT in bird Etc\nlet's have a look at how inference works\nagain we have our English sentence I\nlove you very much we want to map it\ninto an Italian sentence\nwe have our usual Transformer we prepare\nthe input for the encoder which is start\nof sentence I love you very much end of\nsentence\nwe convert into input embeddings then we\nadd the positional encoding we prepare\nthe input for the encoder and we send it\nto the encoder the encoder will produce\nan output which is sequenced by the\nmodel and we saw it before that it's a\nsequence of special embeddings that\ncapture the meaning the position but\nalso the interaction of all the words\nwith other words\nwhat we do is for the decoder we give\nhim just the start of sentence and of\ncourse we keep the we add enough\nembedding padding tokens to reach our\nsequence length we just give the model\nthe start of sentence token and we again\nwe for this single token we convert into\nembeddings we add the positional\nencoding and we send it to the decoder\nas decoder input the decoder will take\nthis\num his input as a query and the key and\nthe values coming from the encoder\nand it will produce an output which is\nsequenced by D model again we want the\nlinear layer to project it back to our\nvocabulary and this projection is called\nlogits\nwhat we do is we apply the soft Max\nwhich will select given the logists will\ngive the position of the output word\nwill have the maximum score with the\nsoft Max this is how we know what words\nto select from the vocabulary\nand this hopefully should produce the\nfirst output token which is T\nif the model has been trained correctly\nthis however happens at time step one so\nwhen we train the model Transformer\nmodel it happens in one pass so we have\none input sequence one output sequence\nwe give it to the model we do it one\ntime step and the model will learn it\nwhen we inference however we need to do\nit token by token and we will also see\nwhy this is the case\nat time Step 2 we don't need to\nrecompute the encoder output again\nbecause the over English sentence didn't\nchange so we hope the the encoder should\nproduce the same output for it and then\nwhat we do is we take the output of the\nprevious sentence so\num\nas T we append it to the input of the\ndecoder and then we feed it to the\ndecoder again with the output of the\nencoder from the previous step\nwhich will produce an output sequence\nfrom the decoder side which we again\nproject back into our vocabulary and\nwe get the next token which is ammo\nso as I saw before as I as I said before\nwe are not recalculating the output of\nthe encoder for every time step because\nour English sentence didn't change at\nall what is changing is the input of the\ndecoder because at every time step we\nare appending the output of the previous\nstep to the input of the decoder we do\nthe same for the time step 3\nand we do the same for the time step 4\nand hopefully we will stop when we see\nthe end of sentence token because that\nis that's how the model tells us to stop\ninferencing\nand this is how the inference works why\nwe needed four time steps\nwhen we inference a model\num like the in this case the translation\nmodel there are many strategies for\ninferencing what we used is called\ngreedy strategy so for every step we get\nthe word with the maximum soft max value\nand however this strategy Works uh\nusually not bad but there are better\nstrategies\nand one of them is called beam search\nin beam search instead of always\ngreedily so this is that's why it's\ncalled greedy instead of greedily taking\nthe maximum soft value we take the top B\nvalues and then for each of these\nchoices we inference what are the next\npossible tokens for each of the top B\nvalues at every step and we keep only\nthe one with the B most probable\nsequences and we delete the others this\nis called beam search and it generally\nit performs better\nso thank you guys for watching uh I know\nit was a long video but it was really\nworth it to go through each aspect of\nthe Transformer I hope you enjoyed this\njourney with me so please subscribe to\nthe channel and don't forget to watch my\nother video on how to code a Transformer\nmodel from scratch in which I describe\nnot only again the structure of the\nTransformer model while coding it but I\nalso show you how to train it on a data\nset of your choice how to inference it\nand I also provided the code on GitHub\nand the Ecolab notebook to train the\nmodel directly on collab\nplease subscribe to the to the channel\nand let me know what you didn't\nunderstand so that I can give more\nexplanation and please tell me what are\nthe problems in this kind of videos or\nin this particular video that I can\nimprove for the next videos thank you\nvery much and have a great rest of the\nday",
  "transcript_chars": 46053,
  "ingested_at": "2026-05-15T16:39:49.248265+00:00",
  "source": "channel",
  "yt_meta": {
    "view_count": 694072,
    "like_count": 19154,
    "channel_id": "UCtAcpQcYerN8xxZJYTfWBMw",
    "categories": [
      "Science & Technology"
    ],
    "tags": [
      "transformer",
      "deep learning",
      "pytorch",
      "ai",
      "ml",
      "machine learning",
      "attention is all you need"
    ]
  }
}