{
  "video_id": "GDN649X_acE",
  "channel_slug": "statquest",
  "channel_handle": "statquest",
  "title": "Encoder-Only Transformers (like BERT) for RAG, Clearly Explained!!!",
  "duration_seconds": 1132.0,
  "url": "https://www.youtube.com/watch?v=GDN649X_acE",
  "upload_date": "",
  "transcript": "in code only Transformers can cluster\nthings and classify things that's cool\nstack\nQuest hello I'm Josh starmer and welcome\nto stack Quest today we're going to talk\nabout incoder only Transformers and\nthey're going to be clearly\nexplained this stack Quest is brought to\nyou by the letters a b and c a always b\nb c curious always B\ncurious note this stack Quest assumes\nthat you are already familiar with the\nmain ideas of how neural networks work\nif not check out the quest also note\nthis stack Quest focuses on the main\nideas of how encoder only Transformers\nwork and what we can do with them that\nsaid if you want to see the nitty-gritty\ndetails check out these fine\nquests okay way back in 2017 the first\nTransformer ever made had one part\ncalled an\nencoder and a second part called a\ndecoder and the encoder and the decoder\nwere connected to each other so they\ncould work together this first\nTransformer was based on something\ncalled a seek to seek or an encoder\ndecoder model encoder decoder models\nwere designed to translate text in one\nlanguage like English into another\nlanguage like\nSpanish for for example Squatch might\nsay pizza is\ngreat and the encoder would crunch a lot\nof numbers to encode the\ninput and the decoder would crunch a lot\nof numbers including the output from the\nencoder and ultimately decode those\nnumbers into a\ntranslation La piz s\nmagnifico\nbam however it wasn't long after this\nfirst encoder decoder transform was\npublished that people realized that both\nparts could work just fine on their\nown it turned out that we could generate\ntext including translations of text with\njust a\ndecoder and these models which form the\nbasis for chat GPT were named decoder\nonly\nTransformers likewise models based\nentirely on the encoder started to be\nvery useful on their\nown and these models which form the\nbasis for Bert and many other models\nwere named encoder only\nTransformers however over the years the\nmodels based on decoder only\nTransformers kind of stole the\nshow pretty much everyone has used or at\nleast heard of Chad GPT or gemini or any\nof the other countless decoder only\nTransformers out there and I've heard\nsome super smart people who keep up with\nAI and everything ask me if people still\nuse encoder only\nTransformers so in this stat Quest I\nwant to dive deep into encoder only\nTransformers so we can understand their\nsignificant but understated power and\nthat means starting with a deep dive\ninto the first thing a Transformer does\nwhich is create word\nembeddings word embedding converts words\nbits of words and symbols collectively\ncalled tokens into\nnumbers we need to convert tokens into\nto numbers because Transformers are a\ntype of neural network and neural\nnetworks only operate on\nnumbers one super easy way to convert\nwords into numbers is to just assign\neach word to a random number for example\nif Squatch just ate a delicious pizza\nthey might say pizza is\ngreat and we could assign a random\nnumber to each\nword now if Norm came along and said p\nis\nawesome then we could reuse the random\nnumbers that we already assigned to\npizza and\nis and assign a new random number to\nawesome in theory this is\nfine but it means that even though great\nand awesome mean similar things and are\nused in similar\nways they have very different numbers\nassociated with them and that means the\nneural network will probably need a lot\nmore more complexity in training because\nlearning how to correctly process the\nword great won't help the neural network\ncorrectly use the word\nawesome so it would be nice if similar\nwords that are used in similar ways\ncould be given similar\nnumbers so that learning how to use one\nword will help learn how to use the\nother at the same time and because the\nsame words can be used in different\ncontexts or made plural or used in some\nother way it might be nice to assign\neach word more than one number so that\nthe neural network can more easily\nadjust to different\ncontexts for example the word great can\nbe used in a positive way like pizza is\ngreat and it can also be used in a\nsarcastic negative way like my cell\nphone's broken\ngreat and it would be nice if we had one\nnumber that could keep track of the\npositive way that grade is used and a\ndifferent number to keep track of the\nnegative\nways hey Josh deciding what words are\nsimilar and are used in similar context\nsounds like a lot of work and using more\nthan one number per word to account for\ndifferent contexts sounds like even more\nwork don't worry Squatch the good news\nis that we can get a relatively simple\nneural network to do all of the work for\nus for for example let's imagine we have\ntwo\nphrases Pizza is great and pizza is\nawesome the first thing we do is create\ninputs to a relatively simple neural\nnetwork for each unique word then we\ncreate an output for each\nword then we connect all of the inputs\nto at least one activation function and\nin this example we'll connect the inputs\nto two activation functions the number\nof activation functions determines how\nmany numbers we will use to represent\neach word in this case since we have two\nactivation\nfunctions we'll end up with two numbers\nor word embeddings representing each\nword then we add weights to the\nconnections from the inputs to the\nactivation\nfunctions these weights are initialized\nwith random numbers so right now they're\nnot very useful but the plan is to track\nthem and thus change them using this\ndata lastly we connect the activation\nfunctions to the outputs with some\nboring details that we don't need to\nworry about right\nnow because we have one word embedding\nfor each word going to the activation\nfunction on the top and one word\nembedding for each word going to the\nactivation function on the\nbottom we can plot each word on a graph\nthat has the top word embeddings on the\nX x\naxis and the bottom word embeddings on\nthe Y\nAIS for example the word Pizza goes here\nbecause its top W embedding is\n0.11 and its bottom word embedding is\n0.10 likewise the word is goes here\ngreat goes\nhere and awesome goes\nhere now with this graph we see that the\nwords great and awesome are currently no\nmore similar to each other than they are\nto any of the other\nwords however because both words appear\nin the same context in the training data\nwe hope that training the network will\nmake their word embeddings more\nsimilar the idea is that we want each\nword in the training data to predict the\nnext\nword for example we want the first word\nin each sentence Pizza to predict the\nword that comes after it\nis and we want the word is to predict\nthe words that come after it great and\nawesome so in order to see which word\nthe network predicts should come after\nPizza we put a one in the input for\npizza and we put zeros in all of the\nother\ninputs then we do the math with the\nrandomly initialized\nparameters and we end up predicting in\ngreat because it has the largest output\nvalue\n0.45 thus with the randomly initialized\nparameters the network does not\ncorrectly predict is the word that comes\nafter\nPizza however after we train the model\nwe end up with these new word\nembeddings and pizza correctly predicts\nis and is correctly predicts great and\nawesome now when we graph the words with\nthe new word\nembeddings great and awesome cluster\ntogether this result is great and\nawesome because great and awesome are\nsimilar words used in similar\ncontexts and they ended up with similar\nword\nembeddings\nbam so far we've seen the simplest way\nto create word\nembeddings we used a simple net work\nthat we train to predict the word that\ncomes after the\ninput however just predicting the next\nword doesn't give us a lot of context to\ndetermine the optimal word\nembeddings in contrast if we had a more\ncomplicated training data set then we\nwould have more inputs and outputs for\nour neural\nnetwork and we could connect everything\nlike we did\nbefore but now because we have more\ninputs and outputs\nand longer sentences in the training\ndata we can add more context to the\ntraining\nprocess for example we can use the pizza\ncame out to predict the next word\nof in other words instead of just using\none word to predict the next we can use\nthe preceding four words to predict the\nnext increasing the context can help\ncreate better words embedding values but\nI want to point out that the way we are\ndoing things right now ignores word\norder and because we are not currently\nkeeping track of word order any jumble\nis just as good as anything\nelse in other words the pizza came out\nof would give us the same inputs and\noutput as the jumbled up phrase Pizza\nout came the\nof so it would be nice if there were\nsome way to create word embedding values\nthat also took word order into\naccount and that leads us to the part of\na transformer that comes after the\ninitial word embedding\nlayer called positional\nencoding positional encoding helps keep\ntrack of word\norder for example if Norm said Squatch\neats\npizza then Squatch might say\nyum in in contrast if Norm said Pizza\neats\nSquatch then Squatch might say\nyikes so these two\nphrases Squatch eats pizza and pizza\neats Squatch use the exact same words\nbut have very different\nmeanings so keeping track of word order\nis super\nimportant there are a bunch of ways to\nimplement positional encoding but these\ndetails are out of the scope of this\nQuest anyway now we know that positional\nencoding helps keep track of word\norder however it would be great if there\nwas also a way to keep track of the\nrelationships among\nwords for example if the input sentence\nwas this the pizza came out of the oven\nand it tasted good then this word it\ncould refer to pizza or or potentially\nit could refer to the word\noven Josh I've heard of good tasting\npizza but never a good tasting\noven I know Squatch that's why it's\nimportant that the Transformer correctly\nAssociates the word it with\npizza and that leads us to the part of a\ntransformer that comes after positional\nencoding called\nattention and specifically in an encoder\nonly Transformer it's called self\nattention attention helps keep track of\nthe relationships among\nwords going back to our example about\npizza coming out of an\noven attention can help correctly\nassociate the word it with the word\nPizza note there are different types of\nattention but incoder only Transformers\nonly use self attention so we'll focus\non that in general terms self attention\nworks by seeing how similar each word is\nto all of the other words in the\nsentence including\nitself for example self- attention\ncalculates the similarity between the\nfirst word the and all of the other\nwords in the sentence including\nitself and self attention calculates\nthese similarities for every word in the\nsentence once the similarities are\ncalculated they are used to determine\nhow the Transformer encodes each word\nfor example if you looked at a lot of\nsentences about pizza and the word it\nwas more commonly associated with pizza\nthan\noven then the similarity score for pizza\nwill cause it to have a larger impact on\nhow the word it is encoded by the\nTransformer\nbam and now that we understand the ideas\nbehind self\nattention we understand the ideas behind\nthe three fundamental building blocks\nthat make up an encoder only\nTransformer word embedding converts the\ninput into\nnumbers positional encoding helps keep\ntrack of word\norder and self attention helps establish\nrelationships among\nwords combining all three layers creates\na new kind of embedding for each token\nthat takes position and relationships\namong words into\naccount and this new type of embedding\nis sometimes called context aware\nembedding or contextualized\nembedding because context aware\nembeddings include information about the\nposition of each word as well as the\nrelationships among the\nwords context towar embeddings can help\ncluster similar\nsentences or even similar\ndocuments\nbam to summarize encoder only\nTransformers like Bert that only use\nself\nattention can create context aware\nembeddings and just like plain old word\nembeddings can help cluster similar\nwords that are used in similar\nways context aware embeddings can help\ncluster similar\nsentences or similar\ndocuments note the ability to Cluster\nsimilar sentences and documents is the\nfoundation for something called\nretrieval augmented generation or r\nrag rag works by breaking a document\ninto blocks of text and then using an\nencoder only Transformer to create\ncontext aware embeddings for each\none then when someone gives an AI a\nprompt like what is\npizza rag generates embeddings for what\nis pizza and finds the chunks of text\nthat are the most\nsimilar double\nbam now that we understand the main\nideas behind encoder only Transformers\nand context aware\nembeddings let's talk about another cool\nthing we can do with them one thing that\nwe can do is use the context aware\nembeddings as inputs to a normal neural\nnetwork that classifies the sentiment of\nthe\ninput for example we might want to see\nif people are posting positive or\nnegative sentiments about pizza on\nLinkedIn and the context aware embedded\nare great inputs for a neural network\nthat can do that type of\nclassification alternatively we could\nuse the context aware embeddings as\nvariables in a logistic regression model\nthat does\nclassification anyway even though\ndecoder only Transformers like chat GPT\nget all the hype the context aware\nembeddings that encoder only\nTransformers create can be used in a\nwide variety of settings to do very cool\ncool things triple\nbam now it's time for some Shameless\nself-promotion if you want to review\nstatistics and machine learning offline\ncheck out the stack Quest PDF study\nguides in my books the stat Quest\nIllustrated guide to machine learning\nand the stat Quest Illustrated guide to\nneural networks and AI at stat quest.org\nthere's something for\neveryone hooray we've made it to the end\nof another exciting stack Quest if you\nlike this stack Quest and want to see\nmore please subscribe and if you want to\nsupport stack Quest consider\ncontributing to my patreon campaign\nbecoming a channel member buying one or\ntwo of my original songs or a t-shirt or\na hoodie or just donate the links are in\nthe description below all right until\nnext time Quest on",
  "transcript_chars": 14096,
  "ingested_at": "2026-05-15T10:53:20.745182+00:00",
  "source": "channel",
  "yt_meta": {
    "view_count": 96837,
    "like_count": 2681,
    "channel_id": "UCtYLUTtgS3k1Fg4y5tAhLbw",
    "categories": [
      "Education"
    ],
    "tags": [
      "Josh Starmer",
      "StatQuest",
      "Machine Learning",
      "BERT",
      "Transformers",
      "LLM",
      "RAG",
      "AI",
      "Neural Network"
    ]
  }
}