{
  "video_id": "ZTt9gsGcdDo",
  "channel_slug": "statquest",
  "channel_handle": "statquest",
  "title": "Essential Matrix Algebra for Neural Networks, Clearly Explained!!!",
  "duration_seconds": 1801.0,
  "url": "https://www.youtube.com/watch?v=ZTt9gsGcdDo",
  "upload_date": "",
  "transcript": "If you can add and multiply, then you can do matrix stuff! Hooray! StatQuest. Hello,\nI'm Josh Starmer and welcome to StatQuest. Today we're going to talk about essential\nmatrix algebra for neural networks and it's going to be clearly explained. Start\nsmall but then build as big as you want.\nLightning! Bam! This StatQuest is also brought to you by the letters 'A', 'B', and\n'C'. 'A' always, 'B' be, 'C', curious. Always be curious. Not that you need to, but\nif you watched any of the StatQuest of videos on neural networks, then you already\nknow that even the most state-of-the-art models, like transformers, can be explained\nwith relatively simple diagrams and flowcharts. However, if you try to code neural\nnetworks using any of the tools that make the job easier, like PyTorch and PyTorch\nLightning, then it won't be long before you need to look at the documentation and you'll see something like this.\nUgh!\nOr you'll have a small bug in your code and you'll see an error like this.\nDouble ugh!\nOr you might just want to read up on the latest trends and neural networks and not\nwait for StatQuest to explain it, and you'll see stuff like this.\nTriple ugh! All this math looks complicated and hard to understand.\nDon't worry, 'Squatch. We're going to learn how to read all of this gobbledygook one step at a time.\nBam!\nAll of these things are based on matrix equations, which provide a very compact way\nto describe neural networks. So if you want to understand the documentation, debug\nyour code, or just read the latest manuscripts about neural networks, you need to know how to work with matrix equations.\nSo in order to understand how matrix equations apply to neural networks, we're going\nto show how this specific neural network translates into this compact matrix equation.\nUgh, that equation looks weird. Are you sure I'm going to understand it?\nDon't worry 'Squatch. We will go through it one step at a time.\nOkay.\nHowever, we'll start by learning some basic terminology and build up an understanding\nof the strangest part of matrix equations, matrix multiplication.\nAnd in order to do that, imagine StatSquatch just got a ticket to see Taylor Swift\nat the Friends Arena in Stockholm, Sweden. 'Squatch, are you ready for it?\nYes.\nWhere is your seat?\nMy seat is close to the front on stage left.\nNow, if we put an x and y axis centered on the stage, then in terms of those axes,\nthe coordinates for 'Squatch's seat are 2, 1. Now, for reasons that we will never,\never, ever understand, Taylor decides to flip the direction of her stage 180 degrees.\nI knew she was trouble.\nAnyway, rotating the stage means 'Squatch's seat and everyone else's needs to rotate\nas well. To determine the new x and y coordinates of 'Squatch's seat, we can use\nthese equations. For example, if we plug in the original coordinates, x = 2 and y\n= 1, then the new x and y coordinates are -2, -1. In other words, these two equations\ntransform the point 2, 1 into the point -2, -1. Oh no! It's the dreaded terminology\nalert. Because these equations, which transform the original x and y coordinates\ninto new x and y coordinates, only multiply and add stuff to the original x and y\ncoordinates, this transformation is called a linear transformation. For example,\nif we focus on the first equation and plug in x = 2 and y = 1, then we get -2. If\nwe then increase x to 3, then we get -3. Likewise, increasing x to 4 gives us -4.\nThus, each time we increase the value for x by 1, we decrease the output by -1. In\nother words, a constant change in the value for x results in a constant change in\nthe output. Because the amount of change in the output is always the same, just like\nthe slope, or the amount of change in the y axis, of a straight line is always the\nsame, this transformation is called a linear transformation. In contrast, if the\ntransformation was 2 to the x, then plugging in x = 2 would give us 4. Plugging in\nx = 3 would give us 8. And plugging in x = 4 would give us 16. And we see that each\ntime we increase the value for x by 1, the output increases by a different amount.\nBecause the amount of change in the output changes. Just like the slope of a curve\nchanges, this transformation is called a nonlinear transformation. Anyway, going\nback to our original linear transformations, the reason I'm making big deal about\nlinear versus nonlinear transformations is that matrix algebra is especially useful\nfor linear transformations. Thus, using matrix notation, we can write out our linear\ntransformation like this. In this example, we've put the old x and y coordinates\ninto something called a row matrix or row vector. We can also call this row matrix\na 1 by 2 matrix because it has one row and two numbers in that row. So we say it\nhas two columns. We can also say that this matrix is one-dimensional because everything\nis on a single row. The coefficients, the numbers we multiply x and y by and define\nthe transformation, go into a matrix. And we can call this a 2 by 2 matrix because\nit has two rows and two columns. Or we can call this a two dimensional matrix because\nit has multiple rows and columns. The coefficients in the first equation go into\nthe first column in the matrix and the coefficients in the second equation go into the second column in the matrix.\nJosh, this seems really weird and I have no idea how I will remember this.\nDon't worry 'Squatch. We will explain exactly why we're doing all this in just a second.\nAnd knowing why we do this will help you derive it on your own.\nBam!\nAnyway, if, just like before, we let x = 2 and y = 1, then we can multiply the row\nmatrix that contains the original coordinates by the transformation matrix with the coefficients.\nHey Josh, I know how to multiply numbers, but how do I multiply matrices?\nWe start by multiplying the row of x and y coordinates by the numbers in the first\ncolumn in the transformation matrix. So, in this case, we start by multiplying 2\nby -1, then we multiply 1 by 0. And lastly, we add the terms together. And this gives\nus the original transformation for the new x coordinate that we started with, which\ntells us that the new x coordinate is -2. Then we multiply the same row of x and\ny coordinates by the numbers in the second column in the transformation matrix. So\nthat means we multiply 2 by 0 and 1 by -1. And lastly, add the products together.\nAnd that gives us the original transformation for the new y coordinate that we started\nwith, which tells us that the new y coordinate is -1. Combined, the matrix multiplication\ntransforms the old x and y coordinates to the new x and y coordinates.\nBam? It seems like a lot of trouble to do the same thing as before.\nWell, what if Taylor rotated the stage 90 degrees clockwise from where it was before?\nOops, she did it again?\nNo 'Squatch, that's Britney Spears. Anyway, where should 'Squatch sit? To determine\nthe newest x and y coordinates for 'Squatch's seat, we can plug the coordinates we\njust calculated into these equations. And just like before, we can convert them into\nmatrix notation by putting the x and y coordinates we just calculated in a one-dimensional\nrow matrix with one row and two columns, and putting the coefficients into a two\ndimensional transformation matrix with 2 rows and 2 columns. The coefficients in\nthe first equation go into the first column in the transformation matrix, and the\ncoefficients in the second equation go into the second column in the transformation\nmatrix. Now we plug in the x and y coordinates we calculated earlier, -2 and -1,\nand do the matrix multiplication just like before, row by column. We start by multiplying\nthe row of x and y coordinates by the numbers in the first column in the transformation\nmatrix. So we multiply -2 by 0, and we multiply -1 by 1, and then we add the terms\ntogether. And that gives us the newest x coordinate, -1. Then we multiply the same\nrow of x and y coordinates by the numbers in the second column in the transformation\nmatrix. So we multiply -2 by -1 and multiply -1 by 0, and then add the terms together.\nThat gives us the newest y coordinate 2. Thus, 'Squatch moves from -2 , -1 to -1, 2.\nJosh, I still don't get why we multiply matrices in such a strange way.\nWell, the reason why we multiply the numbers in a row by the numbers in a column and\nthen add the products together, is that when we use matrix multiplication to multiply\nthe first transformation by the second transformation, which means we multiply the\nfirst row by the first column and add the products, and we multiply the first row\nby the second column and add the products, and then we multiply the second row by\nthe first column and add the products. Note, because we are now using this second\nrow of the first matrix, we put the result in the second row in the output. In other\nwords, when we use a new row to do multiplication, we start a new row in the output\nmatrix. Anyway, then we multiply the second row by the second column and add the\nproducts. And we end up with a new two-by-two transformation that combines the effects\nof both transformations. The new combined transformation transforms the original\npoint directly to the newest point without having to take the intermediate steps.\nIn other words, matrix multiplication is funky so that we can easily combine a sequence\nof transformations into a single transformation. To convince ourselves that we can\nuse the combined transformation to transform the original point to the newest point\nwithout having to do the intermediate steps. Let's multiply the original coordinates\nby the combined transformation. So we start by plugging in the original coordinates\n2 and 1. Then we just do the row by column math. Beep boop boop boop boop beep boop boop boop boop, beep boop boop!\nAnd we see that the combined transformation transforms the original coordinates, 2, 1, to the newest coordinates, -1, 2.\nBam!\nNote, matrix multiplication is such that no matter how many times Taylor Swift changes\nher mind about which direction the stage should face, we can just multiply the individual\ntransformations together to get a combined transformation. And that combined transformation\nallows us to go directly from the original location to the final location without\nhaving to do the intermediate steps. Bam. Okay, now that we understand why matrix\nmultiplication is so funky, let's talk about another confusing thing about matrix\nmultiplication. Because matrix multiplication is applied row by column, we can't\njust reverse the order of the matrices and expect things to work out the same. In\nfact, we can't even multiply these two matrices together because each row in the\nfirst matrix has two numbers and each column in the second matrix only has one number.\nFor example, if we try to multiply the first row of numbers in the first matrix by\nthe first column of numbers in the second matrix, then we can multiply 0 by 2, but\nthere is nothing to multiply this 1 by. In contrast, it's possible to multiply these\ntwo matrices because we have the same number of values in this row that we have in\nthese columns. In general, the number of columns in the first matrix has to match\nthe number of rows in the second matrix. If, for some reason, we want to switch the\norder of these matrices, then we need to change each row into a corresponding column.\nNow we can do the multiplication correctly, row by column. Beep, beep, boop, beep,\nboop, beep, boop, boop. And after turning each row into a corresponding column, the\nmath works out just like it did before. Note, because we turned the rows in the original\nmatrices into columns, the output is now a column instead of a row. Oh no, it's the\ndreaded terminology alert. Changing the rows in a matrix into corresponding columns\nis called transposing a matrix, and we notate the transpose function with a superscript\n't'. So this matrix is the transpose of the original coordinates. This is the transpose\nof the combined transformation matrix, and this is the transpose of the newest coordinates.\nAs we saw earlier, the original coordinates were in a one row by two column row matrix,\nwhich is also called a 1 by 2 matrix. But after we transpose the original matrix,\nwe end up with a 2 row by 1 column column matrix, which is also called a 2 by 1 matrix.\nLikewise, the transformed coordinates were also in a 1 row by 2 column row matrix,\nor a 1 by 2 matrix, and the transposed matrix is a 2 row by 1 column column matrix,\nor a 2 by 1 matrix. In contrast, the combined transformation has 2 rows and 2 columns,\nand thus is a 2 by 2 matrix, and the transposed matrix also has 2 rows and 2 columns,\nand is also called a 2 by 2 matrix. Now, because writing out matrices can be cumbersome,\npeople often replace them with variable names. Matrices with only one row or column\nare usually referred to with lowercase letters that are sometimes italicized and\nsometimes not. And matrices with more than one row and column are usually referred\nto with uppercase letters that are also sometimes italicized and sometimes not. Out\nin the wild, you'll find additional notation styles, so just be prepared to be flexible.\nAnyway, given these variable names, we can rewrite the matrix equation like this 'a' times 'W' equals 'b'.\nOr we can refer to the transposed matrices by simply adding a superscript 't' to each\nvariable name and rewrite the matrix equation like this: the transpose of 'W' times\nthe transpose of 'a' equals the transpose of 'b'. Either way we do it, we will get\nthe same transformed coordinates. Note, we could have just as easily called this\ncolumn matrix 'a' and called this row matrix the transpose of 'a'. Likewise, we could\ncall this matrix 'W' and this matrix the transpose of 'W'. Lastly, this column matrix\ncould be called 'b', and this could be called the transpose of 'b'. Thus, we could\nalso write the matrix equations like these: the transpose of 'a' times the transpose\nof 'W' equals the transpose of 'b', or 'W' times 'a' equals 'b'. I mention these\nalternative ways to write the equations because out in the wild, you'll see all kinds\nof notation, so just be prepared to be flexible. Lastly, if we want to refer to an\nindividual value in a matrix, like this -1 in the bottom left-hand corner of the\nmatrix we originally named 'W', then, if the variable is uppercase, we convert it\nto lowercase and add subscript row and column indices like this. Bam! Now let's talk\nabout how all this applies to a neural network. This neural network may look really\nfancy, but all it does is take two measurements from an iris flower, the width of\na petal, which is this part of the flower, and the width of a sepal, which is this\npart of the flower. And with that information, it predicts the species either Setosa,\nVersicolor, or Virginica. Anyway, let's start with a petal that is 0.5 units wide\nand a sepal that is 0.4 units wide. The first thing this neural network does is multiply the petal width by -2.5.\nHey, wait a minute. Where did that -2.5 come from?\nAll of the numbers in this and pretty much every neural network come from something\ncalled backpropagation. And there's a lot to be said about backpropagation, so if\nyou want to learn about it, check out the Quests. Anyway, then we add that term to the sepal width multiplied by 0.6.\nAnd at the same time, this neural network also multiplies the petal width by -1.5.\nThen we add that term to the sepal width multiplied by 0.4. Now, even though we are\nlooking at these equations for the first time, they may remind you of the equations\nused to transform 'Squatch's ticket to see Taylor Swift.\nThose memories follow me around.\nIn both cases we have two variables multiplied by numbers, and then the terms are\nadded together. And just like how we converted the ticket transformation equations\ninto a matrix equation, we can convert the neural networks transformation into a\nmatrix equation, where we put the petal and sepal width into a row matrix and multiply\nthem by a matrix containing the coefficients, or the weights, in a neural network.\nNow we just plug in the petal and sepal widths, 0.5 and 0.4, and do the math by multiplying\nthe row of input values by the coefficients, or weights, in the first column and\nthen adding the terms, and then multiplying the row of input values by the weights\nin the second column and then adding the terms. And that gives us a 1 by 2 row matrix\nof transformed values, -1.0 and -0.6. So this first part of the neural network transforms\nthe input values. The next thing the neural network does is add bias values to each\ntransformed value. So that means we add 1.6 to the first transformed value and we\nadd 0.7 to the second transformed value. And when we do the addition, we end up with\na row matrix with 0.6 in the first column and 0.1 in the second column. Now we are\nready to use 0.6 as the x axis coordinate for the activation function on top and\n0.1 as the x axis coordinate for the activation function on the bottom. Now we run\nboth values through the ReLU activation functions. Note, there's a lot to be said\nabout the ReLU activation function, so if you're interested, check out the Quest.\nOtherwise, just know that the ReLU activation function outputs zero or the input\nvalue, whichever is larger. Thus, because the input for the top ReLU is 0.6, which\nis greater than 0, the output is 0.6. And because the input for the bottom ReLU is\n0.1, which is also greater than 0, the output is 0.1. Now the network has three transformations\nthat it applies to the ReLU output. And doing the row by column multiplication and\nsummation gives us this one-dimensional 1 by 3 row matrix. Then we add bias terms\nto each transformed value and we end up with the output values for Setosa, Versicolor,\nand Virginica. Bam! Thus, when the petal with is 0.5 and the sepal width is 0.4,\nthis neural network predicts that the measurements came from Versicolor, since its\noutput value is closest to one. So, now we've seen how this neural network can be\nwritten out as a sequence of matrix multiplication and addition. Bip bip bip bip\nbip bip bip. Now, if we call the row matrix, that contains the input values 'a',\nand call the first transformation matrix, which contains weights, w1, and the first\nrow matrix of bias terms b1, then we can use 'a' times w1 + b1 as the input for the\nReLU activation function. We can then multiply the output of the ReLU by the next\ntransformation matrix, which will call w2, because it also contains weights. And\nthen we add the final bias values, b2, to get the predicted species. And at long\nlast, we've seen how to transform a neural network into a super compact matrix equation.\nDouble bam! Now to review, let's go back to the matrix equations we started with\nand see if we can make sense of them. This first bit comes from the PyTorch documentation\nfor the nn.linear class. And we just saw that an equation like this corresponds to\na part of a neural network that looks like this. This 'x' corresponds to the input\nvalues, and like we saw before, the input values can be put in a one-dimensional\nrow matrix like this. This transpose of 'A' corresponds to the multiplication of\nthe inputs by the weights and the sums of the terms. And all that math, the multiplication\nby the weights and the summation of the terms, can be accomplished by multiplying\nthe inputs by a transformation matrix containing the weights. Note, the transpose\n't' of matrix 'a' just means that nn.linear assumes the weights are in rows. But\nin order for this math to work out, they need to be in columns. Lastly, this 'b'\ncorresponds to the addition of the bias terms to the result of the matrix multiplication,\nand that just means that we put the bias terms in a one-dimensional row matrix and\nadd those values to the transformed values. Thus, this simple neural network multiplies\nand adds these matrices, and that corresponds to this matrix equation.\nThus, the nn.linear class builds a very common part of a neural network. And do you\nremember how early on in this StatQuest we made a big deal about linear versus nonlinear\ntransformations and how a linear transformation means a constant change in the input\nresults in a constant change in the output? Well, this part of a neural network and\nits corresponding matrix equation is a linear transformation. And now we also know\nthat the nn.linear class gets its name from the fact that it performs a linear transformation.\nNow, let's talk about this error message. Although there are some technical differences\nbetween tensors and matrices, they have a lot in common and we know that a 1D, or\none dimensional matrix, is just a single row or a single column, but the function\nthat generated the error was expecting something with at least two dimensions like\nthis matrix. So the error just means we need to think a little bit more about the data we are giving the function.\nNote, there's a lot more to be said about tensors in the context of neural networks.\nSo if you want to learn more, check out the Quest. Lastly, this is the equation for\nattention, which is used by transformers, the neural network architecture behind\nChatGPT. Attention requires three matrices, 'Q', 'K', and 'V'. For example, given\nthese three matrices, the first thing we do is multiply 'Q' by the transpose of 'K'.\nAnd given these example matrices that gives us 11.7. Then we divide by the square\nroot root of d sub k. And for the sake of this example, let's assume d sub k equals\n2. So we divide 11.7 by the square root of 2, and that gives us 8.3. Note, if we\nhad started out with matrices that had more rows and columns, we'd end up with a\nlarger matrix at this point. And to be clear, attention is expecting larger matrices,\nand that is why these variable names are capitalized. However, we're just using row\nmatrices because they keep the example as simple as possible and they work. Anyway,\nnow we take the SoftMax of our little tiny matrix and we get 1.0. Note, this is probably\nthe world's lamest example of the SoftMax function in action, but if you want to\nlearn more about it, check out the Quest. Anyway. The last thing we do is multiply\nour tiny 1 row by 1 column matrix by 'V'. And that gives us these two values. And\nthose are the attention values for the example matrices. So given these matrices,\nwe might not know what attention means, but at least we can do the math. Bam? Well,\nif you want to know more about what attention does and means, check out the Quest.\nTriple Bam! Now it's time for some shameless self-promotion. If you want to review\nstatistics and machine learning offline, check out the StatQuest PDF study guides\nand my book, the StatQuest Illustrated Guide to Machine Learning at statquest.org.\nThere's something for everyone! Hooray! We've made it to the end of another exciting\nStatQuest. If you liked this StatQuest and want to see more, please subscribe. And\nif you want to support StatQuest, consider contributing to my patreon campaign, becoming\na channel member, buying one or two of my original songs, or a t-shirt, or a hoodie,\nor just donate. The links are in the description below. All right, until next time, Quest on.",
  "transcript_chars": 23071,
  "ingested_at": "2026-05-15T10:54:58.694607+00:00",
  "source": "channel",
  "yt_meta": {
    "view_count": 82172,
    "like_count": 2304,
    "channel_id": "UCtYLUTtgS3k1Fg4y5tAhLbw",
    "categories": [
      "Education"
    ],
    "tags": [
      "Josh Starmer",
      "StatQuest",
      "Machine Learning",
      "Statistics",
      "Data Science"
    ]
  }
}