{
  "video_id": "-PFIkkwWdnM",
  "channel_slug": "umarjamilai",
  "channel_handle": "umarjamilai",
  "title": "Kolmogorov-Arnold Networks: MLP vs KAN, Math, B-Splines, Universal Approximation Theorem",
  "duration_seconds": 4539.0,
  "url": "https://www.youtube.com/watch?v=-PFIkkwWdnM",
  "upload_date": "",
  "transcript": "hello guys welcome back to my Channel\ntoday we are going to talk about colog\nArnold networks colog Arnold networks\nare a new type of network that came came\nout in a paper around 10 days ago and\nthey are they want to compete with\nmultilayer perceptron so in this video I\nwill be reviewing what are multi-layer\nperceptrons because we want to compare\nthe two models then I will introduce\ndata fitting because this um um topic is\nneeded for us to understand bsic and B\nlines later on we will talk about the\nuniversal approximation theorem at least\nfor the multier perception and the kog\nAR representation theorem which is the\nequivalent for the uh cons later we will\ncompare the two architectures and look\nat some properties that this colog\nArnold networks have what are the\nprerequisite for watching this video\nwell for sure that you have a background\nin calculus for example you know what is\nthe derivative or what is the continuity\nof a function and uh you have a little\nbit of knowledge of neural networks even\nif I will review most of the concept\nthat we need to understand this new type\nof network so let's start our\njourney uh as you know the multi-layer\nperceptron is a neural network made up\nof layers of neurons each one one after\nanother organized in a feed forward way\nwhich means that the output of one layer\nbecomes the input of the next layer and\nusually at each layer we also put some\nnonlinear activ ation function for\nexample the reu in this case we have a\nvery simple Network as you can see we\nhave a input Vector of three features so\nyou can think of this as the first\nfeature this one as the second and this\none as the third then we have a first\nlayer made up of five\nNeons uh you can think of it in pych\nit's a network that is from Tes as input\nthree features and produces output Fe\nfive output features this network here\ntakes as input uh five features and\nproduces five output features and then\nwe have the last one which is the which\nhas the same structure usually we put\nactivation functions here at the\nlayers so let's view how they work in\npytorch if you look at the documentation\nyou will see that the linear layer\nperforms this very simple operation it\ntakes the input you can think of it as a\nvector made up of features or you can\nthink of it as a matrix made up of many\nitems each one having and input\nfeatures um then we multiply it by some\nweight Matrix they call it a I will call\nit a w I think it's more easier to\nunderstand this way and plus we have a\nbias let's analyze the structure in\ndetail so imagine we have a bnch of\ninput the we have like 10 inputs and\neach input Vector is made up of three\nfeatures I call here fub1 F2 and F3 the\nthree input feature our linear layer\nwill perform the following operation\nwhich is it will take the input multiply\nit by some weight Matrix that is made up\nof Weights that are learnable by the\nnetwork plus a bias the weight Matrix in\nthis case uh we are talking about a\nlinear layer that is taking as input\nthree features and is producing five\noutput features so the weight Matrix the\ntransposed of the weight Matrix will be\n3x 5 where you can think of each neuron\nas being one column of Weights so each\nneurons will have an um n weights if n\nis the number of input features one\nweight for each input feature and it\nwill produce some output so when we do\nthe X multiplied by W transposed it will\nperform this matrix multiplication which\nwill produce the following result let's\nanalyze how these values are generated\nthe first output feature of this uh\nMatrix because you can think of this\nMatrix as being a batch of 10 items\nbecause in the input we have 10 items as\num a batch of 10 items each one having\nfive features because the linear layer\nis from three features to five output\nfeatures the first feature is generated\nby the dot product of the first item\nwith the first\nneuron and it's this value here the\nsecond output feature of the first item\nis the values of the the features of the\nfirst item multiplied by the second\nneuron so the weights of the second\nneuron which means that each neuron is\nresponsible for one output feature in\nthis linear layer later after performing\nthis multiplication we also add a bias\nterm which in this case is a vector one\nfor each which has one value for each\nneuron and we broadcast it to all of the\num this metrix here which means that\neach output feature for each item will\nhave an additional term which is the\nbias associated with this particular\nneuron so the first neuron will add for\nthe first feature will have um the Value\nPlus B1 the second feature will have the\nvalue of the second feature plus B2 etc\netc and this will produce the output of\nthe linear layer so as you can see we\nstarted with a batch of 10 items with\nthree features each and we end up with\n10 items with five features each because\nthe linear layer is from three to five\nand if you want the formula on how to\ncalculate the uh the output of this\nmatrix it's here so basically it's the\nthe input features of each item\nmultiplied by the weight of the\ncorresponding neon plus the bias\nterm now we may be wondering why do we\nneed activation functions in linear\nlayers well let's analyze what happens\nwhen we don't have any activation\nfunction because uh imagine uh here\nlet's go back here imagine you only\napply linear layers one after another\nwithout introducing any nonlinearities\nlike Theo what will happen well um let's\ndo this product by hand so uh imagine we\nhave an input we perform uh the\nmultiplication with the weights of the\nfirst layer I call it W1 plus the bias\nof the first layer and this will produce\nsome output just like before imagine we\nare going from three features to five\nfeatures it means that the input is okay\nin this case it's a vector but you can\nthink of it also as a batch it will be\n10 by or 1 by three and it will produce\nan output that is\n1x5 then we apply a second linear layer\nwhich means that the output of the first\nlinear layer becomes the input of the\nsecond linear layer because we are not\napplying any nonlinearities for now so\nit will become the input of the second\nlinear layer will be will be the output\nof the first one so it's this is the new\ninput if we expand the express of this\ninput it becomes like this we distribute\nthis product it will become this\nexpression here and you can see that\nthis expression is nothing more than\njust a linear relationship between the\ninput and the output uh let me draw here\nso this one is just a kind of you can\nthink of it as a weight Matrix and this\none is just a pi term so the input is\nrelated to the output by a linear\nrelationship which means that our\nmulti-layer perceptron without any\nnonlinear activation functions will only\nbe able to map um data that has linear\nrelationship um which means that what is\na linear relationship which means that\nin two Dimensions the the the data is on\na line or can be separated by the line a\nline in the case we are doing\nclassification or in the case of three\ndimensions the data is on the plane or\nit can be separated by a plane in case\nwe are doing classification or in the\nmultiple Dimensions it can be separated\nby a hyper plane so if the data is not\nlinearly separable we say then it cannot\nbe modeled by the multi-layer perceptron\nwithout activation functions let me show\nyou with a concrete\nexample um from the Boolean Gates you\nknow the end gate and the exor gate so\nthe end function takes as input is a\nbuan\nfunction that takes as input two bans\nand produces the end of\nthem let's do it here so uh let's call\nit input\none input two and the output of this\nend for 0 0 it is zero for 01 it is zero\nfor 1 0 it is 0er and for 1 one it is\none so the end is one only when both are\nboth the input are one if we plot this\nuh the input and the output relationship\nso suppose this is the first input so\ninput one and this\nis uh input two suppose also that here\nwe have the zero and here we have one\nand here we have zero and here we have\none it's one only here and all the other\nplaces it's zero now we can always draw\na line for example this one that can\nseparate the zero and the one in case we\nwant to create a Neal Network that\ncalculates the end and we can do that\nbecause the data is linearly\nseparable um however we cannot do this\nfor the exor so let me show you\nhere so if we have uh let's R the truth\ntable for theor it\nis so it's input one\ninput two and the output so we have 0 0\nand it's zero 0 one it is one 1 Z it is\n1 and 1 one it is zero if we draw\nthis let's say this is input one and\nthis is input\ntwo 0\n1 so it is one when you have zero 0 1 or\n1 0 and it's zero in the other places\nnow here no matter where you put the\nline you will never be able to separate\nall the zeros from all the ones because\nif we put the line here you still get in\nthis region 0 one one and this region\nzero if we put the line like this we\nstill get 01 and 01 if we put the line\nlike this we still get 01 so because the\ndata is not linearly separable it cannot\nbe uh learned by um neural network\nwithout uh act nonlinear activation\nfunctions this is why we need\nnonlinearities and later we will see the\ntheorem that is backing this now let's\ntalk about another topic which is data\nfitting now data fitting means that we\nhave a series of points and we want to\nfit a line usually a polinomial line\nbetween that passes through them now I\nwant you to imagine this scenario so\nimagine we are creating a game a 2d game\nin which we have a character that we\nwant to animate through a part of made\nup of points\nof course one way uh to animate this\ncharacter would be to uh take these\npoints and make straight lines between\nthem so we can animate our character\nfrom here to here from here to here from\nhere to here but it doesn't look so good\nright it's not so smooth this movement\nyou would see the very rigid movement of\nthis character so one better way is to\nmake a a polinomial curve that is more\ncurvy that passes through these points\nand generates a more smooth path and\nthis looks better it's more beautiful\nlet's say and one way to do it is to\ncreate a polinomial line that passes\nthrough these points that is curvy how\nto do that practically well U if you\nremember from high school when you have\ntwo points you can only draw a line\nbetween them and if you have three non-\ncinear points means that they are not on\nthe same straight line you can uh make a\nquadratic curve pass through them like a\nparabola for example and if you have\nfour points you need an equation of\ndegree 3 to make a polinomial line that\npasses through them how to calculate the\nthe equation of this line that passes\nthrough all these points well it's very\neasy we for example imagine we have four\npoints and for example This One X and Y\nyou can see here we have four points and\nwhat we do is we write the generic\nequation of the polinomial curve of n\nminus one Dee because we have four\npoints so n minus one is third degree\nand then we create a system of equations\nin which we impose the fact that this\ncurve has to pass from all these points\nso we write the equation of the curve by\nsubstituting the coordinates of the\nfirst point through it then from the\nsecond point then the third and then the\nfourth now this is a system of equations\nthat can be solved to retrieve the\ncoefficients needed to get this line you\ncan do it with linear algebra you can do\nit with Excel whatever you like or you\njust you the substitution me method\nwhich means that you get the for example\nthe a variable from here from this uh\nexpression you substituted the second um\nequation and you will get an equation\nthat only depends on b c and d then you\nget another variable and you replace it\nin the third one so you get one less\nvariable to think about and then you do\nit then until you arrive to the last\nequation in which you will only have one\nvariable you solve for it and then you\nreplace it in the other to get the other\nthe values of the other variables or you\ncan use Excel\nyou can draw the the trend line This is\ncalled the trend line and the Excel can\nalso write you the equation of the curve\nthat passes from these\npoints and this is a very simple method\nfor generating polinomial curves now\nwhat if we have hundreds of points and\nwe want to generate a smooth line that\npasses through them we can do that so\nthe procedure that we saw before we can\ndo that also with hundreds of points but\nwe have two problems the first problem\nis the computational complexity because\nyou would need to solve a very big um\nsystem of equations and the second thing\nis that as the number of points grows\nthis polinomial line starts to behave\nmore and more in a weird\nway at the extremes for example this is\nthe plot of a polinomial degree Tre line\nthat passes from 11 points as you can\nsee in the middle we can still accept\nthis kind of curvy lines but in the\nextremes the the curves starts behaving\nin a crazy way it means that if we\nanimate our character using this\npolinomial line it will move like this\nit will move down and then go out of the\nscreen and then come back and then etc\netc and then again we'll go out of the\nscreen and then again we'll go out of\nthe screen and then come back so as you\ncan see we want some way of also\ncontrolling how smooth this line is and\nto not make it um go crazy like this in\nthe extremes that's why someone studied\nuh this problem and solve it by\nintroducing B curves so let's see how\nthey work now a Bia curve is a\nparametric curve and later we will see\nwhat it means to be parametric that\nallow us to draw a smooth line given a\nseries of points without the\ncomplexities that we saw\nbefore um it is parametric in a sense\nthat um it all it allow us to calculate\nto calculate the equation of the\ninterpolated curve so the curve that\npasses from these points but we will see\nthat in the case of a z curve it does\nnot pass from all the points just get\nclose close to them so it's parametric\nin the sense that the um coordinates of\nthe point on this interpolated curve\ndepends on a independent variable called\nT you can think of it as time that goes\nfrom 0o to one in this case for example\nimagine we have only two points as you\nknow with two points you can only draw a\nline and the basic curve allow us to\ndraw a line so imagine you want to make\na line between p 0 and P1 what do we do\nwe start from p 0 and we go towards P1\nand this is our interpolated line and as\nyou can see as time moves the point gets\ncloser to the second point and moves\naway from the first point so you can\nthink of the variable t as the variable\ntime or the percentage of the animation\nthat you are trying to make because uh\nremember always the scenario that we are\nworking in which is we have a game and\nwe want to animate our character between\npoints so at time Step Zero the I the\ncharacter is still at the point zero as\ntime moves on it moves toward the second\npoint and this is the linear\ninterpolation and looks quite good I\nmean it's it's a line so how to get the\nequation of this point as you can see\nwith respect to time well this is the\nequation and it's very simple and it's\nvery intuitive so uh the equation is B\nbasically we start it's a p 0 plus\nsomething that is kind of a percentage\nand you can think of this term here as\nthe vector that go from p 0 to P1 which\nmeans that we start from p 0 and as time\nmoves on we add some percentage of this\nVector that move takes us from p 0 to P1\nso when it will be 0.5 for example it\nmeans that it's p 0 plus half the\ndistance from p 0 to P1 so it make also\nintuitive Sense on how we get the\nequation of this uh point on this\ninterpolated curve now this curve is\nparametric so then that's also the\nreason why you can see this B as bold\nand T is not bold because it can be a\nvector it means that this point can be\nin X and Y or Z coordinates and all of\nthese coordinates will depend on the T\nvariable so at each time step it will\ntell us the position of our character in\nthis 2D game that you are creating uh\nwith respect to time okay the example I\nshow you is very simple for uh the\nlinear curve between these two points\nbut of course we can extend it to three\npoints so if we have three points we can\ndraw a very smooth curve that um that\ninterpolates them however unlike the\nprevious case in which we have the\npolinomial curve that passes through all\nthe points in our path in the case of B\ncurve it only passes from the first\npoint and the last point and\ninterpolates between the intermediate\nones so is is not touching the\nintermediate ones it just goes closer to\nthem so how to calculate the the the the\nthe equation of this interpolated curve\nwhich is the red one well it's very\nsimple also in this case what we do is\nwe do a recursive calculation which\nmeans that we first do a linear\ninterpolation between p 0 and P1 and we\nknow how to do that we have this\nequation here so let me write here some\nstuff so we saw that to get the linear\ninterpolation between two points you can\nuse this expression which can also be\nwritten like this so I just rearranged\nthe terms it's nothing fancy\nhere um and to get for three points we\nfirst do a linear interpolation between\np 0 and P1 which means that we apply\nthis expression so this ex this\nexpression here to q0 to p 0 and P1 and\nit's this one and we get a point that is\nq0 that in time we move from p 0 to P1\njust like this case here then we do\nanother linear interpolation between P1\nso this point here and this point here\nand it will result in a new point q1\nthat in time will move from P1 to\nP2 these two points q0 and q1 we create\nanother linear interpolation between\nthem so we do another linear\ninterpolation between q0 and\nq1 and this will give us the coordinate\nof the interpolated point of the uh the\ncurve that interpolates three points so\nfirst we do the linear interpolation\nbetween the first two points another\nlinear interpolation between the other\ntwo points and then we interpolate the\ninterpolated curves between these two\npoints between these three points and\nthis will give us the the equation of\nthe last the final curve that\ninterpolates the three of them so if we\nreplace the expression of q0 and q1 we\nget this one here\nand then we can rearrange the terms to\nget this last expression and this is the\num this expression tells us the\ncoordinates of the point of our\ncharacter in this red curve you can see\nhere as time moves on so at a Time setep\nZero it will be exactly here at Point p\n0 at time step one it will be exactly in\nthe point P2 and at time step let's say\n0.5 it will be more or less here\nhalfway so this is how you read this\nexpression and um and as we can see the\nthe in the basic curve we only touch the\nfirst point and the last point in the\npath but not the intermediate points we\ncan extend this also to Four Points and\nalso by using this recursive uh\ncalculation which is we do the linear\ninterpolation between p 0 and P1 the\nlinear interpolation between P1 and P2\nthe linear interpolation between P2 and\nP3 we will get three points in this case\nin this interpolated uh curves then we\ndo the linear interpolation between the\nfirst and the second the second and the\nthird this will give us two points and\nthen we do another linear interpolation\nbetween the first and the second in this\nuh third level interpolation so as you\ncan see we do a kind of a recursive um\nreasoning to arrive to the equation of\nthe interpolated curve that interpolates\nall of these points and also when we\nhave four points as you you can see we\ntouch the first point and the last one\nuh but not the intermediate\nones uh one thing nice thing about basic\ncurve is that we have an expression to\ncalculate the equation of the B curve\nwithout doing all this recursive\ncalculation so if we have a series of\npoints let's um n + one points so uh\nthese points here so p 0 P1 P2 P3\nPN we can calculate the expression of\nthe interpolated curve between these\npoints doing using this expression here\nin which you see one term that is called\nthe B basis\npolinomial uh which depends only on T\neach one multiplied by a point uh a\npoint now a little bit of terminology\nthe points in a basic curve but later we\nwill see also in baselines are called\nthe control points because they are the\npoints in which we want to interpolate\nwe want our curve to\ninterpolate and uh the B basis\npolinomial is one one for each control\npoint it tells later we will see what\nthey mean but it's this expression here\nso it's a polinomial in t that depends\non the point because as you can see\nthere is I here and it's there is I also\nhere and on the number of points that we\nhave uh this term here so ni I you can\nsee it's called the binomial coefficient\nand it has a very simple formula so it\nis the the um the factorial of nide by i\nn multiplied by n n minus I factorial\nwhat does it mean in practice if you\nremember from high school the binomial\ncoefficients tell us the coefficients of\nthe binomial when it's raised to the end\npower so when you do a + b to the power\nof two you get some coefficients and I\ncan write them here so here you have 1 a\nto the^ of two here you have 2 a + 1 B2\nand they are the same coefficients that\nwe get in the previous equation then\nwhen we calculated\nthe the Bas curve between three points\nso here we get one because as you can\nsee here we don't have any coefficient\nthen we have a two and then we have\nanother one here because this is the\nsecond order um uh B curve because it's\na quadratic\nequation and um and uh so basically you\nif you do it for two points you will get\nthe same equation that we arrived to in\nthe previous\nslide now these B State polinomial as\nyou can see we have the equation of the\nfinal curve so this uh B of T you can\nsee here is obtained by a summation of\neach point multiplied by this function\ncalled B and basis polinomial so how to\ninterpret them well if I plot the B\npolinomial for example having um in this\ncase I think we have one 1 2 3 four four\nfour points so this is a basic curve of\ndegree three in which we have four\npoints uh how to this will result in The\nFollowing polinomial being generated so\nhow to interpret this uh factors here so\nthese polinomial here well since we are\nmultiplying each polinomial by the point\nthis polinomial tells us what is the\ncontribution of each point to the uh\nequation of the curve of the\ninterpolated\ncurve let's see for example this example\nhere so at the time Step Zero because\nyou can think of the x axis as the T\ntime Step at time Step Zero only one\ncurve is contributing to the equation of\nthe LA the final curve you can see here\nin this summation and it's this blue\nline here and it's the polinomial\nassociated with the first\npoint um\nand it means that at the time Step Zero\nthere is the our character which we are\nanimating to this point will be exactly\nat the point the first point in our um\nuh basic curve because it's the only one\ncontributing to the coordinates of the\ninterpolated curve but at time moves on\nwe start moving away from the first\npoint and we start getting closer to the\nsecond point which is this green line\nhere and more or less L at 0.33 we will\nbe getting the closest to the second\npoint then we start moving away from the\nsecond point and we start getting closer\nto the third point which is this red\nline here and we will get the closest to\nthe third point when we reach 0.66 so\n66% of the animation percentage and then\nwe start moving away for the third point\nand we start getting closer to the\nfourth Point as times moves on and when\nwe reach time step\n100% we will be exactly at the last\npoint in our bizer curve so that's why\nwe will touch only the first point and\nthe second point but not all the\nintermediate points because when we are\nin the intermediate phases of the time\nstep there are multiple points\ncontributing to the final coordinate on\nthe interpolated point so they will kind\nof it's like one point is pulling the\ncurve in this direction but there is\nanother point that is also pulling the\nthe the the curve towards itself so\nthere is no one single point that wins\nin the intermediate step but at the\nbeginning at the end there is only one\npoint that is kind of winning in this\npulling\ngame okay now uh we have seen Biz curves\nnow imagine that we have 100 points so\nwith Biz curve when you have 100 points\nto interpolate you need a basic curve of\ndegree 99 which can be quite uh compet\naddtionally expensive to\ncalculate and um so someone thought well\nwhy don't instead of extending this Bas\ncurve to very big polinomial why don't\nwe stitch them multiple of them together\nso we take if we have 100 points I can\ndraw many B curve and then find a way to\nstitch them together such that they also\nhave some nice properties and this is\nhow B plines work so let's see them in\ndetail now imagine we have six points\nand we want to uh generate a line that\ninterpolates between\nthem uh one way of course is to generate\na Bas curve of degree five that\ninterpolates between these six points\nanother way is to do a bis line in which\nwe choose the degree of the basier curve\nthat we want we generate multiple of\nthem and then we stitch them together\nand this is what is happening here so\nlet me show you with a concrete example\nI took this from W from Alpha\nuh imagine we have these six points and\nwe want each uh basic curve to be of\ndegree two which means that it's a\nquadratic equation Bas a curve of degree\n2 needs three points so what we can see\nis that uh the first three points so one\ntwo and three will be one basic curve\nthe another basic Curve will be the\npoint two three and four this will be\nanother basic curve the point 345 will\ngenerate another basic curve and the\npoint 456 will generate another Basia\ncurve so in total we have four Basia\ncurves and the points where they meet\nare called knots and you can see them\nhere this red um red dots\nhere now the Bas curve the be plines uh\nallow us to basically have this basic\ncurve meet in this Nots and also Define\nsome nice properties that we will see\nlater but the the the thing that you\nneed to understand now is that we have\nsix points and we have four basic curve\nso when you have n points and each\nbasier curve that you want is of degree\nk then you get n minus K Bas curves so\nif you want to visualize them here in\nthis uh graph I can draw them so between\nthe first three points we get one Bas\ncurve which is this one between the\nsecond three points you get another one\nwhich is this one uh between the third\nthree points you get another one which\nis this one\noops and then between the last three\npoints you get another one and then we\nstitch them together so this one and\nthis one will be stitched here here at\nthis point and this one and this one\nwill be stitched here at this point and\nthis one and this one will be stitched\nhere at this\npoint if I increase the degree of the b\nline I will get less busier curve so I\nwill get less knots let's see so now I\nincrease the degree of the basic curve\nto three I get less basic curve because\nnow each basier curve needs Four Points\nso they have less stitching points and\nso now we have only three basic curve\none is between 1 2 3 and four the second\none is between\num uh two uh three four five and the\nthird one is between three four five and\nsix so we have three basic curve when\nyou stitch them together you get two\nknots\num let's see some more um properties so\nthe first property that this Beast ples\ngives us is the\ncontinuity So based on the degree of the\nB curve that makes up this bis planine\nwe get different level of continuity if\nyou remember from high school what is\ncontinuity continuity means that the\nfunction evaluation when you uh okay\nfirst of all simplified version of\ncontinuity if you remember from high\nschool a function is continuous when you\ncan draw the graph of the function with\never without ever leaving the pen from\nthe paper so this function is continuous\nwhile this function here for example is\nnot continuous because I uh there is a\ndiscontinuity Point here you can see\nhere um in mathematical terms the\ncontinuity means that the limit from\nleft and from right it equal to the\nevaluation of the function at that point\nso it means that if if I go if I go I\nnavigate this function from right and\nfrom left in the plot I will reach the\nsame height which is the case here here\nis continuity because if I go like this\nor go like this I will reach the same\npoint but it's not the same here because\nif I go from right I arrive here at this\npoint but if I go from left I arrive\nhere at this point and these two points\ndo not match so it's not\ncontinuous with B lines we get some\ndifferent depending on the degree of the\nunderlying basic curve we get different\nlevels of continuity when we have a a b\nline of degree\n2 we get C1 continuity uh C 0 continuity\nwhich means that the function here\nbasically it means that the two basic\nCurve will touch because it means that\nthe function evaluation from left from\nleft and from right will be the same so\nthe two functions will touch\num when we increase the degree of the\nbasic curve so we use the cubic for\nexample it will give us a C1 continuity\nwhat does it mean to be C1 continuous it\nmeans that the function is continuous in\nthe point in the\nNots uh but also that its derivative its\nfirst derivative will be\ncontinuous which means that the first\nderivative from right and from left will\nbe the same uh and you can see that here\nso if you remember what is the first\nderivative the value of the first\nderivative it indicates the inclination\nof the tangent line so as you can see\nhere the inclination of the tangent line\nif I move in this function from right\nit's the same as the one on the left and\nif we increase further the degree of the\nbasic curve that makes up these B lines\nwe get C2 continuity which means that\nnot only the function is continuous at\nthe notes not only the first derivative\nis continuous but also the second\nderivative is continuous and equal what\ndoes it mean practically if you remember\nalso the second derivative indicates the\nconcavity of the function in the point\nthe concavity can be ascending or it can\nbe descending in this case for example\nwe do not have C2 continuity so as you\ncan see if I move the function from\nright here at this note I get upward\nconcavity because the function is moving\nupward but if I move from left I get\ndownward\nconcavity which is not the case is here\nbecause now we have um C2 continuity so\nif I move from right it's upward and if\nI move from left I still get upward so\nas you can see the be lines also let us\ncontrol the level of continuity that we\nwant in the underlying curves by\nincreasing the degree of the bis\nPline uh there is a nice formula also\nfor calculating bis plines which is very\nsimilar to the formula that we got for\nthe B function so given a series of\npoints that they are called pi and Pi\ngoes from 0 to n so we have n + one\npoints we can calculate the equation of\nthe final interpolated curve using each\npoint multiplied by a basis function\nwhich just like in the case of the basic\ncurve indicates the contribution to of\neach point to the final interpolated\ncurve let's visualize it with example uh\nbut okay first this B this basis\nfunction functions are generated using a\nrecursive formula you can see here so to\ncalculate the because the basis\nfunctions the B lines depends on the\ndegree that you want to choose you can\nchoose the underlying Basia curve to be\nquadratic to be cubic to be degree 4 Etc\nso you uh you can um for the same number\nof points you will have a different\nbasis\nfunctions and the the basis functions to\ncalculate the basis function of the K uh\nlevel B planine you the K minus one\nlevels expression and it's this\nrecursive formula you can see here I\nmean there is a there is no strange term\nhere it's just a polinomial function\nthat uses the previous expression so K\nminus one to calculate the next one so\nyou just you start from this one for\nexample this is the first degree which\nis just a number one in the case um the\nthe point Falls between these two kns or\nnot and then you increase it and you use\nthe the previous one to calculate the\nnext one anyway let's look at the basis\nfunctions like we did for the basic\ncurves so the basis functions basically\nonce you fix the number of points that\nyou have and the degree of the basic\ncurve that makes up these pie lines you\nget some fixed basis functions that tell\nhow much each point will contribute to\nthe expression of the interpolation line\nlet me show you here with the\ninteractive\nexample um here in this case I have a\nBas a curve of degree three and I have\nsix points and you can see the this the\nbasis functions are this one which means\nthat the first point will contribute the\nmost to the um to the position of the\nour character on The Animated line when\ntime step is zero so it means that our\ncharacter will be exactly on the first\npoint because all the other basis\nfunctions are zero here what as time\nmoves on we start going away from the\nfirst point and start getting closer to\nthe second point which happens at time\nstep uh let's say here\n0 15 I think so at 15% more or less we\nare the closest to the second point and\nyou can see here right when s time moves\nfurther we start going away from the\nsecond point and start getting closer to\nthe third point and you can see here so\nwe start going away from the first point\nand start getting closer to the second\nuh to the third\npoint and etc etc for all the other\npoints until we reach the last point in\nwhich all the bases functions go to zero\nand we start getting very very very\nclose to the last point until we touch\nit because it's at the time step equal\none it will be the only one contributing\nto the position of our character in the\nuh interpolated curve so because it's\nthe only one contributing it means that\nit will exactly pass from that point and\nthis is the case as you can see so we\npass from the first point and the last\npoint but not the intermediate ones and\nnow you also have a demonstration of how\nit\nworks there is another very interesting\nproperty that we will use in the colog\nof Arnold networks which is local\ncontrol local control means that if I\nmove let me repeat this animation again\nso if if I move a point in a bis\nline it will change the shape of the\ninterpolated line only in the locality\nin the proximity of the point that I'm\nmoving it will not move all the rest of\nthe curve this depends on the degree of\nthe B Bas function that you are using uh\nin this case as you can see when I move\nthe fifth control point it's only\naffecting the line that passes close to\nthe fourth point and the sixth point but\nnot here for example here you see some\nlittle changes mostly because of the\nredrawing it's not like the the the the\ncurve has changed it's mostly for the\nrendering as you can see here so this is\na very interesting property that we will\nexploit\nlater thank you for bearing me with me\nso far I know that it has been a long\njourney uh but I promise we will go to\nthe kmog of Arnold networks very briefly\nbut first we need to talk about the\nuniversal approximation theorem so as\nyou know you can think of Neal networks\nas universal fun function approximators\nwhy usually we have some training data\nuh suppose that we have a classification\ntask like we have some pictures of cats\nand dogs and we have the corresponding\nlabel that tells us if this picture is a\ncat or it's a\ndog uh we train a network to classify\nthis cats and\ndogs but uh why because we we think that\nthis this training data comes from a\nfunction that is an ideal function that\nwe do not have access to that can match\nperfectly the picture of the cat with\nthe label cat and the picture of a dog\nwith the with the label dog and we try\nto approximate this ideal function that\nwe do not have with an approximate\nfunction that is our Neal\nnetwork but how do we know that if our\nNeal Network can even approximate this\nideal function how do we know that if\nthat our n network is powerful enough to\nlearn this mapping between input and\noutput well we have some theoretical\nresults that gives us some guarantees\nand the the universal approximation\ntheorem it's about this it tells us what\nare the limits of what neural networks\ncan learn and it has been proven that\nneural network with a certain depth\nwhich means certain number of layers and\ncertain WID which means the certain\nnumber of neurons when using\nnon-continuous uh nonlinear activation\nfunctions for example the Rel function\nthey can approximate any function and I\nalso want to emphasize on what it means\nto approximate a function it means that\nif we have an ideal function that we\nthat we want to\napproximate we can get a an\napproximation that is as good as we want\nit means that we can choose an error\nrate between the um between the ideal\nfunction and the approximate function\nand we can make our approximation as\ngood as we want below this error\nrate however this is a theoretical\nresult so this is why first of all that\nwe can use neural networks when we use\nneural networks we have some guarantees\nthat tells us okay our neural network is\nable to learn this data then how to\nlearn it or which weights to use or how\nmany depth layers to use or how many\nnarrow to use or which activation\nfunctions to use or how to structure it\nthat's of course an undefined problem\nthese are practical problems but we have\nsome Theory theoretical result that says\nthat at least we can do it the our Neal\nnetwork has The expressive power to u to\nmap this data to this to to learn this\nrelationship between input and\noutput of course we have some practical\nproblems when actually training a neural\nnetwork to learn some uh the training\ndata first of all to get good\napproximation it may take some\ncomputational power that we do not have\nmaybe it will take you know thousands of\ngpus that probably we do not have uh it\nmay need a larger quantity of data than\ntraining data that we do not have or\nmaybe our Hardware may not support the\nrepresentation needed for the weights to\ncarry on this training for example\nimagine that an ideal approximation\nwithin that particular error rate that\nwe have chosen needs the weights to be\n0.00000000 something but of course with\nthe 32bit we cannot represent such big\nsmall numbers with high Precision so of\ncourse we also have some Hardware\nlimitations on what we can actually do\nto train our Network to be good\napproximators and also of course there\nis the point of the optimizer right it\nalso depends if our Optimizer can gets\nstuck in the local Minima so it will\nnever be able to go to the actual good\napproximation that we are trying to\nachieve so just because a Neal learn\nNetwork can learn anything doesn't\ndoesn't mean that we are actually able\nto do it in practice but at least we\nhave some we we know that our limits are\npractical now this is valid for neural\nnetworks so the multi-layer perception\nthat we have seen\nbefore we start our tration of the cor\nkogo Arnold networks by introducing the\nkogo Arnold representation theorem which\nis the theoretical backing for the kogo\nArnold networks just like the universal\napproximation theorem is the theoretical\nbacking for the multi-layer perceptor it\ntells us that if we have a function that\nis a multivariate function it means that\nas input you have a vector of features\non a bounded domain and this is the\ndomain then it can be written as a\nfinite composition it means that this\nsummation is a finite and not infinite\nof functions of single variable which\nare this Fe here and the binary addition\nof operation so basically it says that\nany function that we want can be\napproximated like this more or less\nwithin the uh constraint written here\nand this is the basis foundation of the\nkog Arnold networks so they exploit this\ntheoretical backing to say we can\napproximate any function if we structure\nour Network like this so now let's\nexplore how this works in practice so\nlet's rewrite our um the function of the\ncologo Arnold representation theorem\nwhich is this one and suppose that we\nhave an input and input that is made up\nof two features so the in the the\nformula here n indicates the number of\nfeatures of the incoming Vector X in\nthis case suppose that we have two\nfeatures so X1 and X2 this is the input\nof our\nNetwork then we as you can see from the\nformula this um each feature is run\nthrough a function called Fe which\ndepends on two indices one is q and one\nis p q goes from one to 2 N + 1 n is\nequal to 2 so 2 n + 1 is equal to 5 so Q\nwill go from 1 to 5 and P is goes from\none to n so n is the number of features\nso P will go from 1 to two so we have q\nthat goes from 1 to 5 and P that goes\nfrom 1 to two so in total we will have\n10 functions Fe 1 one Fe 2 one Fe 31 Fe\n41 51 etc etc until we reach 52\nthen what happens well we um run for\nexample the first uh suppose Q is equal\nto 1 we run X1 and X2 through\nF1 1 and\nF12 as follows they will produce some\nresults and these results are summed up\nthrough this summation you can see here\nthen the result of this summation is run\nto another function called F1\nand it's this function here we do it for\nall the values of Q so we do it for Q is\nequal to 2 Q is equal to three four and\nfive this will produce Fe um the output\nof F1 F2 F3 F4 F5 and then we run this\nsummation here you can see here now I\nknow that all this looks abstract so I\nwant to give you some analogy with the\nmulti-layer perception to understand\nwhat is happening so in the multi layer\nperceptron if you remember we have two\ninput feature or we have three input\nfeature whatever and then what we do we\nhave some neurons the neurons multiply\neach input feature with one weight in\nthis case we are not multiplying the\ninput feature with any weight we are\nrunning it through a function as you can\nsee\nhere and then we are summing up the\nresults of these functions which is also\nsomething that happened in the Neal\nNetworks multilayer\npons so let me write here so in the Neal\nNetwork in multilayer perceptron here\nyou had like W1 for the first neuron and\nhere you had W2 for the second neuron\nyou take the X2 multiplied by W2 X1 mtip\nthe W1 and then you sum them up here and\nthis would be the output of the first\nneuron in case there is no bias in this\ncase we are not multiplying them with a\nweight but we are running them through\nfunction and these functions are\nlearnable so the kog Arnold networks\ninstead of having weights have functions\nand instead of having weights that are\nlearnable they have functions that are\nlearnable then we take this uh output\nhere we run it through another uh\nfunction this one here um and then we\nsum the results of all this uh functions\nnow um you can think of this\nthis network not as as two layers so\nbecause this we want to introduce the\nmulti-layer colog arold networ so we\nneed to understand what what it means to\nhave a multiple\nlayers by analogy with the multi-layer\nperceptron you we can think of this\nnetwork as being multi-layer Network in\nwhich the first layer is going from two\nfeatures to five features where each\nfeature is produced by not two weights\nbut by two functions\nso you can think of this as the output\nof a neuron in the colog Arnold networks\nthat takes as input the first feature\nruns it through a function takes as\ninput the second feature runs it through\nanother function and then sums up the\nresult and produce one feature we do it\nfor the second feature we do it for the\nthird feature we do it for the fourth\nfeature and the fifth features then what\ndo we do we have another Network layer\nthat is going from five features to one\nfeature so we have have um we run each\nof this feature through a function it's\nlike we are multiplying by a weight in\nthe multi-layer perceptron and then we\nsum up all these um uh the the results\nof these functions in the case of the\nmulti layer percepton we would sum what\neach feature multiply by its weight in\nthis case we are summing up each feature\nrunning through its own\nfunction so as you can see we have now\nthe concept of multi-layer colog arog\nnetworks because this can be thought of\nas a first layer that is going from two\nfeatures to five and the second layer\nthat is going from five features to one\ninstead of having each features\nmultiplied by its weight we have each\nfeature running through its own function\ninstead of having learnable weights we\nhave learnable functions and later we\nwill see what do we learn of these\nfunctions because I didn't tell you what\nare we learning from this function how\ndo we learn this function what are the\nparameters that we are learning in these\nfunctions and in the paper they also\nmake a very nice comparison between the\nmultilayer perceptron and the kog of\nAral networks so as I said before we\nhave first of all in the multilayer\nperceptron the theorem that is backing\nthe their expressive power is called the\nuniversal approximation theorem in the\ncase of the cologo AR Network it's the\ncolog arold representation theorem and\nin the case of the multilayer perceptron\nwe have some input features so in this\ncase we have two features so let's call\nthis one X1 and X\nX2 they say that we have learnable\nweights on edges because actually if you\nremember each X1 is multiplied by W1 for\nthe first neon and X2 is multiplied by\nW2 so you can think of here having a\nweight that is learnable then we sum up\nso X1 multiplied by W1 X2 multiplied by\nW2 then we sum up we run it through some\nactivation function in this case it is\nthe Rel function looks like the re\nfunction then the output\nis and it will produce some output and\nthen we have another layer that is going\nfrom these are this will produce five\nfeatures so this is five\nfeatures and uh and then we have another\nlayer that is going from five features\nto one output feature uh so it's also a\nmulti-layer uh percepton here because we\nhave two layers one is from 2 to 5 and\nthe second one is from 5 to one here in\nthe case of the con we also have a two\nto five and 5 to one because we have\nhere two features and here we have five\nfeatures and here we have one feature\nbut instead of having weights we have\nfunctions like we saw before so each\ninput feature is not multiplied by a\nweight but it's run to a function that\nis learnable and then we sum\nup\num and this will produce one feature the\nsecond feature the third feature the\nfourth feature and the fifth feature\nthen we have another layer that will\ntake each feature and we'll run run it\nthrough a network instead of multiplying\nby the weight and then we sum the all of\nthem to to produce one output\nfeature um here you can see they write\nthat we have the sum operation no and\nthis is what we did so each feature run\nthrough its own function and then we sum\num uh each um for each output feature\nthat we want to produce at each layer\nand this is uh the comparison of how it\nworks now let's look at their\nproperties now the multilayer account\nokay I have already described it how it\nworks but they also make a\nrepresentation in the paper so you can\nthink of this as a multi-layer uh Con in\nwhich the first layer is going from two\nfeatures so here you have two features\nto five and then from five to one and\ninstead of having weights we have these\nfunctions that are learnable so now\nlet's concentrate on what does what do\nwe learn from these\nfunctions um here in the paper they also\nwrite the implementation details so the\nformula for the uh multi-layer uh not\nfor the for the colog network is this\none and we have this Fe function here\nthat are\nlearnable uh here even if the expression\nis different actually you have to think\nthat this fq and fqp are both functions\nthat are learnable so there is no big\ndifference between they are both Bas\nples and they are\nlearnable but um they they model this Fe\nof X which are the functions that we\nwant to learn not as has only a b line\nbut as a b line plus something something\nthat is fixed and it's another function\nthat is the suu function here\num so the only thing that we are\nactually learning is this part of the\nfunction so they not only uh they don't\num treat this f of x as a learnable\nfunction in its entirety but only it's\none part because it's a composite it's a\nsum of two functions and only one of\nthem is actually learnable and it's the\nB planine part now if you remember the\nformula for the B planine as we saw\nbefore it is each control Point\nmultiplied by its basis\nfunction uh the basis functions as I\ntold you before once you have chosen how\nmany control points you have and the\ndegree of the underlying basis um basic\ncurve you have a fixed uh basis\nfunctions because the basis functions\njust tells you what is the contribution\nof each control point to the final\ninterpolated curve so the only thing\nthat we need to learn in a b line is\nactually how to put the control points\nto interpolate any function that we want\nfor example imagine let's do a concrete\nexample imagine that our Network needs\nfor its purposes to reduce the loss to\nlearn senoidal function so our network\nwill learn to put the control points\nlike this using the same basis function\nfunctions another part of the network\nusing the same basis function so because\nthe degree didn't change it's fixed for\nall the uh basis functions that we want\nto learn for all the B plines that we\nwant to learn another part of the\nnetwork probably maybe needs to learn\nlet's say an exponential function so it\nwill learn using the same basis\nfunctions to put the control points like\nthis so it will create something that is\ngrowing oops very ugly so we go up then\nwe go up then we go up Up and Up Like\nThis maybe another part and as you can\nsee the basis functions didn't change\nthe only thing that is changing is the\nposition of how we put our control\npoints and we can get different shapes\nso this is the power of the bis lines so\nwe have these parameters that we have\ncan learn which are the positions of our\ncontrol points that can make our bis\nPline have any shape that we want if the\nnumber of control points is enough of\ncourse we we cannot create a senoidal\nfunction only have two points right\nbecause you can only draw a line uh so\nbut we will see later how can we\nincrease the number of control\npoints uh let's compare the parameters\ncount between the multi-layer perceptron\nand the kogo Arnold networks well\nimagine that you have a multi-layer\nmulti MLP so multi-layer percepton and\nit's made up of let's say l layers so\nyou have one layer after another like\nthis and each layer each\nlayer is taking as input n features and\nit's producing n features as\noutput as we saw before the layer of a\nlinear layer has a weight Matrix that is\ninput feature by output feature so in\nthis case the input feature and the\noutput feature are the same and they are\nequal to\nn so we will have a WID Matrix that is n\nthe^ of two for each of the layer here\nwe will also have n the^ of two weight\nMatrix here also Etc suppose that we\ndon't have any pasas and we have L\nlayers so the total parameter count of\nthis neural network is n to the^ of 2\nmultiplied by L in the case of the\nmulti-layer\nperceptron in the case of the colog of\narnal network we do not have weights but\nwe have learnable functions we have the\nsame number of learnable functions\nbecause once you have an input layer in\nthe cons that is taking for example we\nsaw before we have two features input\nand we have five features output so in\ntotal we have 10 functions to learn so\nit is uh it means that when we have n\ninput features and N output features we\nwill have n the power of two functions\nto learn for L layers means that we have\nn the power of two multiplied by L\nfunctions to learn but each functions\nalso has some parameters that we need to\nlearn which are the position of of these\ncontrol points so we need to learn this\nP of I term here in the expression of\nthis B\nplines and um it depends on the degree\nof the function uh because depends on\nwhat is the degree of the underlying B\ncurve that we are using and how many\nknots we have this they call it grid\nhere uh but they are the not so the\nmeeting point of this um Bas uh curves\nso so in the case of the cons we have\nmore parameters to learn because we have\nn the^ of 2 L functions to learn and\neach function has to learn five G Plus K\nparameters where G is the number of Nod\nand K is the degree of the B function\nunderlying B function uh curve another\ninteresting feature of colog networks is\nthat we can increase the number of\ncontrol points in each of these\nlearnable functions to give the\nlearnable function more expressive power\nto map any kind of function that it\nneeds for its purpose so imagine we are\ntraining a network and our Network to\nreduce its loss needs to uh learn a sign\nsenoidal function but we defined only\ntwo control points so it's not possible\nto learn a senoidal function having only\ntwo control points what we can do is\nthat instead of without changing the\nstructure of the network we can increase\nthe number of control points at each of\nthese functions so they can learn more\ncomplex\nmappings um this is called grid\nextension in the paper and it's\nsomething that we cannot do in the\nmulti-layer perceptron because imagine\nin the multi-layer perceptron you have a\nlinear layer uh a sequence of linear\nlayers one that takes as input let's say\nthree features and one that produces\nfive features and another one that takes\nas input five features and produces 10\nfeatures if you want to increase the\nexpressive power of this linear layer\nyou need to either increase the number\nof output\nfeatures which means basically you\nincrease the number of Neon\nuh but this will affect also the next\nlayer because if you increase it to six\nthen the next layer will also be\naffected uh this however does not happen\nwith the kog Aral networks because let\nme give you an example actually we can\nalso go\nbefore here we can increase um the power\nexpressive power of this function here\nor this function here or this function\nhere or all of them without changing the\nstructure of the network so they will\nstill produce five features as output\nbut this single function can learn more\ncomplex mappings because maybe we chose\ninitially only three points and the\nthree points are not enough to learn a\ncomplex function like the coine so we\ncan increase it to 10 points and the\nstructure of the network will be the\nsame and they also provide a a way to in\nbecause of course we increase the\nparameters and we need to find way to\ntransfer the parameters of the old\nNetwork into the new extended Network\nwithout changing the behavior of the\noverall functioning of the cogra network\nand basically they say that we can\ninitialize the new parameters so the\nadditional parameters using the old ones\nby making sure that the output of the\nnetwork at these functions is behave the\nsame for the same input so basically\nthey reduce the um you can see here the\nloss function here the minimizing the\ndistance between the old output of the\nfunction and the new output for the same\ninput so you can initialize this\nadditional parameters in this way and\nit's something that we can do because we\nhave learnable functions\nnow uh another interest interesting\nproperty is interpretability of these\nnetworks now what is interpretability\ninterpretability means that we want to\nbe able to understand how our Network\nthinks even if it's very wrong to say\nthinks but um how our networ Network\nmakes its prediction something that we\nas humans can interpret\nimagine we have a language model with\nbillions of parameters we will never be\nable to understand uh by analyzing each\nsingle parameter how it made it\nprediction because there are too many of\nthese parameters and to give an\ninterpretation to each of them it's\nhumanly\nimpossible however for simpler networks\nit is possible but the problem with\nneural networks is that because they do\nnot learn any functions they are just\nsum of weights with nonlinearities they\nstill make it very difficult to give an\ninterpretation even for small networks\nin this toy example in the paper they\nsay imagine we have some training data\nthat is coming from this\nexpression so we have some two input one\nis X and Y and then we have the\ncorresponding output according to this\nfunction here if you want to learn this\num train this network you don't know\nbeforeand how many uh nodes you need or\nhow many uh layers you need or how many\nlearn functions you need so what we do\nusually when we want to um train a\nnetwork like this we create a slightly\nbigger Network than we think we need and\nthen we sparsify it to reduce it to its\nbare minimum and this sparsification is\ndone with regularization when we work\nwith a multi-layer perceptron when we\nwant to spfy our Network we have a loss\nfunction right because we want our\noutput of our Network map with the match\nwith the training data so for example we\nuse the mean squared error and let's\ncall this one the prediction\nRoss then we add another term that is\nmultiplied by same coefficient called\nLambda that we can\nchoose that is um that multiplies it\nwith this is the L1 L1 regular\nregularizer which is the absolute value\nof the weights of this network this is\nin the case of the multilayer\nperceptron because if we use this loss\nhere this combined loss what will happen\nis that the network will make to have a\ncompromise between reducing the\nprediction loss and also reducing the\nweights of the network so it will force\nthe network to spfy it it means that not\nall neurons will be activated it will\nalso make it such that more a lot of\nWeights will be zero completely zero\nthey do a similar procedure in this case\nfor the AR networks in which they cannot\nuse the L1 regularizer because we do not\nhave learnable weights but we have\nlearnable functions so they find a way\nto create a regularizer using the\nactivation of this function so they use\nas regularizer what is the magnitude of\nthe activations of each of these\nfunctions so that our Network so they\ncreate a loss like this like the L1 L\nthe prediction loss plus the regularizer\nso let me the let's say Lambda\nregularizer\nso the network will have to make a\ncompromise between uh activating too\nmany functions and and reducing the loss\nso it will be forced to reduce the loss\nand to reduce the number of activations\nthat it of these functions that are\nactivated so minimizing the act the the\nmagnitude of these activations so once\nthey train the network with this loss\nthey see that a lot of functions are\nactually have a very little magnitude of\nactivation mean it means that the they\ncontribute very little to the output so\nfor example uh to this output they see\nthat or let's say this output or this\noutput or the final output they see that\nthis one contributes less than 1% for\nexample so they can just delete it this\nis called pruning that you delete the\nthe parts of the network that contribute\nvery little to the final\nprediction once they prune it they\nrealize that the network learned\nfunctions that are actually the same as\nthe initial expression from which they\nderived the training data so the network\nactually learns the structure of the\nexpression from which it received the\ntraining data so as you can see we have\nan input X we have an input y the input\nX is run through a function that is very\nsimilar to a sign it's not exactly a\nsign but it's very similar to a sign um\nthis is um Parabola so it's a quadratic\nfunction so the Y is run to a function\nthat is very similar to a quadratic\nfunction and then they are summed up\nbecause we saw before that at nodes here\nwe sum up the output of these two\nfunctions and then run to another\nfunction that is very similar to an\nexponential as you can see to create the\nfinal\noutput um so as you can see the network\nin order to learn a mapping between data\nthat has this expression actually\nrecreates the same structure of this\nexpression inside each each of these\nlearnable functions given that they you\ngive it enough uh expressive power so\nyou give enough control points to these\nlearned functions to actually be able to\napproximate these functions correctly so\nyou you need to give at least five or\nsix points in this uh function here to\nmake it look like a\nsign um of course this function is not\nreally a sign it could be a sign\ntranslated somehow or scaled somehow so\nthey do another procedure to learn the\nscaling factor of this sign and the\ntranslation Factor uh because it's not\nreally s of X it is probably uh uh a s\nof BX + C + D so they they they do\nanother procedure to learn this scaling\nFactor this translation factor and this\ntranslation factor and this scaling\nFactor but anyway the point is the\nnetwork was able to learn exactly the\nmapping of the training data using the\nsame functions and this is very\nimportant for interpretability because\nimagine in some Industries is actually\nuh kind of mandatory because imagine you\nare working in the healthcare sector and\nyou're creating a model that tells you\nhow much dose of a certain medicine you\nneed to give to the patient things go\nwrong and uh you cannot tell to the\njudge that you gave this dose because\nthe model told you so you need to give\nsome kind of uh verifiability some kind\nof uh interpretability of why you made\nthat decision or why the model is making\nthat decision and this\nis this can happen with colog networks\nand it's something that was very\ndifficult if not impossible to do with\nmulti-layer perceptron it's very\ndifficult with do with multi-layer\nperceptrons another very interesting\nfact is this continual learning and the\nand that the fact that we can avoid\ncatastrophic forgetting so let's take a\nstep back when we train a neural network\nsuppose a multi-layer percept\nRon it happens a lot of time that you\ntrain the suppose let's talk about the\nlanguage models so imagine you train a\nlanguage model on Japanese data and then\nyou want to train the language model\nalso to uh learn a little bit of let's\nsay uh English so you train it a little\nmore on English data what will happen is\nthat the language model will improve on\nthe English language tasks but we'll\nprobably forget the Japanese language uh\nthis is called this is a problem with\nmultilayer perceptrons and they they\nmade a toy example here to show that the\nkog Arnold networks react better to this\nproblem so what they did they created\nsome training data that is here so it's\na list of peaks of gausian so like this\nso this is all the training data but\nthey don't give all the training data at\nthe same moment to the networks so first\nthey only give the first part of the\ntraining data so only this point to the\nnetwork so to the con and to the\nmulti-layer perceptron and then they\nevaluate the network on the training\ndata and also outside of the training\ndata so they trained they they run the\nevaluation on this part of the um\ntraining data and they see that both\nperform quite well then they run the\nevaluation also on this part and then\nthey see that of course the both\nnetworks produce uh results that are not\ngood because they didn't learn the rest\nof the data yet then they feed the next\npart so they they take this Network that\nhas already been trained on the phase\none and train it on the next part of the\ninput which is this part\nhere and then they evaluate it again on\nall the range of the input of the input\ndata of the training data not only on\nthe part that it has been trained upon\nbut on all of it and they see that the\nmulti-layer perceptron do not behave\nvery well on the data they have seen in\nthe previous phase because they have\nalready forgotten it while they can\nstill remembers it so still performs\nwell on the part that was in the\nprevious training phase and they do it\nfor the third phase and see that they\ncan actually learn also the third phase\nbut without forgetting the previous two\nwhile the multi-layer perceptron is only\nperforming well on the third phase while\nit has already forgotten how to perform\nwell on the first two and do we do it\nfor all the phases of this training data\nand this is amazing actually and\nactually we also know kind of why it\nhappens and it's related to the local\ncontrol property that we saw of the be\nlines before so let me open the example\nagain U as you saw before if we move a\npoint on a b line it will affect only\nthe local um the local area of the curve\nnot the entire curve so what is\nhappening here when we train the suppose\ninitially all the control points are\nhere uh so the network because if I\nremember correctly in the paper they say\nthat they initialize the function so\nthat they always predict zero for the\ninput so suppose that the uh network is\npredicting always zero like this when\nthey feed the first part of the training\ndata the network learns this\nmapping then they feed the second part\nof the training data then the network\nlearned this mapping here so as you can\nsee the network learns to move only some\ncontrol points without affecting the\nother control points so it retains the\ninformation it has learned in the past\nand this is thanks to the local control\npoint of the Beast\nlines thank you guys for watching my\nvideo I I hope that you learned a lot\nalso today I it was quite a demanding\nvideo I have to say I didn't have much\ntime to prepare it also uh especially\nbecause the the the paper came out like\n10 days ago and I had to kind of um you\nknow review a lot of topics that I had\nforgotten and plus I'm working full-time\nso it's it always takes a lot of effort\nto make this this kind of content but I\nstill I believe that um I hope that you\nthat you learned that you got a lot of\nKnowledge from this video at least you\nknow how to make Bas curves now and uh\nif you like this content please share it\nwith your friends share it on social\nmedia this is the best way to help me\nand to motivate me to make a more\nquality content like this one if there\nare some inaccuracies please forgive me\nI I try to make it as precise as\npossible but I also try to simplify\nConcepts so sometimes I have to use\ninaccurate wording or inaccurate\nphrases um I hope to make more videos\nlike this in the future so please follow\nmy channel subscribe and like the video\nif you like it and thank you for being\nin the audience",
  "transcript_chars": 65892,
  "ingested_at": "2026-05-15T16:37:58.336817+00:00",
  "source": "channel",
  "yt_meta": {
    "view_count": 39376,
    "like_count": 1745,
    "channel_id": "UCtAcpQcYerN8xxZJYTfWBMw",
    "categories": [
      "Science & Technology"
    ],
    "tags": [
      "pytorch",
      "python",
      "tutorial",
      "math",
      "language models",
      "deep learning",
      "machine learning",
      "multi layer perceptron",
      "mlp",
      "kolmogorov-arnold networks",
      "kolmogorov-arnold representation theorem",
      "universal approximation theorem",
      "neural networks",
      "bezier curves",
      "splines",
      "b-splines",
      "linear layers"
    ]
  }
}