This is the most step-by-step spelled-out explanation of backpropagation and training of neural networks. It only assumes basic knowledge of Python and a vague recollection of calculus from high school.

Links:
- micrograd on github: https://github.com/karpathy/micrograd
- jupyter notebooks I built in this video: https://github.com/karpathy/nn-zero-to-hero/tree/master/lectures/micrograd
- my website: https://karpathy.ai
- my twitter: https://twitter.com/karpathy
- "discussion forum": nvm, use youtube comments below for now :)
- (new) Neural Networks: Zero to Hero series Discord channel: https://discord.gg/Hp2m3kheJn , for people who'd like to chat more and go beyond youtube comments

Exercises:
you should now be able to complete the following google collab, good luck!:
https://colab.research.google.com/drive/1FPTx1RXtBfc4MaTkf7viZZD4U2F9gtKN?usp=sharing

Chapters:
00:00:00 intro
00:00:25 micrograd overview
00:08:08 derivative of a simple function with one input
00:14:12 derivative of a function with multiple inputs
00:19:09 starting the core Value object of micrograd and its visualization
00:32:10 manual backpropagation example #1: simple expression
00:51:10 preview of a single optimization step
00:52:52 manual backpropagation example #2: a neuron
01:09:02 implementing the backward function for each operation
01:17:32 implementing the backward function for a whole expression graph
01:22:28 fixing a backprop bug when one node is used multiple times
01:27:05 breaking up a tanh, exercising with more operations
01:39:31 doing the same thing but in PyTorch: comparison
01:43:55 building out a neural net library (multi-layer perceptron) in micrograd
01:51:04 creating a tiny dataset, writing the loss function
01:57:56 collecting all of the parameters of the neural net
02:01:12 doing gradient descent optimization manually, training the network
02:14:03 summary of what we learned, how to go towards modern neural nets
02:16:46 walkthrough of the full code of micrograd on github
02:21:10 real stuff: diving into PyTorch, finding their backward pass for tanh
02:24:39 conclusion
02:25:20 outtakes :)
 Hello, my name is Andre and I've been training deep neural networks for a bit more than a decade and in this lecture I'd like to show you what neural network training looks like under the hood So in particular we are going to start with a blank super notebook and by the end of this lecture We will define and train in neural net and you'll get to see everything that goes on under the hood and exactly Sort of how that works and then to it in a little Now specifically what I would like to do is I would like to take you through Building of micrograd now micrograd is this library that I released on GitHub about two years ago But at the time I only uploaded this source code and you'd have to go in by yourself and really Figure out how it works So in this lecture I will take you through it step-by-step and kind of comment on all the pieces of it So what's micrograd and why is it interesting? cute micrograd is basically an auto-grad engine Auto-grad is short for automatic gradient and really what it does is it implements back propagation Now back propagation is this algorithm that allows you to efficiently evaluate the gradient of Some kind of a loss function with respect to the weights of a neural network And what that allows us to do then is we can editorively tune the weights of that neural network to minimize the loss function And therefore improve the accuracy of the network So back propagation would be at the mathematical core of any modern deep neural network library like say PyTorch or Jax So the functionality of micrograd is I think best illustrated by an example. So if we just scroll down here You'll see that micrograd basically allows you to build out mathematical expressions and Here what we are doing is we have an expression that we're building out where you have two inputs a and b And you'll see that a and b are negative 4 and 2 but we are wrapping those Values into this value object that we are going to build out as part of micrograd So this value object will wrap the numbers themselves and then we are going to build out a mathematical expression here where a and b are Transformed into cd and eventually e f and g And I'm showing some of the function some of the functionality of micrograd and the operations that it supports So you can add two value objects. You can multiply them. You can raise them to a constant power You can also by one the gate squash at zero Square divide by constant divide by it etc And so we're building out an expression graph with with these two inputs a and b and we're creating out the value of g and micrograd will in the background Build out this entire mathematical expression So it will for example know that c is also a value c was a result of an addition operation and the Child nodes of c are a and b because the and all maintain pointers to a and b value objects So we'll basically know exactly how all of this is laid out and Then not only can we do what we call the forward pass where we actually look at the value of g Of course, that's pretty straightforward We will access that using the dot data attribute And so the output of the forward pass the value of g is 24.7 it turns out But the big deal is that we can also take this g value object and we can call dot backward And this will basically initialize back propagation at the node g And what back propagation is going to do is it's going to start at g and it's going to go backwards through that expression graph and it's going to recursively apply the chain rule from calculus and And what that allows us to do then is we're going to evaluate basically the derivative of g with respect to all the internal nodes Like ed and c, but also with respect to the inputs a and b And then we can actually query this derivative of g with respect to a for example That's a dot grad in this case it happens to be 138 and a derivative of g with respect to b Which also happens to be here 645 And this derivative we'll see soon is very important information because it's telling us how a and b are affecting g Through this mathematical expression. So in particular a dot grad is 138 So if we slightly nudge a and make it slightly larger 138 is telling us that g will grow in the slope of that growth is going to be 138 And the slope of growth of b is going to be 645 So that's going to tell us about how g will respond if a and b get tweaked a tiny amount in a positive direction Okay Now you might be confused about what this expression is that we built out here and this expression by the way is completely meaningless I just made it up. I'm just flexing about the kinds of operations that are supported by micrograd What we actually really care about are neural networks But it turns out that neural networks are just mathematical expressions just like this one But actually a slightly bit less crazy even um Neural networks are just a mathematical expression They take the input data as an input and they take the weights of a neural network as an input and some mathematical expression And the output are your predictions of your neural net or the loss function. We'll see this in a bit But basically neural networks teaches us happen to be a certain class of mathematical expressions But back propagation is actually significantly more general It doesn't actually care about neural networks at all. It only tells us about arbitrary mathematical expressions And then we happen to use that machinery for training of neural networks Now one more note I would like to make at the stage is that as you see here micrograd is a scalar valued auto-grad engine So it's working on the you know level of individual scalers like negative four and two And we're taking neural nets and we're breaking them down all the way to these atoms of individual scalers And all the little pluses and times and it's just excessive And so obviously you would never be doing any of this in production It's really just for them for pedagogical reasons because it allows us to not have to deal with these and dimensional tensors that you would use in modern deep neural network library So this is really uh done so that you understand and refactor out back propagation and chain rule and understanding of your training And then if you actually want to train bigger networks You have to be using these tensors, but none of the math changes. This is done purely for efficiency We are basically taking scale value All the scale values we're packaging them up into tensors Which are just arrays of these scalers and then because we have these large arrays We're making operations on those large arrays that allows us to take advantage of the parallelism in a computer and All those operations can be done in parallel and then the whole thing runs faster But really none of the math changes and that done purely for efficiency So I don't think that it's pedagogically useful to be dealing with tensors from scratch Uh, and I think and that's why I fundamentally wrote micrograd because you can understand how things work Uh, at the fundamental level and then you can speed it up later Okay, so here's the fun part my claim is that micrograd is what you need to train your networks and everything else It's just efficiency So you'd think that micrograd would be a very complex piece of code and that turns out to not be the case So if we just go to micrograd and you will see that there's only two files here in micrograd This is the actual engine. It doesn't know anything about neural nets And this is the entire neural nets library on top of micrograd. So engine and nn.pi so the actual back propagation quadrgrad engine That gives you the power of neural networks is literally 100 lines of code of like very simple python Which we'll understand by the end of this lecture and then nn.pi This neural network library built on top of the autograd engine Um, is like a joke. It's like We have to define what is in neuron and then we have to define what is the layer of neurons and then we define what is a multilateral perceptron Which is just a sequence of layers of neurons and so it's just a total joke So basically um, there's a lot of power that comes from only 115 lines of code and that's only need to understand to understand You know, or training and everything else is just efficiency and of course there's a lot too efficiency But fundamentally that's all that's happening Okay, so now let's dive right in and implement micrograd step by step the first thing I'd like to do is I'd like to make sure that you have a very good understanding Intuitively of what a derivative is and exactly what information it gives you So let's start with some basic imports that I copy-based in every Jupyter notebook always And let's define the function scalar valid function f of x as follows So I just make this up randomly I just want to scale a valid function that takes a single scalar x and returns a single scalar y And we can call this function of course so we can pass it say 3.0 and get 20 back Now we can also plot this function to get a sense of its shape You can tell from the mathematical expression that this is probably a parabola it's quadratic And so if we just create a set of um um Skip skill the values that we can feed in using for example a range from negative 5 to 5 and steps up 0.25 So this is so x is just from negative 5 to 5 not including 5 in steps of 0.25 And we can actually call this function on this non-py array as well So we get a set of y's if we call f on x's and These y's are basically also applying function on every one of these elements independently and we can plot this using math plotlib So plt.plot x's and y's and we get nice parabola So previously here we fed in 3.0 somewhere here and we received 20 back which is here the y coordinate So now I'd like to think through what is the derivative of this function at any single input point x Right, so what is the derivative at different points x of this function Now if you remember back to your calculus class you've probably derived derivatives So we take this mathematical expression 3x square minus 4x plus 5 and you would write out on a piece of paper And you would you know apply the product rule and all the other rules and derive the mathematical expression of the great derivative of the original function And then you could plug in different taxes and see what the derivative is We're not going to actually do that because no one in neural networks actually writes out the expression for the neural net It would be a massive expression um it would be you know thousands since thousands of terms no one actually Derives that derivative of course And so we're not going to take this kind of like symbolic approach Instead what I'd like to do is I'd like to look at the Definition of derivative and just make sure that we really understand what derivative is measuring What is telling you about the function? And so if we just look up derivative We see that um Okay, so this is not a very good definition of derivative. This is a definition of what it means to be differentiable But if you remember from your calculus it is the limit sh goes to 0 of f of x plus h minus f of x over h So basically what it's saying is if you slightly bump up You're at some point x that you're interested in or hey and if you slightly bump up You know you slightly increase it by small number h How does the function respond with what sensitivity does it respond? Where's the slope at that point does the function go up or does it go down and by how much and that's the slope of that function The the slope of that response at that point and so we can basically evaluate The derivative here numerically by taking a very small h of course the definition would ask us to take h to zero We're just going to pick a very small h 0.001 And let's say we're interested in 0.3.0 So we can look at f of x of course is 20 And now f of x plus h So if we slightly nudge x in a positive direction how is the function going to respond? And just looking at this do you expect you expect f of x plus h to be slightly greater than 20 or do you expect to be slightly lower than 20? And so since 3 is here and this is 20 if we slightly go positively the function will respond positively So you'd expect this to be slightly greater than 20 And now by how much is telling you the sort of the strength of that slope right the size of the slope So f of x plus h of f of x this is how much the function responded in the positive direction and we have to normalize by the run so we have the rise over run to get the slope So this of course is just numerical approximation of the slope because we have to make a very very small to converge to the exact amount Now if I'm doing too many zeros At some point I'm going to get an incorrect answer because we're using floating point arithmetic And the representations of all these numbers in computer memory is finite and at some point we get into trouble So we can converge towards the right answer with this approach But basically at 3 the slope is 14 And you can see that by taking 3x square minus 4x plus 5 and differentiating it in our head So 3x square would be 6x minus 4 And then we plug in x equals 3 so that's 18 minus 4 is 14 so this is correct So that's at 3 now how about the slope at say negative 3 Would you expect would you expect for the slope now telling the exact value is really hard But what is the sign of that slope? So at negative 3 If we slightly go in the positive direction at x the function would actually go down And so that tells you that the slope would be negative So we'll get a slight number below Below 20 and so if we take the slope we expect something negative negative 22 Okay And at some point here of course the slope would be 0 Now for this specific function I looked it up previously and it's at point 2 over 3 So at roughly 2 over 3 But somewhere here This derivative would be 0 So basically at that precise point Yeah At that precise point if we nudge in a positive direction the function doesn't respond This stays the same almost and so that's why the slope is 0 Okay now let's look at a bit more complex case So we're going to start you know complexifying a bit So now we have a function Here With output variable d That is a function of 3 scalar inputs a b and c So a b and c are some specific values 3 inputs into our expression graph And a single output d And so if we just print d we get 4 And now what I like to do is I'd like to again look at the derivative of d with respect to a b and c And think through Again just the intuition of what this derivative is telling us So in rooted evaluates derivative We're going to get a bit hacky here. We're going to again have a very small value of h And then we're gonna fix the inputs at some values that we're interested in So these are the this is the point a bc at which we're going to be evaluating the derivative of d with respect to all a b and c at that point So there's the inputs and now we have d1 is that expression And then we're going to for example look at the derivative of d with respect to a So we'll take a and we'll bump it by h and then we'll get d2 to be the exact same function And now we're going to print You know fun I want d1 is d1 d2 is d2 and print slope So the derivative or slope here will be of course d2 minus d1 divided h So d2 minus d1 is how much the function increased when we bumped the The specific input that we're interested in by a tiny amount and This is the normalized by h to get the slope So Yeah, so this so I just run this we're going to print d1 Which we know is for Now d2 will be bumped a will be bumped by h So let's just think through a little bit What d2 will be printed out here in particular d1 will be for Will d2 be a number slightly greater than 4 or slightly lower than 4 and it's going to tell us the the sign of the derivative So We're bumping a by h b is minus 3 c is 10 So you can just in total think through this derivative and what is doing a will be slightly more positive And but b is a negative number So if a is slightly more positive Because b is negative 3 We're actually going to be adding less to d So you'd actually expect that the value of the function will go down So let's just see this Yeah, and so we went from 4 to 3.9996 And that tells you that the slope will be negative and then Uh will be a negative number because we went down and then the exact number of slope will be exact amount of slope is negative 3 And you can also convince yourself that negative 3 is the right answer mathematically and analytically Because if you have 8 times b plus c and you are you know you have calculus Then differentiating 8 times b plus c with respect to a gives you just b And indeed the value of b is negative 3 which is the derivative that we have so you can tell that that's correct So now if we do this with b So if we bump b by a little bit in a positive direction We'd get different slopes. So what is the influence of b on the output d So if we bump b by tiny amount in the positive direction then because a is positive We'll be adding more to d Right So um, and now what is the what is the sensitivity? What is the slope of that addition? And it might not surprise you that this should be 2 And why is it 2 because d of d by db The differentiating respect to b would be would give us a and the value of a is 2 So that's also working all And then if c gets bumped a tiny amount in h by h Then of course 8 times b is unaffected and now c becomes slightly bit higher What does that do to the function? It makes it slightly bit higher because we're simply adding c And it makes it slightly bit higher by the exact same amount that we added to c And so that tells you that the slope is 1 That will be the um The rate at which d will increase as we scale c Okay, so we now have some intuitive sense of what this derivative is telling you about the function And we'd like to move to neural networks Now as I mentioned neural networks will be pretty massive expressions mathematical expressions So we need some data structures that maintain these expressions and that's what we're going to start to build out now So we're going to Build out this value object that I showed you in the read me page of micrograd So let me copy paste a skeleton of the first very simple value object So class value takes a single scalar value that it wraps and keeps track of and that's it So we can for example do value of 2.0 and then we can Get we can look at its content and um python will internally use the wrapper function to return uh this straight clips like that So this is a value object with data equals to the overcreating here Now what we'd like to do is like we'd like to be able to Have not just like two values But we'd like to do a blocky right we'd like to add them So currently you would get an error because python doesn't know how to add Two value objects. So we have to tell it So here's an addition So you have to basically use these special double underscore methods in python to define these operators for these objects so if we call um the If we use this plus operator python will internally call a dot Add of b that's what will happen internally and so b will be the other And self will be a And so we see that what we're going to return is a new value object And it's just uh is it going to be wrapping the plus of their data But remember now because uh data is the actual like numbered python number So this operator here is just the typical floating point plus addition now It's not an addition of value objects and will return a new value So now a plus b should work and it should print value of Negative one Because that's two plus minus three There we go Okay, let's now implement multiply Just so we can recreate this expression here So multiply I think it won't surprise you will be fairly similar So instead of add we're going to be using mul And then here of course we want to do times And so now we can create a C value object which will be 10.0 And now we should be able to do a times b Well, let's just do a times b first um That's value of negative six now And by the way I skipped over this a little bit uh suppose that I didn't have the wrapper function here Then it's just that you'll get some kind of an ugly expression So what wrapper is doing is it's providing us a way to print out like a nicer looking expression in python Uh, so we don't just have something cryptic. We actually are you know, it's value of negative six So this gives us a times and then this we should now be able to add C to it because we've defined and told the python how to do mul and add And so this will call this will basically be equivalent to a dot mul B and then this new value object will be dot add of C And so let's see if that worked Yep, so that worked well that gave us four which is what we expect from before And I believe you can just call the manually as well. There we go. So Yeah, okay, so now what we are missing is the connected tissue of this expression As I mentioned we want to keep these expression graphs So we need to know and keep pointers about what values produce what other values So here for example, we are going to introduce a new variable which will call children And by default it will be an empty tuple And then we're actually going to keep a slightly different variable in the class which will call underscore prime Which will be the set of children Uh, this is how I done. I did it in the original micro grad looking at my code here I can't remember exactly the reason I believe it was efficiency But this underscore children will be a tuple for convenience But then when we actually maintain it in the class it will be just this set Yes, I believe for efficiency um So now when we are creating a value like this with a constructor children will be empty and prep will be the empty set But when we're creating a value through addition or multiplication We're going to feed in the children of this value which in this case is self and other So those are the children here So now we can do d dot prep and we'll see that the children of the We now know are this a value of negative six and value of ten and this of course is the value resulting from a times b And the c value which is ten Now the last piece of information we don't know so we know now that the children of every single value We don't know what operation created this value So we need one more element here. Let's call it underscore pop And by default this is the empty set for leaves And then we'll just maintain it here And now the operation will be just a simple string and in the case of addition it's plus in the case of multiplication it's times So now we Not just have d dot prep we also have a d dot up And we know that d was produced by an addition of those two values And so now we have the full mathematical expression And we're building out this data structure and we know exactly how each value came to be By word expression and from what other values Now because these expressions are about to get quite a bit larger We'd like a way to nicely visualize these expressions that we're building out So for that I'm going to copy paste a bunch of slightly scary code That's going to visualize this this expression graphs for us So here's the code and I'll explain it in a bit But first let me just show you what this code does Basically what it does is it creates a new function draw dot that we can call on some root node And then it's going to visualize it So if we call draw dot on d which is this final value here that is a times b plus c It creates something like this. So this is d and you see that this is a times b Create a value plus c gives us this output node d So that's draw dot of d and I'm not going to go through this in complete detail You can take a look at graph is and it's api A graph is an open source graph visualization software And what we're doing here is we're building out this graph in the graph is api And you can basically see that trace is this helper function that enumerates all the nodes and edges in the graph So that's just built a set of all the nodes and edges And then we iterate through all the nodes and we create special node objects for them in using dot node And then we also create edges using dot dot edge And the only thing that's like slightly tricky here is you notice that I basically add these fake nodes Which are these operation nodes. So for example this node here is just like a plus node and I create these These special op nodes here and I connect them accordingly So these nodes of course are not actual nodes in the original graph They're not actually a value object. The only value objects here are the things in squares Those are actual value objects or representations thereof And these op nodes are just created in this draw dot routine so that it looks nice Let's also add labels to these graphs just so we know what variables are where so let's create a special underscore label Um, or let's just do label equals ft by default and save it to each node And then here we're going to do label is a label is the label is c um And then Let's create a special um E equals a times b And dot label will be It's kind of notty And E will be E plus C And a d dot label will be B Okay, so nothing really changes. I just added this new E function A new E variable And then here when we are printing this I'm going to print the label here. So this will be a percent s bar and this will be end dot label And so now We have the label on the left here. So it says a b creating e and then E plus C creates d Just like we have it here And finally, let's make this expression just one layer deeper So d will not be the final output node Instead after d we are going to create a new value object Called f we're going to start running out of variable soon f will be negative 2.0 And it's label will of course just the f And then l will capital L will be the output of our graph And l will be p times f Okay, so l will be negative eight is the output Uh, so Now we don't just draw a d draw L Okay And somehow the label of L is undefined loops. I'll that label as to be explicitly So given to it There we go. So l is the output So let's quickly recap what we've done so far We are able to build out mathematical expressions using only plus and times so far Uh, they are scalar valued along the way and we can do this forward pass And build out a mathematical expression So we have multiple inputs here a b c and f Going into a mathematical expression that produces a single output L And this here is vis-visualizing the forward pass So the output of the forward pass is negative eight. That's the value Now what we'd like to do next is we'd like to run back propagation And in back propagation we are going to start here at the end And we're going to reverse And calculate the gradient along along all these intermediate values And really what we're computing for every single value here Um, we're going to compute the derivative of that node with respect to L So the derivative of l with respect to l is just uh one And then we're going to derive what is the derivative of l with respect to f with respect to d With respect to c with respect to e With respect to b and with respect to a And in neural network setting It'd be very interested in the derivative of basically this loss function l With respect to the weights of a neural network And here of course we have just these variables a b c and f But some of these will eventually represent the weights of a neural net And so we'll need to know how those weights are impacting The loss function So we'll be interested basically in the derivative of the output With respect to some of its leaf nodes And those leaf nodes will be the weights of the neural net And the other leaf nodes of course will be the data itself But usually we will not want or use the derivative of the loss function with respect to data Because the data is fixed But the weights will be iterated on Using the gradient information So next we are going to create a variable inside the value class That maintains the derivative of l with respect to that value And we will call this variable grad So there is a dot data and there's a self-adgrad And initially it will be zero And remember that zero is basically means no effect So at initialization we're assuming that every value does not impact Does not affect the output Right because if the gradient is zero That means that changing this variable is not changing the loss function So by default we assume that the gradient is zero And then now that we have grad and it's 0.0 We are going to be able to visualize it here after data So here grad is 0.4f And this will be in that grad And now we are going to be showing both the data and the grad And initialize that zero And we are just about getting ready to calculate the back propagation And of course this grad again as I mentioned Is representing the derivative of the output In this case l with respect to this value So with respect to So this is the derivative of l with respect to f Respect to d and so on So let's now fill in those gradients And actually do back propagation manually So let's start filling in these gradients And start all the way at the end as I mentioned here First we are interested to fill in this gradient here So what is the derivative of l with respect to l In other words if I change l by a tiny amount of h How much does l change? It changes by h So it's proportional and therefore the derivative will be 1 We can of course measure these or estimate these numerical gradients Numerically just like we've seen before So if I take this expression And I create a def lol function here And put this here Now the reason I'm creating a gating function lol here Is because I don't want to pollute or mess up the global scope here This is just kind of like a little staging area And as you know in Python all of these will be local variables to this function So I'm not changing any of the global scope here So here l1 will be l And then copy based on this expression We're going to add a small amount h In for example a Right and this will be measuring the derivative of l with respect to a So here this will be l2 And then we want to print that derivative So print l2 minus l1 Which is how much l changed And then normalize it by h So this is the rise over run And we have to be careful because l is a valid node So we actually want its data Um So that these are floats dividing by h And this should print the derivative of l with respect to a Because a is the one that we bumped a little bit by h So what is the derivative of l with respect to a It's 6 Okay And obviously If we change l by h Then that would be Here effectively Um This looks really awkward but changing l by h You see the derivative here is one Um That's kind of like the base case of what we are doing here So basically we come out comp here And we can manually set l dot grad to one This is our manual back propagation l dot grad is one And let's redraw And we'll see that we filled in Grad is one for l We're now going to continue the back propagation So let's here look at the derivatives of l with respect to d and f Uh let's do a d first So what we are interested in if I create a mark down on here Is we'd like to know Basically we have that l is d times f And we'd like to know what is uh dl by dd What is that? And if you know you're a calculus uh l is d times f So what is dl by dd? It would be f And if you don't believe me We can also just derive it because the proof would be fairly straightforward Uh we go to the definition of the A derivative Which is f of x plus h minus f of x divide h As a limit Limit of h goes to zero of this kind of expression So when we have l is d times f Then increasing d by h would give us the output of d plus h times f That's basically a full of x plus h, right minus d times f And then divide h And symbolically expanding out here We would have basically d times f plus h times f minus d times f divide h And then you see how the df minus df cancels So you're left with h times f divide h Which is f So in the limit as h goes to zero of You know derivative um definition We just get f in a case of d times f So symmetrically dl by d f will just be d So what we have is that f dot grad We see now is just the value of d Which is four And we see that d dot grad is just uh the value of f And so the value of f is negative two So we'll set those manually Let me erase this markdown node and then let's redraw what we have Okay, and let's just make sure that these were correct So we seem to think that dl by dd is negative two. So let's double check Um, let me erase this plus h from before and now we want the derivative with respect to f So let's just come here when I create f and let's do a plus h here And they should print a derivative of l with respect to f. So we expect to see four Yeah, and this is four up to floating point funquiness And then dl by dd should be f Which is negative two grad is negative two So if we again come here and we change d d dot d dot plus equals h right here So we expect so we've added a little h and then we see how l changed and we expect to print Uh negative two There we go So we've numerically verified what we're doing here is what kind of like an inline gradient check gradient check is when we are deriving this like back propagation and getting the derivative with respect to all the intermediate results and then numerical gradient is just you know Um, estimating it using small step size Now we're going to the crux of back propagation So this will be the most important node to understand because if you understand the gradient for this node You understand all of back propagation and all of training on neural nets basically So we need to derive dl by dc in other words the derivative of l with respect to c Because we've computed all these other gradients already Now we're coming here and we're continuing the back propagation manually So we want dl by dc and then we'll also derive dl by dE Now here's the problem How do we derive dl by dc? We actually know the derivative l with respect to d so we know how l is sensitive to d But how is l sensitive to c? So if we wiggle c how does that impact l through d? So we know dl by dc And we also here know how c impacts d and so just very intuitively if you know the impact that c is having on d And the impact that d is having on l then you should be able to somehow put that information together to figure out how c impacts l And indeed this is what we can actually do So in particular we know just concentrating on d first Let's look at how what is the derivative basically of d with respect to c? So in other words what is dd by dc So here we know that d is c times c plus ee That's what we know and our interesting dd by dc If you would just know your calculus again and you remember then differentiating c plus e with respect to c You know that that gives you 1.0 And we can also go back to the basics and derive this because again we can go to our f of x plus h minus f of x derogh divide by h That's the definition of a derivative as h goes to 0 And so here Focusing on c and its effect on d we can basically do the f of x plus h will be c is incremented by h plus e That's the first evaluation of our function minus c plus e And then divide h and so what is this? Just expanding the sound this will be c plus h plus e minus c minus e And then you see here how c minus c cancels e minus e cancels were left with h over h which is 1.0 And so by symmetry also dd by dd will be 1.0 as well So basically the derivative of a some expression is very simple and this is the local derivative So I call this the local derivative because we have the final output value all the way at the end of this graph And we're now like a small node here and this is a little plus node And it the little plus node doesn't know anything about the rest of the graph that it's embedded in All it knows is that it did it plus it took a c and a e added them and created a d And this plus node also knows the local influence of c on d Or rather were rather the derivative of d with respect to c And it also knows the the derivative of d with respect to e But that's not what we want. That's just a local derivative What we actually want is dl by dc and l could l is here just one step away But in a general case this little plus node is could be embedded in like a massive graph So Again, we know how l impacts d and now we know how c and e impact d How do we put that information together to write dl by dc and the answer of course is the chain rule in calculus And so I pulled up a chain rule here from capidia And I'm going to go through this very briefly. So chain rule We capidia sometimes can be very confusing and calculus can can be very confusing like this is the way I learned Chain rule and was very confusing like what is happening? It's just complicated. So I like this expression much better If a variable z depends on a variable y which itself depends on a variable x Then z depends on x as well obviously through the intermediate variable y And in this case the chain rule is expressed as if you want dz by dx Then you take the dz by dy and you multiply it by dy by dx So the chain rule fundamentally is telling you how We chain these Uh derivatives together correctly So to differentiate through a function composition We have to apply a multiplication of those derivatives So that's really what chain rule is telling us and there's a nice little intuitive explanation here Which I also think is kind of cute The chain rule says that knowing the instantaneous rate of change of z with respect to y and y relative to x allows one to calculate the instantaneous rate of change of z relative to x As a product of those two rates of change simply the product of those two So here's a good one If a car travels twice as fast as a bicycle and the bicycle is four times as fast as a walking man Then the car travels two times four eight times as fast as a man And so this makes it very clear that the correct thing to do sort of is to multiply So cars twice as fast as a bicycle and bicycle is four times as fast as man So the car will be eight times as fast as the man And so we can take these intermediate rates of change if you will and Multiply them together and that justifies the Chain rule intuitively. So I have a look at chain rule about here. Really what it means for us is there's a very simple recipe for deriving what we want Which is dfl by dc And what we have so far is we know one And we know What is the impact of d on l so we know dl by dd the derivative of l respect to dd We know that that's negative two and now because of this local Reason that we've done here we know dd by dc So how does c impact d and in particular this is a plus node So the local derivative is simply 1.0 is very simple And so the chain rule tells us that dl by dc going through this intermediate variable We'll just be simply dl by dd times dd by dc That's chain rule So this is identical to what's happening here except Z is rl y is our d and x is our c So we literally just have to multiply these and because These local derivatives like dd by dc are just one We basically just copy over dl by dd because this is just times one So what is it did so because dl by dd is negative two what is dl by dc? Well, it's the local gradient 1.0 times dl by dd which is negative two So literally what a plus node does you can look at it that way is it literally just routes the gradient Because the plus nodes local derivatives are just one and so in the chain rule one times dl by dd is Is just dl by dd and so that derivative just gets routed to both c and to e in the skates So basically We have that e dot grad Or what's our good c since that's the one we built that is negative two times one negative two And in the same way by symmetry e dot grad will be negative two that's the claim So we can set those We can redraw And you see how we just assign negative two negative two So there's back propagating signal which is carrying the information of like what is the derivative of l with respect to all the intermediate nodes We can imagine it almost like flowing backwards through the graph and a plus node will simply distribute the derivative to all the leaf nodes Assuring to all the children nodes of it So this is the claim and now let's verify it So let me remove the plus h here from before And now instead what we're going to do is we want to increment c so c dot data will be incremented by h And when I run this we expect to see negative two Negative two and then of course for e So e dot data plus equals h and we expect to see negative two Simple So those are the derivatives of these internal nodes And now we're going to recurse our way backwards again And we're again going to apply the chain rule So here we go our second application of chain rule and we will apply it all the way through the graph Which just happened to only have one more node remaining We have that d l by d e As we have just calculated is negative two so we know that So we know the derivative of l with respect to e And now we want d l by d a Right and the chain rule is telling us that that's just d l by d e Negative two times the local gradient So what is the local gradient basically d e by d a We have to look at that So I'm a little times node Inside a massive graph and I only know that I did a times b and I produced an e So now what is d e by d a and d e by d b that's the only thing that I sort of know about that's my local gradient So because we have that e is a times b We're asking what is d e by d a And of course we just did that here we have a times so I'm not going to redrive it But if you want to differentiate this with respect to a you'll just get b right the value of b Which in this case is negative 3.0 So basically we have that d l by d a Well, let me just do it right here We have that a dot grad and we are applying chain rule here Is d l by d e which we see here is negative two times what is d e by d a It's the value of b which is negative three That's it And then we have b dot grad is again d l by d e which is negative two Just the same way times What is d e by d um d b is the value of a which is 2.0 That's the value of a So these are our claimed derivatives Let's read draw and we see here that a dot grad turns out to be six because that is negative two times negative three And b dot grad is negative four times sorry is negative two times two which is negative four So those are our claims let's delete this and let's verify them We have a here a dot data plus equals h So the claim is that a dot grad is six. Let's verify six and we have b dot data plus equals h So nudging b by h and looking at what happens We claim it's negative four And indeed it's negative four plus minus again float oddness um And uh That's it. This that was the manual backpropagation All the way from here to all the leaf notes and I've done it piece by piece And really all we've done is as you saw we iterated through all the nodes one by one And locally applied the chain rule We always know what is the derivative of l with respect to this little output And then we look at how this output was produced This output was produced through some operation And we have the pointers to the children nodes of this operation And so in this little operation we know what the local derivatives are And we just multiply them onto the derivative always So we just go through and recursively multiply on the local derivatives And that's what backpropagation is is just a recursive application of chain rule Backwards through the computation graph Let's see this power in action just very briefly What we're good to do is we're going to Uh, nudge our inputs to try to make l go up So in particular what we're doing is we want a dot data. We're going to change it And if we want l to go up that means we just have to go in the direction of the gradient So a Should increase in the direction of gradient by like some small step amount. This is the step size And we don't just want this for b, but also for b Also for c Also for f Those are leaf nodes which we usually have control over And if we nudge in direction of the gradient we expect a positive influence on l So we expect l to go up positively Uh, so it should become less negative it should go up to say negative, you know, six or something like that Uh, it's hard to tell exactly and we have to reroute the forward path. So let me just um Do that here um This would be the forward pass f would be unchanged. This is effectively the forward pass And now if we print l dot data We expect because we nudge all the values or the inputs in the rational gradient We expect it less negative l. We expect it to go up So maybe it's negative six or so. Let's see what happens Okay negative seven And uh, this is basically one step of an optimization that will end up running And really this gradient just give us some power because we know how to influence the final outcome And this will be extremely useful for training. You know, that's as we'll see So now I would like to do one more uh example of manual back propagation using a bit more complex and uh useful example We are going to back propagate through a neuron so We want to eventually build out neuron that works in an as simplest cases are multi-layer perceptrons as they're called So this is a two layer neuron that And it's got these hidden layers made up of neurons and these neurons are fully connected to each other Now biologically neurons are very complicated devices But we have very simple mathematical models of them And so this is a very simple mathematical model of a neuron. You have some inputs xs And then you have these synapses that have weights on them So um, the w's are weights Um, and then the synapse interacts with the input to this neuron multiplicatively So what flows to the cell body Of this neuron is w times x But there's multiple inputs. There's many w times x is flowing to the cell body The cell body then has also like some bias So this is kind of like the In in their innate sort of trigger happiness of this neuron So this bias can make it a bit more trigger happy or a little less trigger happy regardless of the input But basically we're taking all the w times x Of all the inputs adding the bias and then we take it through an activation function And this activation function is usually some kind of a squashing function Like a sigmoid or 10-h or something like that. So as an example We're going to use the 10-h in this example um, numpy has a NP.10-h So um, we can call it on a range Then we can plot it This is the 10-h function and you see that the inputs as they come in Get squashed on the wipe coordinate here. So um Right at zero. We're going to get exactly zero and then as you go more positive in the input Then you'll see that the function will only go up to one and then plateau out And so if you pass in very positive inputs, we're gonna cap it smoothly at one And on the negative side we're gonna cap it smoothly to negative one So that's 10-h and that's the squashing function or an activation function And what comes out of this neuron is just the activation function applied to the dot product of the weights and the inputs So let's write one out um I'm going to copy paste because I don't want to type too much But okay, so here we have the inputs x1 x2 so this is a two-dimensional neuron. So two inputs are gonna come in These are thought out as the weights of the neuron weights w1 w2 and these weights again are the synaptic strengths for each input And this is the bias of the neuron b And now we want to do is according to this model we need to multiply x1 times w1 and x2 times w2 And then we need to add bias on top of it And it gets a little messy here, but all we are trying to do is x1 w1 plus x2 w2 plus b And these are multiply here Except I'm doing it in small steps so that we actually have pointers to all these intermediate nodes So we have x1 w1 variable x times x2 w2 variable and I'm also labeling them So n is now the cell body raw activation without the activation function for now And this should be enough to basically plot it so draw out of n Gives us x1 times w1 x2 times w2 Being added then the bias gets added on top of this and this n is this sum So we're now going to take it through an activation function And let's say we use the 10h so that we produce the output So what we'd like to do here is we'd like to do the output and I'll call it O is n dot 10h Okay, but we haven't yet written the 10h Now the reason that we need to implement another 10h function here is that 10h is a hyperbolic function and we've only so far implemented plus and at times and you can't make a 10h out of just pluses and times You also need explanation So 10h is this kind of a formula here You can use either one of these and you see that there is Explanation involved which we have not implemented yet for our little value node here So we're not going to be able to produce 10h yet and we have to go back up and implement something like it now one option here is We could actually implement X-managementation And we could return the X-off value instead of a 10h of a value Because if we had X then we have everything else that we need So because we know how to add and we know how to um We know how to add and we know how to multiply So we'd be able to create 10h if we knew how to X-off But for the purposes of this example, I specifically wanted to Show you That we don't necessarily need to have the most atomic pieces in um In this value object we can actually like create functions at arbitrary Points of abstraction they can be complicated functions But they can be also very very simple functions like a plus and it's totally up to us The only thing that matters is that we know how to differentiate through any one function So we take some inputs and we make an output the only thing that matters can be arbitrarily complex function As long as you know how to create the local derivative If you know the local derivative of how the inputs impact the output then that's all you need So we're going to cluster up all of this expression And we're not going to break it down to its atomic pieces We're just going to directly implement 10h So let's do that depth 10h And then out will be a value Of And we need this expression here. So um Let me actually copy-based What's graph n which is a solid data and then this I believe is the 10h Math dot x off two No n that m minus one over two n plus one maybe I can call this x Just that it matches exactly Okay, and now this will be t And uh children of this node they're just one child And I'm wrapping it in a tuple so this is a couple of one object just self And here the name of this operation will be 10h And we're going to return that Okay So now values should be implementing 10h and now it's rolled away down here And we can actually do n dot 10h and that's going to return the 10h Output of n and now we should be able to draw that of oh not of n So let's see how that worked There we go and went through 10h To produce this up it So now 10h is a sort of Our little micrograt supported node here as an operation And as long as we know derivative of 10h Then we'll be able to back propagate through it Now let's see this 10h in action Currently, it's not squashing too much because the input to it is pretty low So if the bias was increased to say eight Then we'll see that what's flowing into the 10h now is two And 10h is squashing to point nine six So we're already hitting the tail of this 10h And it will sort of smoothly go up to one and then plateau out over there Okay, so now I'm going to do something slightly strange I'm going to change this bias from eight to this number 6.88 etc And I'm going to do this for specific reasons because we're about to start back propagation And I want to make sure that our numbers come out nice They're not like very crazy numbers. They're nice numbers that we can sort of understand in our head Let me also add pose label Oh, it's short for output here So that's the R Okay, so 28 flows into 10h comes up point seven So now we're going to do back propagation and we're going to fill in all the gradients So what is the derivative or with respect to all the inputs here And of course in a typical neural network setting what we really care about the most is the derivative of These neurons on the weights specifically the W2 and W1 because those are the weights that we're going to be changing part of the optimization And the other thing that we have to remember is here we have only single neuron But in the neural net you typically have many neurons and they're connected So this is only like a one small neuron a piece of a much bigger puzzle and eventually there's a loss function That sort of measures the accuracy of the neural net and we're back propagating with respect to that accuracy and trying to increase it So let's start off back propagation here and and What is the derivative? Oh with respect to oh the base case sort of we know always is that the gradient is just one point there So let me fill it in and then let me split out the drawing function um here And then here sell Clear this output here, okay So now when we draw oh we'll see that oh that grad is one So now we're going to back propagate through the 10 H So to back propagate through 10 H we need to know the local derivative of 10 H So if we have that oh is 10 H of n Then what is d oh by d n? Now what you could do is you could come here and you could take this expression and you could do your calculus derivative taking Um, and that would work But we can also just scroll down with the pd i here Into a section that hopefully tells us that derivative uh d by dx of 10 H of x is Any of these I like this one one minus 10 H square of x So this is one minus 10 H of x squared So basically what this is saying is that d oh by d n is one minus 10 H often squared And we already have 10 H of n. It's just oh So it's one minus oh squared. So it was the output here. So the output is this number Odadega is this number and then What this is saying is that d oh by d n is one minus this squared. So one minus Odadega squared It's point five conveniently So the local derivative of this 10 H operation here is point five and uh, so that would be d oh by d n so we can fill in that n dot grad Is point five. We'll just fill in So this is exactly point five one half So now we're going to continue the back propagation This is point five and this is a plus node So how is back prop going to what is back prop going to do here And if you remember our previous example a plus is just a distributor of gradient So this gradient will simply flow to both of these equally And that's because the local derivative of this operation is one for every one of its nodes So one times point five is point five So therefore we know that this node here which we called this It's grad is just point five and we know that b dot grad is also point five So let's set those and let's draw So those are point five continuing. We have another plus point five again. We'll just distribute you So point five will flow to both of these so we can set theirs x2w2 as well that grad is point five And let's read wrong. Pluses are my favorite operations to back propagate through because it's very simple So now it's flowing into these expressions is point five And so really again keep in mind what the derivative is telling us at every point in time along here This is saying that if we want the output of this neuron to increase Then the influence on these expressions is positive on the output both of them are positive Contribution to the output So now back propagating to x2 and w2 first This is a times node so we know that the local derivative is no the other term So if we want to calculate x2 dot grad Then can you think through what it's going to be So x2 dot grad will be w2 dot data times This x2w2 dot grad right And w2 dot grad will be x2 dot data times x2w2 dot grad Right, so that's the little local piece of chain rule Let's set them and let's redraw So here we see that the gradient on our weight two is zero because x2's data was zero Right, but x2 will have the gradient point five because data here was one And so what's interesting here, right is because the input x2 was zero And because of the way the times works Um, of course this gradient will be zero and think about intuitively why that is Derbit it always tells us the influence of This on the final output if I will w2 how is the output changing? It's not changing because we're multiplying by zero So because it's not changing there is no derivative and zero is the correct answer Because we're multiplying or swashing with that zero And let's do it here point five should come here and flow through this times And so we'll have that x1 dot grad is Can you think through a little bit what what this should be The local derivative of times with respect to x1 is going to be w1 So w1's data times x1 w1 dot grad And w1 dot grad will be x1 dot data times x1 w2 w1 dot grad Let's see what those came out to be So this is point five. So this would be negative 1.5 and this would be one And we backpropagated through this expression these are the actual final derivatives So if we want this neurons output to increase We know that what's necessary is that uh W2 we have no gradient W2 doesn't actually matter to this neuron right now But this neuron this weight should uh go up So if this weight goes up then this neuron's output would have gone up And proportionally because the gradient is one Okay, so during the backpropagation manual is obviously ridiculous So we are now going to put an end to this suffering And we're going to see how we can implement uh the backward pass a bit more automatically. We're not going to be doing all of it manually out here It's now pretty obvious to us by example how these pluses and times are backpropagated ingredients So let's go up to the value object and we're going to start codifying what we've seen uh in the examples below So we're going to do this by storing a special self-doubt backward And uh underscore backward and this will be a function Which is going to do that little piece of chain rule at each little node that complete that took inputs and produced output Uh we're going to store How we are going to chain the the outputs gradient into the inputs gradients So by default This will be a function that uh doesn't do anything Uh so um And you can also see that here in the value in micrograd So with this backward function And by default doesn't do anything This is a function And that would be sort of the case for example for leaf node for leaf node. There's nothing to do But now if when we're creating these out values These out values are an addition of self and other And so we're going to want to self set Out's backward to be the function that propagates the gradient So So Let's define what should happen And we're going to store it in a closure. Let's define what should happen when we call Out's grad for an addition Our job is to take Out's grad and propagate it into self-scrad and other dot grad So basically we want to self-self grad to something And We want to set others that grad to something Okay And the way we saw below how chain rule works We want to take the local derivative times the um sort of global derivative I should call it which is the derivative of the final output of the expression with respect to out's data Respect to out so The local derivative of self in an addition is 1.0 So it's just 1.0 times out's grad That's the chain rule And others that grad will be 1.0 times out grad And what you basically what you're seeing here is that out's grad Will simply be copied onto self-scrad and others grad as we saw happens for an addition operation So we're going to later call this function to propagate the gradient having done an addition Let's not do multiplication we're going to also define a dot backward And we're going to set its backward to be backward And we want to chain out grad into self-scrad And others that grad And this will be a little piece of chain rule for multiplication So we'll have so what should it be? Can you think through So what is the local derivative? Here the local derivative was others that data And then There's other stuff data and then times out that grad that's chain rule And here we have self-that data times out that grad That's what we've been doing And finally here for 10h that backward And then we want to set out backwards to be just backward And here we need to Back-propagate we have out that grad and we want to chain it into self-that grad And self-that grad will be The local derivative of this operation that we've done here which is 10h And so we saw that the local gradient is 1 minus the 10h of x squared which here is t That's the local derivative because that's t is the output of this 10h So 1 minus t squared is the local derivative And then gradient Has to be multiplied because of the chain rule So out grad is chained through the local gradient into self-that grad And that should be basically it So we're going to redefine our value node We're going to swing all the way down here And we're going to redefine our expression Make sure that all the grads are zero Okay, but now we don't have to do this manually anymore We are going to basically be calling the dot backward in the right order So first we want to call oaths dot backward So o was the outcome of 10h Right so column oaths that back those those backward Will be this function. This is what it will do Now we have to be careful Because there's times out that grad and out that grad remember is initialized to zero So here we see grad zero. So as a base case We need to set oath dot grad to 1.0 To initialize this with one And then once this is one We can call o dot backward and what that should do is it should propagate this grad through 10h So the local derivative times the global derivative which is initialize at one. So this should Um Uh, no So I thought about redoing it but I figured I should just leave the error in here because it's pretty funny Why is not I object not collable Uh, it's because I screwed up we're trying to save these functions. So this is correct this here We don't want to call the function because that returns none these functions return none Which is want to store the function So let me redefine the value object And then we're going to come back and redefine the expression draw dot Everything is great. O dot grad is one O dot grad is one and now Now this should work of course Okay, so all that backward should have This grad should now be point five if we withdraw and everything was correctly point five yay Okay, so now we need to call ns dot grad ns dot backward sorry ns backward So that seems to have worked So ns dot backward Rapted the gradient to both of these so this is looking great Now we can of course call b dot grad Be the backwards or What's going to happen? Well b doesn't have it backward Bees backward because b is a leaf node Bees backward is by initialization the empty function So nothing would happen but we can call call it on it But when we call This one backwards Then we expect this point five to get further routed Right, so there we go point five point five And then finally We want to call it here on x2w2 And on x1w1 Let's do both of those and there we go So we get 0.5 negative 1.5 and 1 exactly as we did before But now we've done it through calling that backward Sir manually So we have one last piece to get rid of which is us calling underscore backward manually So let's think through what we are actually doing Um We've laid out a mathematical expression and now we're trying to go backwards through that expression Um, so going backwards through the expression just means that we never want to call a dot backward for any node Before we've done sort of um Everything after it So we have to do everything after it before ever going to call dot backward on any one node We have to get all of its full dependencies everything that it depends on has to Propagate to it before we can continue that propagation So this ordering of graphs can be achieved using something called topological sort So topological sort Is basically a laying out of a graph Such that all the edges go only from left to right basically So here we have a graph So direction as such like a graph a dag And this is two different topological orders of it, I believe Where basically you'll see that it's a laying out of the nodes such that all the edges go only one way from left to right And implementing topological sort you can look in Wikipedia and so on. I'm not going to go through it in detail But basically this is what builds a topological graph Um, we maintain a set of visited nodes and then we are um Going through starting at some root node which for us is oh, that's what we want to start the top logical sort And starting at oh we go through all of its children and we need to lay them out from left to right And basically this starts at oh if it's not visited then it marks it as visited and then it iterates through all of its children And calls build topological on them And then uh after it's gone through all the children it adds itself So basically This node that we're going to call it on like say oh is only going to add itself to the topical list After all of the children have been processed and that's how this function is guaranteeing That you're only going to be in the list once all your children are in the list and that's the invariant that is being maintained So if we built up on oh and then inspect this list We're going to see that it ordered our value objects And the last one is the value of 0.7 which is the output So this is oh and then this is n and then all the other nodes get laid out before it So that built the topological graph and really what we're doing now is we're just calling that underscore backward on all of the nodes in a topological order So if we just reset the gradients they're all zero So what did we do? We started by setting o.grad to be one That's that base case Then we built the topological order And then we went for node in reversed octopo Now in the reverse order because this list goes from you know we need to go through it in reverse order So starting at o node dot backward and this should be it There we go Those are the correct derivatives finally we are going to hide this functionality So I'm going to copy this and we're going to hide it inside the value class because we don't want to have all that code lying around So instead of an underscore backward we're now going to define an actual backward so that backward without the underscore And that's going to do all the stuff that we just derived So let me just clean this up a little bit. So We're first going to Build the topological graph Starting at self So build topo of self Will populate the topological order into the topo list which is a local variable Then we set self-adgrad to be one And then for each node in the reversed list so starting at us and going to all the children Uh, underscore backward And um, that should be it. So save Come down here we define Okay, all the grand are zero And now what we can do is oh, down backward without the underscore and There we go and that's uh, that's back propagation Please for one euro now we shouldn't be too happy with ourselves actually because we have a bad bug Um, and we have not surfaced the bug because of some specific conditions that we are have we have to think about right now So here's the simplest case that shows the bug Say I create a single node a And then I create a b that is e plus a And then I call it backward So what's gonna happen is a is three and then a is b is a plus a so there's two arrows on top of each other here Then we can see that b is of course the forward pass works b is just a plus a which is six But the gradient here is not actually correct That we calculate it automatically And that's because um You Of course, uh, just doing calculus in your head the derivative of b with respect to a should be uh two One plus one it's not one And totally what's happening here, right? So b is the result of a plus a and then we call backward on it So let's go up and see what that does um, b is a result of addition so out as b And then when we call backward what happened is self that grad was set to one And then other that grad was set to one But because we're doing a plus a self and other are actually these as a object So we are overriding the gradient we are setting it to one and then we are setting it again to one and that's why it stays at one So that's a problem There's another way to see this in a little bit more complicated expression So here we have a and b and then uh, d will be the multiplication of the two and he will be the addition of the two and um Then we multiply times d to get f and then we call it f that backward And these gradients if you check will be incorrect So fundamentally what's happening here again is um Basically, we're going to see an issue anytime we use a variable more than once Until now in these expressions above every variable is used exactly once so we didn't see the issue But here if a variable is used more than once what's going to happen during backward pass We're backpropagating from f to e to d so far so good But now e calls it backward and it deposits its gradients to a and b But then we come back to d and call backward and it overrides those gradients at a and b So that's obviously a problem And the solution here if you look at the multi-variate case of the chain rule and its generalization there The solution there is basically that we have to accumulate these gradients these gradients add And so instead of setting those gradients We can simply do plus equals we need to accumulate those gradients plus equals plus equals plus equals plus equals And this will be okay remember because we are initializing them at zero. So they started zero and then any contribution that flows backwards Will simply add So now if we redefine this one Because the plus equals this now works because a dot grad started at zero and we called b dot backward We deposit one and then we deposit one again and now this is two which is correct And here this will also work and we'll get correct gradients Because when we call e dot backward we will deposit the gradients from this branch And then we get to back to d dot backward it will deposit its own gradients And then those gradients simply add on top of each other And so we just accumulate those gradients and that fixes the issue Okay, now before we move on let me actually do a bit of cleanup here and delete some of these some of the intermediate work So I'm not going to need any of this now that we've derived all of it um We are going to keep this because I want to come back to it Delete the 10h delete arm when you can example delete the step delete this keep the code that draws And then delete this example and leave behind only the definition of value And now let's come back to this non-linearity here that we implemented the 10h Now I told you that we could have broken down 10h into its explicit atoms In terms of other expressions if we had the x function So if you remember 10h is defined like this And we chose to develop 10h as a single function And we can do that because we know it's derivative and we can back propagate through it But we can also break down 10h into an expressive function of x And I would like to do that now because I want to prove to you that you can all the same results and all the same gradients Um, but also because it forces us to implement a few more expressions It forces us to do Accumentation, addition, subtraction, division and things like that And I think it's a good exercise to go through a few more of these Okay, so let's scroll up To the definition of value And here one thing that we currently can't do is we can do like a value of say 2.0 But we can't do you know here for example we want to add a constant one and we can't do something like this And we can't do it because it's just into object has no attribute data That's because a plus one comes right here to add And then other is the integer one and then here Python is trying to access one dot data And that's not a thing And that's because basically one is not a value object and we only have addition from value objects So as a matter of convenience so that we can create expressions like this and make them make sense We can simply do something like this Basically we let other alone if other is an instance of value But if it's not an instance of value we're going to assume that it's a number like an integer or float And we're going to simply wrap it in in value And then other will just become value of other and then other will have a data attribute and this should work So if I just say this read the farm value then this should work There we go Okay, now let's do the exact same thing for multiply because we can't do something like this Again for the exact same reason so we just have to go to mall and if other is Not a value then let's wrap it in value Let's redefine value and now this works Now here's a kind of unfortunate and not obvious part A times two works we saw that but two times a is that going to work You'd expect it to write but actually it will not And the reason it won't is because Python doesn't know Like when when you do a times two Basically um, so a times two Python will go and it will basically do something like a dot mall Of two that's basically what we'll call but to it two times a is the same as two dot mall of a And it doesn't two can't multiply Value and so it's really confused about that So instead what happens is in Python the way this works is you are free to define something called the armol and And armol is kind of like a fallback So if the Python can't do two times a it will check if um If by any chance a knows how to multiply it too and that will be called into armol So because Python can't do two times a it will check is there an armol in value and because there is it will now call that And what we'll do here is we will swap the order of the operands So basically two times a will redirect to armol and armol will basically call it times two and that's how that will work So redefining that with armol two times a becomes four Okay, now looking at the other elements that we still need we need to know how to exponentiate and how to divide So let's first the explanation to the explanation part. We're going to introduce a single function x here And x is going to mirror 10h in the sense that it's a single single function that transform a single scalar value and outputs a single scalar value So we pop out the Python number We use method x to x-manitiate it create a new value object Everything that we've seen before The tricky part of course is how do you back propagate through e to the x? And uh, so here you can potentially pause the video and think about what should go here Okay, so basically we need to know what is the local derivative of e to the x So d by dx of e to the x is famously just e to the x And we've already just calculated e to the x and it's inside out that data So we can do about that data times And out that grad that's the chain So we're just chaining on to the current run grad And this is what the expression looks like it looks a little confusing But uh, this is what it is and that's the explanation So redefining we should not be able to call a dot x And uh, hopefully the backward pass works as well Okay, and the last thing we'd like to do of course is if we'd like to be able to divide Now I actually will implement something slightly more powerful than division because division is just a special case of Something a bit more powerful So in particular just by rearranging If we have some kind of a b equals Uh, value of 4.0 here we'd like to basically be able to do a divide b and we'd like this to be able to give us 0.5 Now division actually can be reshoffled as follows If we have a divide b That's actually the same as a multiplying 1 over b And that's the same as a multiplying b to the power of negative 1 And so what I'd like to do instead is I basically like to implement the operation of x to the k for some constant uh, k so it's an integer or a float um, and we would like to be able to differentiate this and then as a special case Uh, negative 1 will be division And so I'm doing that just because uh, it's more general and um, yeah, you might as well do it that way So basically what I'm saying is we can redefine uh, division Which we will put here somewhere You know, we can put this here somewhere What I'm saying is that we can redefine division so self-divided other It can actually be rewritten as self times other to the power of negative 1 And now Value raised to the power of negative 1 we had to now define that So here's So we need to implement the pow function Where am I going to put the pow function maybe here somewhere This is this call it from Fort So this function will be called when we try to raise a value to some power and other will be that power Now I'd like to make sure that other is only an int or a float usually other is some kind of a different value object But here other will be forced to be an int or a float otherwise the math Uh, won't work for for we're trying to achieve in the specific case That would be a different derivative expression if we wanted other to be a value So here we create the up the value which is just uh, you know, this data raised to the power of other and other here could be for example negative 1 That's what we are hoping to achieve And then uh, this is the backward stub and this is the fun part which is what is the uh chain rule expression here for back for um Backpropagating through the power function where the power is to the power of some kind of a constant So this is the exercise and maybe pause the video here and see if you can figure it out yourself as to what we should put here Okay, so um, you can actually go here and look at the derivative rules as an example And we see lots of the derivatives that you can hopefully know from calculus in particular what we're looking for is the power rule Because that's telling us that if we're trying to take d by dx of x to the n which is what we're doing here Then that is just n times x to the n minus 1 right Okay So that's telling us about the local derivative of this power operation So all we want here Basically n is now other and self that data is x and so this now becomes Other which is n times self that data Which is now a python in to reflote Uh, it's not a valley object. We're accessing the data attribute raised To the power of other minus one or n minus one I can put brackets around this, but this doesn't matter because um Power takes precedence over multiply and pi hell so that would have been okay And that's the local derivative only but now we have to chain it and we chain it just as simply by multiplying by on thought graph That's chain rule and this should uh technically work And we're gonna find out soon, but now if we do this this should now work And we get one five so the forward pass works, but thus the backward pass work And I realized that we actually also have to know how to subtract so right now a minus b will not work To make it work. We need one more piece of code here and basically this is the Subtraction and the way we're gonna implement subtraction is we're gonna implement it by addition of an negation and then to implement negation We're gonna multiply by negative one So just again using the stuff we've already built and just um expressing it in terms of what we have And a minus b does not work in Okay, so now let's scroll again to this expression here for this neuron And let's just compute the backward pass here once we've defined oh and let's draw it So here's the gradients for all these lead nodes for this two dimensional neuron that has a 10H that we've seen before So now what I'd like to do is I'd like to break up this 10H into this expression here So let me copy paste this here And now instead of we'll preserve the label and we will change how we define oh So in particular we're going to implement this formula here So we need each of the two x minus one over each of the x plus one So e to the two x we need to take two times m and we need to explain it That's e to the two x and then because we're using it twice Let's create an intermediate variable e And then define oh as e plus one over e minus one over e plus one e minus one over e plus one And that should be it and then we should be able to draw dot of oh So now before I run this what do we expect to see Number one we're expecting to see a much longer graph here because we've broken up 10H into a bunch of other operations But those operations are mathematically equivalent and so what we're expecting to see is number one The same result here. So the forward pass works and number two because of that mathematical equivalence We expect to see the same backward pass and the same gradients on these lead nodes. So these gradients should be identical So let's run this So number one let's verify that instead of a single 10H node we have now x and we have Plus we have times negative one. This is the division And we end up with the same forward pass here and then the gradients we have to be careful because they're in slightly different order potentially The gradients for w2 x2 should be 0 and 0.5 W2 and x2 are 0 and 0.5 and w1 x1 are 1 and negative 1.5 1 and negative 1.5 So that means that both our forward passes and backward passes were correct because this turned out to be equivalent to 10H before And so the reason I wanted to go through this exercises number one we got to practice a few more operations and Writing more backwards passes and number two. I wanted to illustrate the point that the um The level at which you implement your operations is totally up to you. You can implement backward passes for tiny Expressions like a single individual plus or a single times or you can implement them for say 10H Which is a kind of a potentially you can see it as a composite operation because it's made up of all these more atomic operations But really all of this is kind of like a fake concept all that matters is we have some kind of inputs and some kind of an output And this output is a function of the inputs in some way and as long as you can do forward pass and the backward pass of that little operation It doesn't matter what that operation is um and how composite it is If you can write the local gradients you can chain the gradient and you can continue back propagation So the design of what those functions are is completely up to you So now I would like to show you how you can do the exact same thing But using a modern deep neural network library like for example PyTorch Which I've roughly modeled micrograd By and so PyTorch is something you would use in production and I'll show you how you can do the exact same thing But in PyTorch API So I'm just going to copy-paste it in and walk you through it a little bit. This is what it looks like So we're going to import PyTorch and then we need to define these Value objects like we have here now micrograd is a scalar valued um engine so we only have scalar values like 2.0 But in PyTorch everything is based around tensors and like I mentioned tensors are just Indimensional arrays of scalars So that's why things get a little bit more complicated here. I just need a scalar valued tensor A tensor with just a single element But by default when you work with PyTorch you would use um more complicated tensors like this so if I import PyTorch Then I can create tensors like this and this tensor for example is a 2 by 3 array Of scalar scalars Um in a single compact representation. So you can check it shape. We see that it's a 2 by 3 array and so So this is usually what you would work with um in the actual libraries. So here I'm creating a tensor That has only a single element 2.0 And then I'm casting it to be double Because PyTorch is by default using double precision force floating point numbers So I like everything to be identical by default the data type of these tensors will be float 32 So it's only using a single precision float So I'm casting it to double So that we have float 64 just like in Python So I'm casting to double and then we get something similar to value of 2 The next thing I have to do is because these are leaf nodes by default PyTorch assumes that they do not require gradients So I need to explicitly say that all of these nodes require gradients Okay, so this is going to construct scalar valued one element tensors Make sure that PyTorch knows that they require gradients Now by default these are said to false by the way because of efficiency reasons Because usually you would not want gradients for leaf nodes Like the inputs to the network and this is just trying to be efficient in the most common cases So once we've defined all of our values in PyTorch land We can perform arithmetic just like we can here in micrograd land So this will just work and then there's a torshtot 10h also And when we get back is a tensor again And we can just like in micrograd it's got a data attribute and it's got grad attributes So these tensor objects just like in micrograd have a dot data and a dot grad And the only difference here is that we need to call a dot item because otherwise um PyTorch Dot item basically takes a single tensor of one element and it just returns that element stripping out the tensor So let me just run this and hopefully we are going to get this is going to print the forward pass Which is 0.707 and this will be the gradients which hopefully are 0.5 to 0 negative 0.5 and 1 So if we just run this There we go 0.7 so the forward pass agrees and then 0.5 0, 80, 0.5 and 1 So PyTorch agrees with us And just to show you here basically oh Here's a tensor with a single element And it's a double and we can call that item on it to just get the single number out So that's what item does and oh is a tensor object like I mentioned and it's got a backward function just like we've implemented And then all of these also have a dot grad so like x2 for example on the grad and it's a tensor And we can pop out the individual number with dot actum So basically torches Torch can do what we did in micrograd as a special case when your tensors are all single element tensors But the big deal with PyTorch is that everything is significantly more efficient Because we are working with these tensor objects and we can do lots of operations in parallel on all of these tensors But otherwise what we've built very much agrees with the API of PyTorch Okay, so now that we have some machinery to build out pretty complicated mathematical expressions We can also start building up neural nets and as I mentioned neural nets are just a specific class of mathematical expressions So we're going to start building out a neural net piece by piece and eventually we'll build out a two layer multi layer Layer perceptron as it's called and I'll show you exactly what that means Let's start with a single individual neuron We've implemented one here But here I'm going to implement one that also subscribes to the PyTorch API and how it designs its neural network modules So just like we saw that we can like matched API of PyTorch On the autograd side We're going to try to do that on the neural network modules So here's class neuron And just for the sake of efficiency I'm going to copy-base some sections that are relatively straightforward So the constructor will take a number of inputs to this neuron which is how many inputs come to a neuron So this one for example is three inputs And then it's going to create a weight that is some random number between negative one and one for every one of those inputs And a bias that controls the overall trigger happiness of this neuron And then we're going to implement a depth underscore underscore call Of self and x, sum input x And really what we're not going to do here is w times x plus b We're w times x here because they dot power specifically Now if you haven't seen call Let me just return 0.0 here from now The way this works now is we can have an x which is say like 2.0 3.0 Then we can initialize a neuron that is two-dimensional Because these are two numbers And then we can feed those two numbers into that neuron to again and output And so when you use this notation n of x python will use call So currently call just returns 0.0 Now we'd like to actually do the forward pass of this neuron instead So what we're going to do here first is we need to basically multiply all of the elements of w with all of the elements of x pairwise We need to multiply them So the first thing we're going to do is we're going to zip up sultan w and x And in python zip takes two iterators And it creates a new iterator that iterates over the tuples of their corresponding entries So for example just to show you we can print this list And still returns 0.0 here Sorry So we see that these w's are paired up with the x's w with x And now what we're going to do is For wixi in We want to multiply w times wixi And then we want to sum all of that together to come up with an activation And add also sultan b on top So that's the real activation And then of course we need to pass that through a non-mejority So what we're going to be returning is act.h And here's out So now we see that we are getting some outputs And we get a different output from neuron each time because we are initializing different weights and biases And then to be a bit more efficient here actually Some by the way takes a second optional parameter which is the start And by default the start is 0 So these elements of this sum will be added on top of 0 to begin with But actually we can just start with sultan b And then we just have an expression like this And then the generator expression here must be parenthesisized by the line There we go Yep, so now we can forward a single neuron And next up we're going to define a layer of neurons So here we have a schematic for a mlp So we see that these mlp's each layer, this is one layer Has actually a number of neurons And they're not connected to each other but all of them are fully connected to the input So what is a layer of neurons? It's just a set of neurons evaluated independently So in the interest of time I'm going to do something fairly straightforward here It's um Literally a layer is just a list of neurons And then how many neurons do we have? We take that as an input argument here How many neurons do you want in your layer? A number of outputs in this layer And so we just initialize completely independent neurons With this given dimensionality And when we call on it we just independently evaluate them So now instead of a neuron we can make a layer of neurons They are two dimensional neurons and let's say three of them And now we see that we have three independent evaluations of three different neurons Right? Okay and finally let's complete this picture And define an entire multilayer perception or mlp And as we can see here in an mlp These layers just speed into each other sequentially So let's come here and I'm just going to Copy the code here in interest of time So an mlp is very similar We're taking the number of inputs as before But now instead of taking a single n out which is number of neurons and a single layer We're going to take a list of n outs And this list defines the sizes of all the layers that we want in our mlp So here we just put them all together And then iterate over consecutive pairs Of these sizes and create a layer objects for them And then in the call function we are just calling them sequentially So that's an mlp really And let's actually re-implement this picture So we want three input neurons And then two layers of four and an output unit So we want Three dimensional input Say this is an example input We want three inputs into two layers of four and one output And this of course is an mlp And there we go That's a forward passive in mlp To make this a little bit nicer You see how we have just a single element But it's wrapped in a list Because layer always returns lists So for convenience return out at zero if Lend out is exactly a single element Else return fullest And this will allow us to just get a single value out At the last layer that only has a single neuron And finally we should be able to prod out of n of x And as you might imagine These expressions are now getting Relatively involved So this is an entire mlp that we're defining now All the way until a single output Okay And so obviously you would never differentiate on pen and paper These expressions but with micrograd We will be able to back propagate all the way through this And back propagate Into these weights of all these neurons So let's see how that works Okay, so let's create ourselves a very simple Example data set here So this data set has four examples And so we have four possible inputs into the neural net And we have four desired targets So we'd like the neural net to Assign Or output 1.0 when it's fed this example Negative 1 when it's fed these examples And 1 when it's fed this example So it's a very simple binary classifier neural net Basically that we would like here Now let's think what the neural net currently thinks about these four examples We can just get their predictions Basically we can just call n of x for x in axis And then we can print So these are the outputs of the neural net on those four examples So the first one is 0.91 But we'd like it to be 1 So we should push this one higher This one we want to be higher This one says 0.88 And we want this to be negative 1 This is 0.88 We want it to be negative 1 And this one is 0.88 we want it to be 1 So how do we make the neural net And how do we tune the weights To better predict the desired targets And the trick used in deep learning to achieve this Is to calculate a single number That somehow measures the total performance of your neural net And we call this single number the loss So the loss First is a single number That we're going to define That basically measures how well the neural net is performing Right now we have the intuitive sense That it's not performing very well Because we're not very much close to this So the loss will be high And we'll want to minimize the loss So in particular in this case what we're going to do Is we're going to implement the mean squared error loss So what this is doing Is we're going to basically iterate For y ground truth And y output can zip off Ys and y bread So we're going to pair up the Ground truth with the predictions And this zip iterates over tuples of them And for each Y ground truth and y output We're going to subtract them And it's squared So let's first see what these losses are These are individual loss components And so basically for each One of the four We are taking the prediction And the ground truth We are subtracting them And squaring them So because this one is so close to its target 0.91 is almost one Subtracting them gives a very small number So here we would get like a negative point one And then squaring it Just makes sure That regardless of whether we are more negative Or more positive We always get a positive number Instead of squaring the Schrold We could also take for example the absolute value We need to discard the sign And so you see that the expression is Ranged so that you only get 0 exactly When y out is equal to y ground truth When those two are equal So your prediction is exactly the target You are going to get 0 And if your prediction is not the target You are going to get some other number So here for example we are way off And so that's why the loss is quite high And the more off we are The greater the loss will be So we don't want high loss We want low loss And so the final loss here Will be just the sum Of all of these numbers So you see that this should be 0 roughly plus 0 roughly But plus 7 So loss should be about 7 here And now we want to minimize the loss We want the loss to be low Because the loss is low Then every one of the predictions Is equal to 0 Then every one of the predictions is equal to its target So the loss, the loss, the loss, it can be 0 And the greater it is The worse off the neural net is predicting So now of course if we do Loss that backward Something magical happened when I hit enter And the magical thing of course that happened Is that we can look at endout layers That neuron, endout layers At say like the first layer That neurons at 0 Because remember that MLP has layers which is a list And each layer has a neurons which is a list And that gives us individual neuron And then it's got some weights And so we can for example Look at the weights at 0 Oops, it's not cold weights It's called W And that's a value But now this value also has a graph Because of the backward values And so we see that because this gradient here On this particular weight of this particular neuron Of this particular layer is negative We see that its influence on the loss Is also negative So slightly increasing this particular weight Of this neuron of this layer Would make the loss go down And we actually have this information For every single one of our neurons And all of their parameters Actually it's worth looking at Also the draw dot loss So the draw dot loss by the way So previously we looked at the draw dot Of a single neuron Neuralin forward pass And that was already a large expression But what is this expression? We actually forwarded Every one of those four examples And then we have the loss in top with them With the mean squared error And so this is a really massive graph Because this graph that we built up now Oh my gosh This graph that we built up now Which is kind of excessive It's excessive because it has four forward passes Of a neural net for every one of the examples And then it has the loss on top And it ends with the value of the loss Which for 7.1.2 And this loss will now back propagate Through all the forward passes All the way through just every single Intermediate value of the neural net All the way back to Of course the parameters of the weights Which are the input So these weight parameters here To this neural net And these numbers here These scalars are inputs to the neural net So if we went around here We will probably find Some of these examples This 1.0 potentially maybe this 1.0 Or you know some of the others And you'll see that they all have gradients as well The thing is these gradients on the input data Are not that useful to us And that's because The input data seems to be Not changeable And it's not given to the problem And so it's a fixed input We're not going to be changing it or messing with it Even though we do have gradients for it But some of these gradients here Will be for the neural network parameters The W's and the B's And those we of course we want to change Okay so now we're going to Want some convenience code To gather up all of the parameters of the neural net So that we can operate on on all of them simultaneously And every one of them A tiny amount Based on the gradient depermission So let's collect the parameters of the neural net all in one array So let's create a parameters of self That just returns Salt that W which is a list Concatenated with A list of Salt that B So this will just return a list List plus list just You know gives you a list So that's parameters of neural And I'm calling it this way because also Pipe Torch has a parameters on every single And in module And it does exactly what we're doing here It just returns the Parameter tensors for us is the primary scalers Now layer is also a module So it will have parameters Self And basically what we want to do here is Something like this like Param's is here And then for Neuron in salt that neurons We want to get neuron that parameters And we want to Param's that extend Right so these are the Parameters of this neuron And then we want to put them on top of Param's so Param's that extend of Peace And then we want to return Param's So this is way too much code So actually there's a way to simplify this Which is Return P For neuron in self That neurons For P in neuron dot parameters So it's a single list comprehension In python you can sort of nest Then like this and you can Then create The desired array So these are identical We can take this out And then let's do the same here Deframeters Self And return A parameter for layer In self dot layers For P in layer dot parameters And that should be good Now let me pop out this So we don't re-initialize our network Because we need to re-initialize our Okay so unfortunately we Will have to probably re-initialize Network because we just had Functionality because this class Of course we i want to get All the end dot parameters But that's not going to work because this is the old class Okay so unfortunately we do have to re-initialize the network Which will change some of the numbers But let me do that so That we can do that So that we pick up the new API We can now do end dot parameters And these are all the weights and biases Inside the entire neural net So in total this MLP has 41 parameters And now we'll be able to change them If we recalculate the loss here We see that unfortunately we have slightly different Predictionality Predictions and slightly different loss But that's okay Okay so we see that this neuron's Gradient is slightly negative We can also look at its data right now Which is 0.85 So this is the current value of this neuron And this is its gradient on the loss So what we want to do now is We want to iterate for every P N end dot parameters So for all the 41 parameters of this neuron net We actually want to change P data data Slightly according to the gradient information Okay so data to do here But this will be basically a tiny update In this gradient descent scheme And gradient descent we are thinking of the gradient As a vector pointing in the direction of increased loss And so in gradient descent We are modifying P dot data By a small step size in the direction of the gradient So the step size as an example could be Like a very small number of 0.01 is the step size Times P dot grad Right But we have to think through some of the signs here So in particular Working with this specific example here We see that if we just left it like this Then this neurons value Would be currently increased by a tiny amount of the gradient The gradient is negative So this value of this neuron would go slightly down It would become like 0.8 You know, 4 or something like that But if this neurons value goes lower That would actually increase the loss That because the derivative of this neuron is negative So increasing this makes the loss go down So increasing it is what we want to do instead of decreasing it So basically what we are missing here is We are actually missing a negative sign And again this other interpretation And that's because we want to minimize the loss We don't want to maximize the loss We want to decrease it And the other interpretation as I mentioned Is you can think of the gradient vector So basically just the vector of all the gradients As pointing in the direction of increasing the loss But then we want to decrease it So we actually want to go in the opposite direction And so you can convince yourself that this Or like thus the right thing here with the negative Because we want to minimize the loss So if we notch all the parameters by a tiny amount Then we'll see that this data will change a little bit So now this neuron is a tiny amount creator A tiny amount creator Value So 0.854, once it's 0.857 And that's a good thing Because slightly increasing this neuron Data makes the loss go down According to the gradient And so the correct thing has happened signwise And so now what we would expect of course is that Because we've changed all these parameters We expect that the loss should have gone down a bit So we want to re-evaluate the loss Let me basically This is just a data definition that hasn't changed But the forward pass here Of the network we can recalculate And actually let me do it outside here So that we can compare the two loss values So here if I recalculate the loss We'd expect the new loss now to be slightly lower than this number So hopefully what we're getting now Is a tiny bit lower than 4.854 4.36 And remember the way we've arranged this Is that low loss means that our predictions are matching the targets So our predictions now are probably slightly closer to the targets And now all we have to do is we have to iterate this process So again we've done the forward pass And this is the loss Now we can lost that backward And we can do a step size And now we should have a slightly lower loss 4.36 goes to 3.9 And okay so We've done the forward pass Here's the backward pass And now the loss is 3.66 3.47 And you get the idea We just continue doing this And this is gradient descent We're just iteratively doing forward pass Backward pass Updates Forward pass, backward pass, update And the neural net is improving its predictions So here if we look at wide-pred now Wide-pred We see that This value should be getting closer to 1 So this value should be getting more positive These should be getting more negative And this one should be also getting more positive So if we just iterate this A few more times Actually we'll be able to afford to go a bit faster Let's try a slightly higher learning rate Whoops, okay There we go, so now we're at 0.31 If you go too fast by the way If you try to make it too big of a step You may actually overstep It's overconfidence Because again remember we don't actually know exactly about the loss function The loss function has all kinds of structure And we only know about the very local Dependence of all these parameters on the loss But if we step too far We may step into you know A part of the loss that is completely different And that can destabilize training And make your loss actually blow up even So the loss is now 0.04 So actually the predictions should be really quite close Let's take a look So you see how this is almost 1 Almost negative 1, almost 1 We can continue going So, yep, backward, update Up, there we go So we went way too fast And we actually overstepped So we got to 2, 2 eager Where are we now? Oops Okay, 7 in negative 9 So this is very, very low loss And the predictions are basically perfect So somehow we were doing way too big updates And we briefly explored it But then somehow we ended up getting into a really good spot So usually this learning rate and the tuning of it is a subtle art You want to set your learning rate If it's too low, you're going to take way too long to converge But if it's too high, the whole thing gets unstable And you might actually even explode the loss Depending on your loss function So finding the step size to be just right It's a pretty subtle art sometimes When you're using sort of a null-accradient descent But we happen to get into a good spot We can look at end dot parameters And we can see that the result is really good So we can see that we have a very good spot We can look at end dot parameters So this is the setting of weights and biases That makes our network predict the desired targets very, very close And basically we successfully trained in neural nut Okay, let's make this a tiny bit more respectable And implement an actual training loop And what that looks like is the initialization of that state This is the forward pass So for K in range We're going to take a bunch of steps First, you do the forward pass We evaluate the loss Let's re-initialize the neural nut from scratch And here's the data And we first do forward pass Then we do the backward pass And then we do an update That's gradient descent And then we do an update That's gradient descent And then we should be able to iterate this And we should be able to print the current step The current loss Let's just print the sort of Number of the loss And that should be it And then the learning rate 0.01 is a little too small 0.1 we saw is like a little bit dangerous if you buy Let's go somewhere between And we'll optimize this for Not 10 steps But let's go for say 20 steps Let me erase all of this junk And let's run the optimization And you see how we've actually converged slower In a more controlled manner And got through a loss that is very low So I expect wide-pred to be quite good There we go And that's it Okay, so this is kind of embarrassing But we actually have a really terrible bug In here And it's a subtle bug And it's a very common bug And I can't believe I've done it For the 20th time in my life Especially on camera And I could have reshot the bug And I could have reshot the whole thing But I think it's pretty funny And you know, you get to appreciate a bit What working with neural nuts maybe Is like sometimes We are guilty of A common bug I've actually tweeted The most common neural nut mistakes A long time ago now And I'm not really Gonna explain any of these Except for we are guilty of number three You forgot to zero grad Before dot backward What is that? Basically what's happening And it's a subtle bug and I'm not sure if you saw it Is that All of these Wates here have a dot data and a dot grad And dot grad starts at zero And then we do backward And we fill in the gradients And then we do an update on the data But we don't flush the grad It stays there So when we do the second Forward pass and we do backward again Remember that all the backward operations Do a plus equals on the grad And so these gradients Just add up and they never get reset to zero So basically we didn't zero grad So here's how we zero grad Before backward We need to iterate over all the parameters And we need to make sure that P dot grad is set to zero We need to reset it to zero Just like it is in the constructor So remember all the way here For all these value nodes Grad is reset to zero And then all these backward passes do a plus equals on that grad But we need to make sure that We reset these grads to zero So that when we do backward All of them start at zero And the actual backward pass accumulates The loss derivatives into the grads So this is zero grad in PyTorch And we will slightly We will get a slightly different optimization Let's reset the neural net The data is the same This is now I think correct And we get a much more You know we get a much more slower descent We still end up with pretty good results And we can continue this a bit more To get down lower And lower And lower Yeah So the only reason that the previous thing worked It's extremely buggy The only reason that worked Is that This is a very very simple problem And it's very easy for this neural net to fit this data And so the grads ended up accumulating And it effectively gave us a massive step size And it made us converge extremely fast But basically now we have to do more steps To get to very low values of loss And get why I pray to be really good We can try to step a bit greater Yeah We're going to get closer and closer to one minus one And one So we're going to do lots of sometimes tricky Because You may have lots of bugs in the code And your network might actually work Just like ours worked But chances are is that We had a more complex problem Than actually this bug would have made us not optimize the loss very well And we were only able to get away with it because The problem is very simple So let's now bring everything together And summarize what we learned What are neural nets? Neural nets are these mathematical expressions They're really simple mathematical expressions In case of multi-layer perceptron That take Input as the data And they take input the weights and the parameters of the neural net Mathematical expression for the forward pass Followed by a loss function And the loss function tries to measure the accuracy of the predictions And usually the loss will be low When your predictions are matching your targets Or where the network is basically behaving well So we manipulate the loss function so that when the loss is low The network is doing what you wanted to do on your problem And then we backward the loss Use back propagation to get the gradient And then we know how to tune all the parameters to Decrease the loss locally But then we have to iterate that process many times In what's called the gradient descent So we simply follow the gradient information And that minimizes the loss And the loss is arranged so that when the loss is minimized The network is doing what you want it to do And yeah, so we just have a blob of neural stuff And we can make it do arbitrary things And that's what gives neural nets their power It's, you know, this is a very tiny network with 41 parameters But you can build significantly more complicated neural nets With billions at this point, almost trillions of parameters And it's a massive blob of neural tissue Simulated neural tissue Roughly speaking And you can make it do extremely complex problems And these neural nets then have all kinds of very fascinating Emergent properties In when you try to make them do significantly hard problems So you can make it more difficult To do significantly hard problems As in the case of GPT for example We have massive amounts of text from the internet And we're trying to get a neural net to predict To take like a few words And try to predict the next word in a sequence That's the learning problem And it turns out that when you train this on all of the internet The neural net actually has like really remarkable Emergent properties But that neural net would have hundreds of billions of parameters But it works on fundamentally these exact same principles And the neural net of course will be a bit more complex But otherwise the Evaluating the gradient is there And it would be identical And the gradient descent would be there And would be basically identical But people usually use slightly different updates This is a very simple stochastic gradient sent update And the last function would not be an eSquared error They would be using something called the cross-entropy loss For predicting the next token So there's a few more details But fundamentally the neural network setup Or training is identical and pervasive And now you understand intuitively How that works under the hood In the beginning of this video I told you that by the end of it You would understand everything in micrograd And then we'd slowly build it up Let me briefly prove that to you So I'm going to start through all the code that is in micrograd As of today Actually, potentially some of the code will change By the time you watch this video Because I intend to continue developing micrograd But let's look at what we have so far at least When you go to engine.py that has the value Everything here you should mostly recognize So we have the dead.data.grad attributes We have the backward function We have the previous set of children And the operation that produced this value We have addition, multiplication, and raising to a scalar power We have the Rellou nonlinearity Which is slightly different type of nonlinearity than 10H That we used in this video Both of them are nonlinearties And notably 10H is not actually present in micrograd As of right now But I intend to add it later With the backward, which is identical And then all of these other operations Which are built up on top of operations here So value should be very recognizable Except for the nonlinearity used in this video There's no massive difference between Rellou and 10H And sigmoid and these other nonlinearties They're all roughly equivalent and can be used in MLPs So I use 10H because it's a bit smoother And because it's a little bit more complicated than Rellou And therefore it's stressed a little bit more And therefore it's stressed a little bit more And therefore it's stressed a little bit more The local gradients and working with those derivatives Which I probably would be useful And then the pi is the neural networks library as I mentioned So you should recognize identical implementation of your own Layer and MLP Notably, for not so much We have a class module here There's a parent class of all these modules I did that because there's an end-up module class in PyTorch And so this exactly matches that API And end-up module in PyTorch has also a zero-brad Which I refactored out here So that's the end of micro-grad really Then there's a test Which you'll see basically creates two chunks of code One in micro-grad and one in PyTorch And we'll make sure that the forward and the backward pass agree identically For a slightly less complicated expression A slightly more complicated expression Everything agrees So we agree with PyTorch on all of these operations And finally there's a demo that IpyY And finally there's a demo that IpyY And finally there's a demo that IpyY And finally there's a demo that IpyY and B here And it's a bit more complicated binary classification demo than the one I covered in this lecture So we only had a tiny data set of for examples Here we have a bit more complicated example With lots of blue points and lots of red points And we're trying to again build a binary classifier to distinguish Two dimensional points as red or blue It's a bit more complicated MLP here with it's a bigger MLP The loss is a bit more complicated because It supports batches It supports batches So because our data set was so tiny We always did a forward pass on the entire data set of for examples But when your data set is like a million examples What we usually do in practice is we We basically pick out some random subset We call that a batch And then we only process the batch Forward, backward, and update So we don't have to forward the entire training set So this supports batching Because there's a lot more examples here We do a forward pass The loss is slightly more different This is a max margin loss that I implement here The one that we used was the mean squared error loss Because it's the simplest one There's also the binary cross-entropy loss All of them can be used for binary classification And don't make too much of a difference In the simple examples that we looked at so far There's something called Alt-to-regularization used here This has to do with generalization of the neural net And controls the overfitting in machine learning setting So I did not cover these concepts in this video Potentially later And the training loop you should recognize So forward, backward, with, zero grad And update, and so on You'll notice that in the update here The learning rate is scaled as a function of number of iterations And it shrinks And this is something called learning rate decay So in the beginning you have a high learning rate And as the network sort of stabilizes near the end You bring down the learning rate to get some of the fine details in the end And in the end we see the decision surface of the neural net And we see that it learns to separate out the red and the blue area Based on the data points So that's a slightly more complicated example And the demo that I buy by YMB That you're free to go over But yeah, as of today, that is micro-grad I also wanted to show you a little bit of real stuff So that you get to see how this is actually implemented In the production grade library like by torch So in particular, I wanted to show I wanted to find and show you The backward password 10H in PyTorch So here in micro-grad We see that the backward password 10H is 1-t square Where t is the output of the 10H of X Times of that grad which is the chain rule So we're looking for something that looks like this Now I went to PyTorch Which has open source GitHub code base And I looked through a lot of its code And honestly, I wanted to show you I looked through a lot of its code and honestly I spent about 15 minutes and I couldn't find 10H And that's because these libraries Unfortunately, they grow in size and entropy And if you just search for 10H You get apparently 2,800 results And 400 and 406 files So I don't know what these files are doing honestly And why there are so many mentions of 10H But unfortunately, these libraries are quite complex They're meant to be used Not really inspected Eventually, I did stumble on someone Who tries to change the 10H Backward code for some reason And someone here pointed to the CPU kernel And the CUDA kernel for 10H backward So this basically depends on if you're using PyTorch On the CPU device or on the GPU Which these are different devices and I haven't covered this But this is the 10H backward kernel For CPU And the reason it's so large Is that a number one This is like if you're using a complex type Which we haven't even talked about If you're using a specific data type of B-float 16 Which we haven't talked about And then if you're not Then this is the kernel and deep here We see something that resembles our backward pass So they have 8 times 1 minus B-square So this B here Must be the output of the 10H And this is the out.grad So here we found it Deep inside PyTorch and this location For some reason inside binary ops kernel When 10H is not actually binary op And then this is the GPU kernel We're not complex We're here And here we go with online opcode So we did find it But basically unfortunately These code bases are very large And micrograd is very very simple But if you actually Want to use real stuff Finding the code for it You'll actually find that difficult I also wanted to show you The whole example here Where PyTorch is showing you how you can Register a new type of function That you want to add to PyTorch As a LEGO building block So here if you want to for example add A like jonder polynomial 3 Here's how you can do it You will register it And then you have to tell PyTorch How to forward your new function And how to backward through it So as long as you can do the forward pass Of this little function piece that you want to add And as long as you know The local derivatives Local gradients which are implemented in the Backward PyTorch will be able to Backpropagate through your function And then you can use this as a LEGO block In a larger LEGO castle Of all the different LEGO blocks that PyTorch already has And so that's the only thing you have to tell PyTorch And everything would just work And you can register new types of functions In this way following this example And that is everything that I wanted to cover in this lecture So I hope you enjoyed building out micro-grad With me, I hope you find it interesting And insightful And yeah, I will post a lot of the links That are related to this video In the video description below I will also probably post a link to a discussion forum Or discussion group Where you can ask questions related to this video And then I can answer Or someone else can answer your questions And I may also do a follow-up video That answers some of the most common questions But for now, that's it I hope you enjoyed it If you did, then please like and subscribe So that YouTube knows to feature this video to more people And that's it for now I'll see you later Now here's the problem We know DL by Wait, what is the problem And that's everything I wanted to cover in this lecture So I hope you enjoyed Us building out micro-grad Micro-grad Okay now let's do the exact same thing for multiple Because we can't do something like Eight times two Oops I know what happened there I know what happened there I know what happened there I know what happened there
[{"start": 0.0, "end": 5.86, "text": " Hello, my name is Andre and I've been training deep neural networks for a bit more than a decade and in this lecture"}, {"start": 5.86, "end": 9.18, "text": " I'd like to show you what neural network training looks like under the hood"}, {"start": 9.540000000000001, "end": 14.44, "text": " So in particular we are going to start with a blank super notebook and by the end of this lecture"}, {"start": 14.44, "end": 19.580000000000002, "text": " We will define and train in neural net and you'll get to see everything that goes on under the hood and exactly"}, {"start": 20.14, "end": 22.14, "text": " Sort of how that works and then to it in a little"}, {"start": 22.54, "end": 25.82, "text": " Now specifically what I would like to do is I would like to take you through"}, {"start": 25.82, "end": 31.86, "text": " Building of micrograd now micrograd is this library that I released on GitHub about two years ago"}, {"start": 31.86, "end": 36.46, "text": " But at the time I only uploaded this source code and you'd have to go in by yourself and really"}, {"start": 37.26, "end": 39.26, "text": " Figure out how it works"}, {"start": 39.26, "end": 43.78, "text": " So in this lecture I will take you through it step-by-step and kind of comment on all the pieces of it"}, {"start": 43.78, "end": 46.3, "text": " So what's micrograd and why is it interesting?"}, {"start": 47.260000000000005, "end": 49.260000000000005, "text": " cute"}, {"start": 49.74, "end": 52.06, "text": " micrograd is basically an auto-grad engine"}, {"start": 52.06, "end": 57.18, "text": " Auto-grad is short for automatic gradient and really what it does is it implements back propagation"}, {"start": 57.38, "end": 62.1, "text": " Now back propagation is this algorithm that allows you to efficiently evaluate the gradient of"}, {"start": 63.46, "end": 67.7, "text": " Some kind of a loss function with respect to the weights of a neural network"}, {"start": 67.7, "end": 73.86, "text": " And what that allows us to do then is we can editorively tune the weights of that neural network to minimize the loss function"}, {"start": 73.86, "end": 75.86, "text": " And therefore improve the accuracy of the network"}, {"start": 75.86, "end": 83.1, "text": " So back propagation would be at the mathematical core of any modern deep neural network library like say PyTorch or Jax"}, {"start": 83.86, "end": 88.86, "text": " So the functionality of micrograd is I think best illustrated by an example. So if we just scroll down here"}, {"start": 89.7, "end": 93.86, "text": " You'll see that micrograd basically allows you to build out mathematical expressions and"}, {"start": 95.22, "end": 100.14, "text": " Here what we are doing is we have an expression that we're building out where you have two inputs a and b"}, {"start": 100.14, "end": 105.66, "text": " And you'll see that a and b are negative 4 and 2 but we are wrapping those"}, {"start": 106.26, "end": 110.54, "text": " Values into this value object that we are going to build out as part of micrograd"}, {"start": 110.94, "end": 117.68, "text": " So this value object will wrap the numbers themselves and then we are going to build out a mathematical expression here where"}, {"start": 118.3, "end": 119.98, "text": " a and b are"}, {"start": 119.98, "end": 123.34, "text": " Transformed into cd and eventually e f and g"}, {"start": 123.34, "end": 128.52, "text": " And I'm showing some of the function some of the functionality of micrograd and the operations that it supports"}, {"start": 128.52, "end": 134.12, "text": " So you can add two value objects. You can multiply them. You can raise them to a constant power"}, {"start": 134.36, "end": 137.48000000000002, "text": " You can also by one the gate squash at zero"}, {"start": 138.68, "end": 142.16000000000003, "text": " Square divide by constant divide by it etc"}, {"start": 142.52, "end": 149.56, "text": " And so we're building out an expression graph with with these two inputs a and b and we're creating out the value of g"}, {"start": 149.56, "end": 150.76000000000002, "text": " and"}, {"start": 150.76000000000002, "end": 152.60000000000002, "text": " micrograd will in the background"}, {"start": 152.60000000000002, "end": 154.84, "text": " Build out this entire mathematical expression"}, {"start": 154.84, "end": 161.16, "text": " So it will for example know that c is also a value c was a result of an addition operation and"}, {"start": 161.8, "end": 162.84, "text": " the"}, {"start": 162.84, "end": 169.72, "text": " Child nodes of c are a and b because the and all maintain pointers to a and b value objects"}, {"start": 169.8, "end": 172.8, "text": " So we'll basically know exactly how all of this is laid out and"}, {"start": 173.4, "end": 178.04, "text": " Then not only can we do what we call the forward pass where we actually look at the value of g"}, {"start": 178.04, "end": 179.64000000000001, "text": " Of course, that's pretty straightforward"}, {"start": 179.64000000000001, "end": 182.8, "text": " We will access that using the dot data attribute"}, {"start": 182.8, "end": 188.56, "text": " And so the output of the forward pass the value of g is 24.7 it turns out"}, {"start": 188.8, "end": 194.4, "text": " But the big deal is that we can also take this g value object and we can call dot backward"}, {"start": 194.96, "end": 198.64000000000001, "text": " And this will basically initialize back propagation at the node g"}, {"start": 199.84, "end": 206.0, "text": " And what back propagation is going to do is it's going to start at g and it's going to go backwards through that expression graph"}, {"start": 206.24, "end": 209.68, "text": " and it's going to recursively apply the chain rule from calculus and"}, {"start": 209.68, "end": 217.6, "text": " And what that allows us to do then is we're going to evaluate basically the derivative of g with respect to all the internal nodes"}, {"start": 218.24, "end": 222.64000000000001, "text": " Like ed and c, but also with respect to the inputs a and b"}, {"start": 223.28, "end": 228.48000000000002, "text": " And then we can actually query this derivative of g with respect to a for example"}, {"start": 228.48000000000002, "end": 233.76000000000002, "text": " That's a dot grad in this case it happens to be 138 and a derivative of g with respect to b"}, {"start": 234.32, "end": 236.88, "text": " Which also happens to be here 645"}, {"start": 236.88, "end": 244.48, "text": " And this derivative we'll see soon is very important information because it's telling us how a and b are affecting g"}, {"start": 245.04, "end": 250.0, "text": " Through this mathematical expression. So in particular a dot grad is 138"}, {"start": 250.48, "end": 254.07999999999998, "text": " So if we slightly nudge a and make it slightly larger"}, {"start": 254.96, "end": 260.15999999999997, "text": " 138 is telling us that g will grow in the slope of that growth is going to be 138"}, {"start": 260.96, "end": 264.08, "text": " And the slope of growth of b is going to be 645"}, {"start": 264.08, "end": 270.32, "text": " So that's going to tell us about how g will respond if a and b get tweaked a tiny amount in a positive direction"}, {"start": 271.28, "end": 273.28, "text": " Okay"}, {"start": 273.52, "end": 279.52, "text": " Now you might be confused about what this expression is that we built out here and this expression by the way is completely meaningless"}, {"start": 279.84, "end": 284.4, "text": " I just made it up. I'm just flexing about the kinds of operations that are supported by micrograd"}, {"start": 284.88, "end": 287.12, "text": " What we actually really care about are neural networks"}, {"start": 287.52, "end": 291.52, "text": " But it turns out that neural networks are just mathematical expressions just like this one"}, {"start": 291.52, "end": 293.84, "text": " But actually a slightly bit less crazy even"}, {"start": 294.47999999999996, "end": 295.03999999999996, "text": " um"}, {"start": 295.03999999999996, "end": 297.03999999999996, "text": " Neural networks are just a mathematical expression"}, {"start": 297.2, "end": 303.59999999999997, "text": " They take the input data as an input and they take the weights of a neural network as an input and some mathematical expression"}, {"start": 303.91999999999996, "end": 308.4, "text": " And the output are your predictions of your neural net or the loss function. We'll see this in a bit"}, {"start": 309.03999999999996, "end": 313.28, "text": " But basically neural networks teaches us happen to be a certain class of mathematical expressions"}, {"start": 313.91999999999996, "end": 316.71999999999997, "text": " But back propagation is actually significantly more general"}, {"start": 316.72, "end": 321.76000000000005, "text": " It doesn't actually care about neural networks at all. It only tells us about arbitrary mathematical expressions"}, {"start": 322.08000000000004, "end": 326.08000000000004, "text": " And then we happen to use that machinery for training of neural networks"}, {"start": 326.40000000000003, "end": 332.0, "text": " Now one more note I would like to make at the stage is that as you see here micrograd is a scalar valued auto-grad engine"}, {"start": 332.32000000000005, "end": 336.48, "text": " So it's working on the you know level of individual scalers like negative four and two"}, {"start": 336.88000000000005, "end": 341.28000000000003, "text": " And we're taking neural nets and we're breaking them down all the way to these atoms of individual scalers"}, {"start": 341.36, "end": 344.32000000000005, "text": " And all the little pluses and times and it's just excessive"}, {"start": 344.32, "end": 347.68, "text": " And so obviously you would never be doing any of this in production"}, {"start": 348.0, "end": 353.2, "text": " It's really just for them for pedagogical reasons because it allows us to not have to deal with these and dimensional"}, {"start": 353.28, "end": 356.48, "text": " tensors that you would use in modern deep neural network library"}, {"start": 356.88, "end": 363.84, "text": " So this is really uh done so that you understand and refactor out back propagation and chain rule and understanding of your"}, {"start": 364.08, "end": 365.03999999999996, "text": " training"}, {"start": 365.03999999999996, "end": 367.44, "text": " And then if you actually want to train bigger networks"}, {"start": 367.6, "end": 371.92, "text": " You have to be using these tensors, but none of the math changes. This is done purely for efficiency"}, {"start": 371.92, "end": 373.92, "text": " We are basically taking scale value"}, {"start": 374.40000000000003, "end": 377.04, "text": " All the scale values we're packaging them up into tensors"}, {"start": 377.28000000000003, "end": 381.68, "text": " Which are just arrays of these scalers and then because we have these large arrays"}, {"start": 382.0, "end": 387.76, "text": " We're making operations on those large arrays that allows us to take advantage of the parallelism in a computer and"}, {"start": 388.16, "end": 391.92, "text": " All those operations can be done in parallel and then the whole thing runs faster"}, {"start": 392.32, "end": 395.12, "text": " But really none of the math changes and that done purely for efficiency"}, {"start": 395.52000000000004, "end": 399.04, "text": " So I don't think that it's pedagogically useful to be dealing with tensors from scratch"}, {"start": 399.04, "end": 403.84000000000003, "text": " Uh, and I think and that's why I fundamentally wrote micrograd because you can understand how things work"}, {"start": 404.24, "end": 407.36, "text": " Uh, at the fundamental level and then you can speed it up later"}, {"start": 408.08000000000004, "end": 413.52000000000004, "text": " Okay, so here's the fun part my claim is that micrograd is what you need to train your networks and everything else"}, {"start": 413.6, "end": 414.8, "text": " It's just efficiency"}, {"start": 414.88, "end": 420.48, "text": " So you'd think that micrograd would be a very complex piece of code and that turns out to not be the case"}, {"start": 421.12, "end": 426.64000000000004, "text": " So if we just go to micrograd and you will see that there's only two files here in micrograd"}, {"start": 426.64, "end": 429.91999999999996, "text": " This is the actual engine. It doesn't know anything about neural nets"}, {"start": 430.24, "end": 436.64, "text": " And this is the entire neural nets library on top of micrograd. So engine and nn.pi"}, {"start": 437.28, "end": 439.28, "text": " so the actual back propagation"}, {"start": 439.36, "end": 441.2, "text": " quadrgrad engine"}, {"start": 441.28, "end": 444.24, "text": " That gives you the power of neural networks is literally"}, {"start": 446.24, "end": 448.96, "text": " 100 lines of code of like very simple python"}, {"start": 450.08, "end": 453.28, "text": " Which we'll understand by the end of this lecture and then nn.pi"}, {"start": 453.28, "end": 457.11999999999995, "text": " This neural network library built on top of the autograd engine"}, {"start": 457.91999999999996, "end": 460.0, "text": " Um, is like a joke. It's like"}, {"start": 460.71999999999997, "end": 467.28, "text": " We have to define what is in neuron and then we have to define what is the layer of neurons and then we define what is a multilateral perceptron"}, {"start": 467.35999999999996, "end": 471.52, "text": " Which is just a sequence of layers of neurons and so it's just a total joke"}, {"start": 472.15999999999997, "end": 479.67999999999995, "text": " So basically um, there's a lot of power that comes from only 115 lines of code and that's only need to understand to understand"}, {"start": 479.68, "end": 485.2, "text": " You know, or training and everything else is just efficiency and of course there's a lot too efficiency"}, {"start": 485.76, "end": 487.76, "text": " But fundamentally that's all that's happening"}, {"start": 487.84000000000003, "end": 492.40000000000003, "text": " Okay, so now let's dive right in and implement micrograd step by step the first thing I'd like to do is"}, {"start": 492.40000000000003, "end": 494.72, "text": " I'd like to make sure that you have a very good understanding"}, {"start": 494.96000000000004, "end": 499.2, "text": " Intuitively of what a derivative is and exactly what information it gives you"}, {"start": 499.76, "end": 504.4, "text": " So let's start with some basic imports that I copy-based in every Jupyter notebook always"}, {"start": 505.44, "end": 507.44, "text": " And let's define the function"}, {"start": 507.44, "end": 510.88, "text": " scalar valid function f of x as follows"}, {"start": 511.52, "end": 513.12, "text": " So I just make this up randomly"}, {"start": 513.12, "end": 517.84, "text": " I just want to scale a valid function that takes a single scalar x and returns a single scalar y"}, {"start": 518.64, "end": 523.2, "text": " And we can call this function of course so we can pass it say 3.0 and get 20 back"}, {"start": 524.0, "end": 526.64, "text": " Now we can also plot this function to get a sense of its shape"}, {"start": 527.04, "end": 531.36, "text": " You can tell from the mathematical expression that this is probably a parabola it's quadratic"}, {"start": 531.92, "end": 535.28, "text": " And so if we just create a set of um"}, {"start": 535.28, "end": 536.8, "text": " um"}, {"start": 537.52, "end": 543.1999999999999, "text": " Skip skill the values that we can feed in using for example a range from negative 5 to 5 and steps up 0.25"}, {"start": 544.0799999999999, "end": 550.64, "text": " So this is so x is just from negative 5 to 5 not including 5 in steps of 0.25"}, {"start": 551.52, "end": 554.0799999999999, "text": " And we can actually call this function on this non-py array as well"}, {"start": 554.24, "end": 556.9599999999999, "text": " So we get a set of y's if we call f on x's and"}, {"start": 557.8399999999999, "end": 559.8399999999999, "text": " These y's are basically"}, {"start": 560.56, "end": 562.24, "text": " also applying"}, {"start": 562.24, "end": 567.6800000000001, "text": " function on every one of these elements independently and we can plot this using math plotlib"}, {"start": 568.0, "end": 571.84, "text": " So plt.plot x's and y's and we get nice parabola"}, {"start": 572.32, "end": 578.64, "text": " So previously here we fed in 3.0 somewhere here and we received 20 back which is here the y coordinate"}, {"start": 579.12, "end": 584.88, "text": " So now I'd like to think through what is the derivative of this function at any single input point x"}, {"start": 585.6800000000001, "end": 589.28, "text": " Right, so what is the derivative at different points x of this function"}, {"start": 589.28, "end": 592.88, "text": " Now if you remember back to your calculus class you've probably derived derivatives"}, {"start": 593.28, "end": 598.8, "text": " So we take this mathematical expression 3x square minus 4x plus 5 and you would write out on a piece of paper"}, {"start": 598.8, "end": 605.8399999999999, "text": " And you would you know apply the product rule and all the other rules and derive the mathematical expression of the great derivative of the original function"}, {"start": 606.0799999999999, "end": 608.88, "text": " And then you could plug in different taxes and see what the derivative is"}, {"start": 609.8399999999999, "end": 615.92, "text": " We're not going to actually do that because no one in neural networks actually writes out the expression for the neural net"}, {"start": 615.92, "end": 621.04, "text": " It would be a massive expression um it would be you know thousands since thousands of terms no one actually"}, {"start": 621.4399999999999, "end": 623.4399999999999, "text": " Derives that derivative of course"}, {"start": 623.52, "end": 626.3199999999999, "text": " And so we're not going to take this kind of like symbolic approach"}, {"start": 626.3199999999999, "end": 628.3199999999999, "text": " Instead what I'd like to do is I'd like to look at the"}, {"start": 628.3199999999999, "end": 632.4799999999999, "text": " Definition of derivative and just make sure that we really understand what derivative is measuring"}, {"start": 632.4799999999999, "end": 634.0799999999999, "text": " What is telling you about the function?"}, {"start": 634.88, "end": 636.88, "text": " And so if we just look up derivative"}, {"start": 642.4, "end": 643.68, "text": " We see that um"}, {"start": 643.68, "end": 648.0, "text": " Okay, so this is not a very good definition of derivative. This is a definition of what it means to be differentiable"}, {"start": 648.64, "end": 655.3599999999999, "text": " But if you remember from your calculus it is the limit sh goes to 0 of f of x plus h minus f of x over h"}, {"start": 655.8399999999999, "end": 659.52, "text": " So basically what it's saying is if you slightly bump up"}, {"start": 660.16, "end": 664.0799999999999, "text": " You're at some point x that you're interested in or hey and if you slightly bump up"}, {"start": 664.7199999999999, "end": 667.12, "text": " You know you slightly increase it by small number h"}, {"start": 668.16, "end": 671.1999999999999, "text": " How does the function respond with what sensitivity does it respond?"}, {"start": 671.2, "end": 678.1600000000001, "text": " Where's the slope at that point does the function go up or does it go down and by how much and that's the slope of that function"}, {"start": 678.1600000000001, "end": 683.2800000000001, "text": " The the slope of that response at that point and so we can basically evaluate"}, {"start": 684.1600000000001, "end": 690.48, "text": " The derivative here numerically by taking a very small h of course the definition would ask us to take h to zero"}, {"start": 690.88, "end": 693.6, "text": " We're just going to pick a very small h 0.001"}, {"start": 694.08, "end": 696.08, "text": " And let's say we're interested in 0.3.0"}, {"start": 696.24, "end": 698.32, "text": " So we can look at f of x of course is 20"}, {"start": 698.32, "end": 700.48, "text": " And now f of x plus h"}, {"start": 701.0400000000001, "end": 705.0400000000001, "text": " So if we slightly nudge x in a positive direction how is the function going to respond?"}, {"start": 705.7600000000001, "end": 714.0, "text": " And just looking at this do you expect you expect f of x plus h to be slightly greater than 20 or do you expect to be slightly lower than 20?"}, {"start": 714.72, "end": 720.72, "text": " And so since 3 is here and this is 20 if we slightly go positively the function will respond positively"}, {"start": 721.2800000000001, "end": 723.6, "text": " So you'd expect this to be slightly greater than 20"}, {"start": 723.6, "end": 730.96, "text": " And now by how much is telling you the sort of the strength of that slope right the size of the slope"}, {"start": 731.28, "end": 735.44, "text": " So f of x plus h of f of x this is how much the function responded"}, {"start": 736.08, "end": 742.96, "text": " in the positive direction and we have to normalize by the run so we have the rise over run to get the slope"}, {"start": 743.9200000000001, "end": 751.84, "text": " So this of course is just numerical approximation of the slope because we have to make a very very small to converge to the exact amount"}, {"start": 751.84, "end": 754.64, "text": " Now if I'm doing too many zeros"}, {"start": 755.52, "end": 760.24, "text": " At some point I'm going to get an incorrect answer because we're using floating point arithmetic"}, {"start": 760.5600000000001, "end": 766.0, "text": " And the representations of all these numbers in computer memory is finite and at some point we get into trouble"}, {"start": 766.5600000000001, "end": 768.96, "text": " So we can converge towards the right answer with this approach"}, {"start": 770.48, "end": 774.0, "text": " But basically at 3 the slope is 14"}, {"start": 774.64, "end": 779.84, "text": " And you can see that by taking 3x square minus 4x plus 5 and differentiating it in our head"}, {"start": 779.84, "end": 784.24, "text": " So 3x square would be 6x minus 4"}, {"start": 784.96, "end": 789.44, "text": " And then we plug in x equals 3 so that's 18 minus 4 is 14 so this is correct"}, {"start": 790.96, "end": 796.48, "text": " So that's at 3 now how about the slope at say negative 3"}, {"start": 797.44, "end": 802.72, "text": " Would you expect would you expect for the slope now telling the exact value is really hard"}, {"start": 802.8000000000001, "end": 804.8000000000001, "text": " But what is the sign of that slope?"}, {"start": 805.12, "end": 806.08, "text": " So at negative 3"}, {"start": 806.08, "end": 811.6800000000001, "text": " If we slightly go in the positive direction at x the function would actually go down"}, {"start": 812.0, "end": 813.84, "text": " And so that tells you that the slope would be negative"}, {"start": 814.0, "end": 815.5200000000001, "text": " So we'll get a slight number below"}, {"start": 816.88, "end": 820.32, "text": " Below 20 and so if we take the slope we expect something negative"}, {"start": 820.96, "end": 822.0, "text": " negative 22"}, {"start": 822.0, "end": 824.0, "text": " Okay"}, {"start": 824.0, "end": 826.96, "text": " And at some point here of course the slope would be 0"}, {"start": 827.2800000000001, "end": 831.6, "text": " Now for this specific function I looked it up previously and it's at point 2 over 3"}, {"start": 832.32, "end": 834.0, "text": " So at roughly 2 over 3"}, {"start": 834.0, "end": 835.52, "text": " But somewhere here"}, {"start": 837.04, "end": 838.72, "text": " This derivative would be 0"}, {"start": 839.28, "end": 841.28, "text": " So basically at that precise point"}, {"start": 843.52, "end": 844.32, "text": " Yeah"}, {"start": 844.32, "end": 848.32, "text": " At that precise point if we nudge in a positive direction the function doesn't respond"}, {"start": 848.32, "end": 851.12, "text": " This stays the same almost and so that's why the slope is 0"}, {"start": 851.52, "end": 853.52, "text": " Okay now let's look at a bit more complex case"}, {"start": 854.4, "end": 857.12, "text": " So we're going to start you know complexifying a bit"}, {"start": 857.36, "end": 859.36, "text": " So now we have a function"}, {"start": 859.76, "end": 861.04, "text": " Here"}, {"start": 861.04, "end": 862.8, "text": " With output variable d"}, {"start": 862.8, "end": 865.4399999999999, "text": " That is a function of 3 scalar inputs a b and c"}, {"start": 866.3199999999999, "end": 868.56, "text": " So a b and c are some specific values"}, {"start": 868.56, "end": 870.7199999999999, "text": " 3 inputs into our expression graph"}, {"start": 870.7199999999999, "end": 872.16, "text": " And a single output d"}, {"start": 872.9599999999999, "end": 875.52, "text": " And so if we just print d we get 4"}, {"start": 876.4799999999999, "end": 881.76, "text": " And now what I like to do is I'd like to again look at the derivative of d with respect to a b and c"}, {"start": 882.56, "end": 883.76, "text": " And think through"}, {"start": 884.56, "end": 886.8, "text": " Again just the intuition of what this derivative is telling us"}, {"start": 887.5999999999999, "end": 889.92, "text": " So in rooted evaluates derivative"}, {"start": 889.92, "end": 894.64, "text": " We're going to get a bit hacky here. We're going to again have a very small value of h"}, {"start": 895.52, "end": 899.76, "text": " And then we're gonna fix the inputs at some values that we're interested in"}, {"start": 900.4799999999999, "end": 904.88, "text": " So these are the this is the point a bc at which we're going to be evaluating the"}, {"start": 905.5999999999999, "end": 908.88, "text": " derivative of d with respect to all a b and c at that point"}, {"start": 909.8399999999999, "end": 913.12, "text": " So there's the inputs and now we have d1 is that expression"}, {"start": 913.8399999999999, "end": 917.04, "text": " And then we're going to for example look at the derivative of d with respect to a"}, {"start": 917.04, "end": 922.64, "text": " So we'll take a and we'll bump it by h and then we'll get d2 to be the exact same function"}, {"start": 923.92, "end": 925.92, "text": " And now we're going to print"}, {"start": 926.88, "end": 930.16, "text": " You know fun I want d1 is d1"}, {"start": 931.1999999999999, "end": 934.0, "text": " d2 is d2 and print slope"}, {"start": 935.28, "end": 938.48, "text": " So the derivative or slope here will be"}, {"start": 939.76, "end": 941.28, "text": " of course"}, {"start": 941.28, "end": 943.52, "text": " d2 minus d1 divided h"}, {"start": 943.52, "end": 947.92, "text": " So d2 minus d1 is how much the function increased"}, {"start": 949.12, "end": 951.12, "text": " when we bumped the"}, {"start": 952.0, "end": 955.1999999999999, "text": " The specific input that we're interested in by a tiny amount and"}, {"start": 956.0, "end": 960.16, "text": " This is the normalized by h to get the slope"}, {"start": 962.88, "end": 964.88, "text": " So"}, {"start": 965.28, "end": 969.68, "text": " Yeah, so this so I just run this we're going to print"}, {"start": 970.48, "end": 972.16, "text": " d1"}, {"start": 972.16, "end": 974.4, "text": " Which we know is for"}, {"start": 975.52, "end": 979.28, "text": " Now d2 will be bumped a will be bumped by h"}, {"start": 980.4, "end": 983.12, "text": " So let's just think through a little bit"}, {"start": 983.92, "end": 985.92, "text": " What d2 will be"}, {"start": 986.3199999999999, "end": 988.3199999999999, "text": " printed out here in particular"}, {"start": 989.36, "end": 991.36, "text": " d1 will be for"}, {"start": 991.28, "end": 996.8, "text": " Will d2 be a number slightly greater than 4 or slightly lower than 4 and it's going to tell us the"}, {"start": 997.4399999999999, "end": 999.4399999999999, "text": " the sign of the derivative"}, {"start": 999.44, "end": 1001.44, "text": " So"}, {"start": 1002.6400000000001, "end": 1004.6400000000001, "text": " We're bumping a by h"}, {"start": 1005.5200000000001, "end": 1007.6800000000001, "text": " b is minus 3 c is 10"}, {"start": 1008.72, "end": 1013.5200000000001, "text": " So you can just in total think through this derivative and what is doing a will be slightly more positive"}, {"start": 1014.96, "end": 1016.72, "text": " And but b is a negative number"}, {"start": 1017.6800000000001, "end": 1019.6800000000001, "text": " So if a is slightly more positive"}, {"start": 1020.48, "end": 1022.48, "text": " Because b is negative 3"}, {"start": 1023.44, "end": 1026.56, "text": " We're actually going to be adding less to d"}, {"start": 1026.56, "end": 1032.72, "text": " So you'd actually expect that the value of the function will go down"}, {"start": 1033.84, "end": 1035.84, "text": " So let's just see this"}, {"start": 1036.6399999999999, "end": 1040.0, "text": " Yeah, and so we went from 4 to 3.9996"}, {"start": 1040.96, "end": 1043.76, "text": " And that tells you that the slope will be negative and then"}, {"start": 1044.48, "end": 1050.6399999999999, "text": " Uh will be a negative number because we went down and then the exact number of slope will be"}, {"start": 1051.2, "end": 1052.8799999999999, "text": " exact amount of slope is negative 3"}, {"start": 1052.88, "end": 1056.5600000000002, "text": " And you can also convince yourself that negative 3 is the right answer"}, {"start": 1057.0400000000002, "end": 1058.8000000000002, "text": " mathematically and analytically"}, {"start": 1058.88, "end": 1062.64, "text": " Because if you have 8 times b plus c and you are you know you have calculus"}, {"start": 1063.2, "end": 1067.7600000000002, "text": " Then differentiating 8 times b plus c with respect to a gives you just b"}, {"start": 1068.64, "end": 1073.8400000000001, "text": " And indeed the value of b is negative 3 which is the derivative that we have so you can tell that that's correct"}, {"start": 1075.1200000000001, "end": 1076.64, "text": " So now if we do this with b"}, {"start": 1077.8400000000001, "end": 1080.64, "text": " So if we bump b by a little bit in a positive direction"}, {"start": 1080.64, "end": 1085.44, "text": " We'd get different slopes. So what is the influence of b on the output d"}, {"start": 1086.48, "end": 1090.72, "text": " So if we bump b by tiny amount in the positive direction then because a is positive"}, {"start": 1091.68, "end": 1093.2, "text": " We'll be adding more to d"}, {"start": 1093.92, "end": 1094.8000000000002, "text": " Right"}, {"start": 1094.8000000000002, "end": 1099.3600000000001, "text": " So um, and now what is the what is the sensitivity? What is the slope of that addition?"}, {"start": 1100.0, "end": 1102.0, "text": " And it might not surprise you that this should be"}, {"start": 1103.0400000000002, "end": 1104.3200000000002, "text": " 2"}, {"start": 1104.3200000000002, "end": 1108.64, "text": " And why is it 2 because d of d by db"}, {"start": 1108.64, "end": 1113.44, "text": " The differentiating respect to b would be would give us a and the value of a is 2"}, {"start": 1113.6000000000001, "end": 1115.6000000000001, "text": " So that's also working all"}, {"start": 1115.6000000000001, "end": 1119.1200000000001, "text": " And then if c gets bumped a tiny amount in h by h"}, {"start": 1120.0800000000002, "end": 1124.24, "text": " Then of course 8 times b is unaffected and now c becomes slightly bit higher"}, {"start": 1124.48, "end": 1125.8400000000001, "text": " What does that do to the function?"}, {"start": 1125.8400000000001, "end": 1128.4, "text": " It makes it slightly bit higher because we're simply adding c"}, {"start": 1128.96, "end": 1133.1200000000001, "text": " And it makes it slightly bit higher by the exact same amount that we added to c"}, {"start": 1133.2800000000002, "end": 1135.5200000000002, "text": " And so that tells you that the slope is 1"}, {"start": 1135.52, "end": 1138.16, "text": " That will be the um"}, {"start": 1139.28, "end": 1143.12, "text": " The rate at which d will increase as we scale"}, {"start": 1144.24, "end": 1145.28, "text": " c"}, {"start": 1145.28, "end": 1149.12, "text": " Okay, so we now have some intuitive sense of what this derivative is telling you about the function"}, {"start": 1149.44, "end": 1151.12, "text": " And we'd like to move to neural networks"}, {"start": 1151.12, "end": 1154.8799999999999, "text": " Now as I mentioned neural networks will be pretty massive expressions mathematical expressions"}, {"start": 1155.2, "end": 1159.68, "text": " So we need some data structures that maintain these expressions and that's what we're going to start to build out now"}, {"start": 1160.6399999999999, "end": 1162.32, "text": " So we're going to"}, {"start": 1162.32, "end": 1166.8799999999999, "text": " Build out this value object that I showed you in the read me page of micrograd"}, {"start": 1167.6, "end": 1172.56, "text": " So let me copy paste a skeleton of the first very simple value object"}, {"start": 1173.6, "end": 1175.6, "text": " So class value takes a single"}, {"start": 1176.56, "end": 1180.56, "text": " scalar value that it wraps and keeps track of and that's it"}, {"start": 1180.72, "end": 1184.1599999999999, "text": " So we can for example do value of 2.0 and then we can"}, {"start": 1185.52, "end": 1188.1599999999999, "text": " Get we can look at its content and"}, {"start": 1188.16, "end": 1193.76, "text": " um python will internally use the wrapper function to return"}, {"start": 1194.48, "end": 1196.48, "text": " uh this straight clips"}, {"start": 1196.96, "end": 1198.72, "text": " like that"}, {"start": 1198.72, "end": 1202.48, "text": " So this is a value object with data equals to the overcreating here"}, {"start": 1203.3600000000001, "end": 1205.6000000000001, "text": " Now what we'd like to do is like we'd like to be able to"}, {"start": 1207.6000000000001, "end": 1209.6000000000001, "text": " Have not just like two values"}, {"start": 1210.16, "end": 1212.88, "text": " But we'd like to do a blocky right we'd like to add them"}, {"start": 1213.8400000000001, "end": 1217.3600000000001, "text": " So currently you would get an error because python doesn't know how to add"}, {"start": 1217.36, "end": 1220.08, "text": " Two value objects. So we have to tell it"}, {"start": 1221.52, "end": 1223.52, "text": " So here's an addition"}, {"start": 1226.3999999999999, "end": 1232.56, "text": " So you have to basically use these special double underscore methods in python to define these operators for these objects"}, {"start": 1233.12, "end": 1234.6399999999999, "text": " so if we call um"}, {"start": 1235.52, "end": 1236.7199999999998, "text": " the"}, {"start": 1236.7199999999998, "end": 1239.1999999999998, "text": " If we use this plus operator"}, {"start": 1239.1999999999998, "end": 1241.6799999999998, "text": " python will internally call a dot"}, {"start": 1241.68, "end": 1247.3600000000001, "text": " Add of b that's what will happen internally and so b will be the other"}, {"start": 1248.24, "end": 1250.4, "text": " And self will be a"}, {"start": 1251.1200000000001, "end": 1254.0800000000002, "text": " And so we see that what we're going to return is a new value object"}, {"start": 1254.0800000000002, "end": 1259.2, "text": " And it's just uh is it going to be wrapping the plus of their data"}, {"start": 1259.92, "end": 1264.16, "text": " But remember now because uh data is the actual like numbered python number"}, {"start": 1264.24, "end": 1269.04, "text": " So this operator here is just the typical floating point plus addition now"}, {"start": 1269.04, "end": 1273.28, "text": " It's not an addition of value objects and will return a new value"}, {"start": 1273.76, "end": 1276.72, "text": " So now a plus b should work and it should print value of"}, {"start": 1277.28, "end": 1278.48, "text": " Negative one"}, {"start": 1278.48, "end": 1280.48, "text": " Because that's two plus minus three"}, {"start": 1280.48, "end": 1281.76, "text": " There we go"}, {"start": 1281.76, "end": 1283.6, "text": " Okay, let's now implement multiply"}, {"start": 1284.32, "end": 1286.32, "text": " Just so we can recreate this expression here"}, {"start": 1286.96, "end": 1290.32, "text": " So multiply I think it won't surprise you will be fairly similar"}, {"start": 1291.76, "end": 1293.84, "text": " So instead of add we're going to be using mul"}, {"start": 1294.3999999999999, "end": 1296.3999999999999, "text": " And then here of course we want to do times"}, {"start": 1296.4, "end": 1300.16, "text": " And so now we can create a C value object which will be 10.0"}, {"start": 1300.8000000000002, "end": 1302.8000000000002, "text": " And now we should be able to do a times b"}, {"start": 1304.0, "end": 1306.0, "text": " Well, let's just do a times b first"}, {"start": 1306.88, "end": 1308.64, "text": " um"}, {"start": 1308.64, "end": 1310.64, "text": " That's value of negative six now"}, {"start": 1310.88, "end": 1314.96, "text": " And by the way I skipped over this a little bit uh suppose that I didn't have the wrapper function here"}, {"start": 1315.6000000000001, "end": 1318.5600000000002, "text": " Then it's just that you'll get some kind of an ugly expression"}, {"start": 1318.96, "end": 1324.5600000000002, "text": " So what wrapper is doing is it's providing us a way to print out like a nicer looking expression in python"}, {"start": 1324.56, "end": 1331.2, "text": " Uh, so we don't just have something cryptic. We actually are you know, it's value of negative six"}, {"start": 1332.32, "end": 1340.24, "text": " So this gives us a times and then this we should now be able to add C to it because we've defined and told the python how to do mul and add"}, {"start": 1340.72, "end": 1344.32, "text": " And so this will call this will basically be equivalent to a dot"}, {"start": 1345.2, "end": 1347.2, "text": " mul"}, {"start": 1347.2, "end": 1351.9199999999998, "text": " B and then this new value object will be dot add of C"}, {"start": 1351.92, "end": 1353.92, "text": " And so let's see if that worked"}, {"start": 1354.88, "end": 1358.48, "text": " Yep, so that worked well that gave us four which is what we expect from before"}, {"start": 1359.28, "end": 1362.88, "text": " And I believe you can just call the manually as well. There we go. So"}, {"start": 1364.4, "end": 1368.72, "text": " Yeah, okay, so now what we are missing is the connected tissue of this expression"}, {"start": 1369.04, "end": 1371.3600000000001, "text": " As I mentioned we want to keep these expression graphs"}, {"start": 1371.68, "end": 1376.0, "text": " So we need to know and keep pointers about what values produce what other values"}, {"start": 1376.96, "end": 1380.8000000000002, "text": " So here for example, we are going to introduce a new variable which will call children"}, {"start": 1380.8, "end": 1382.8, "text": " And by default it will be an empty tuple"}, {"start": 1383.6, "end": 1387.9199999999998, "text": " And then we're actually going to keep a slightly different variable in the class which will call underscore prime"}, {"start": 1388.48, "end": 1390.48, "text": " Which will be the set of children"}, {"start": 1391.44, "end": 1395.52, "text": " Uh, this is how I done. I did it in the original micro grad looking at my code here"}, {"start": 1395.84, "end": 1398.48, "text": " I can't remember exactly the reason I believe it was efficiency"}, {"start": 1398.8, "end": 1401.68, "text": " But this underscore children will be a tuple for convenience"}, {"start": 1401.84, "end": 1404.56, "text": " But then when we actually maintain it in the class it will be just this set"}, {"start": 1405.04, "end": 1407.04, "text": " Yes, I believe for efficiency"}, {"start": 1407.76, "end": 1408.96, "text": " um"}, {"start": 1408.96, "end": 1415.52, "text": " So now when we are creating a value like this with a constructor children will be empty and prep will be the empty set"}, {"start": 1416.08, "end": 1418.88, "text": " But when we're creating a value through addition or multiplication"}, {"start": 1419.1200000000001, "end": 1424.8, "text": " We're going to feed in the children of this value which in this case is self and other"}, {"start": 1426.56, "end": 1428.56, "text": " So those are the children here"}, {"start": 1430.8, "end": 1435.3600000000001, "text": " So now we can do d dot prep and we'll see that the children of the"}, {"start": 1435.36, "end": 1443.04, "text": " We now know are this a value of negative six and value of ten and this of course is the value resulting from a times b"}, {"start": 1443.52, "end": 1445.52, "text": " And the c value which is ten"}, {"start": 1446.7199999999998, "end": 1451.76, "text": " Now the last piece of information we don't know so we know now that the children of every single value"}, {"start": 1452.0, "end": 1454.24, "text": " We don't know what operation created this value"}, {"start": 1454.8799999999999, "end": 1457.76, "text": " So we need one more element here. Let's call it underscore pop"}, {"start": 1459.28, "end": 1461.6799999999998, "text": " And by default this is the empty set for leaves"}, {"start": 1461.68, "end": 1464.72, "text": " And then we'll just maintain it here"}, {"start": 1465.76, "end": 1472.64, "text": " And now the operation will be just a simple string and in the case of addition it's plus in the case of multiplication it's times"}, {"start": 1474.0, "end": 1475.44, "text": " So now we"}, {"start": 1475.44, "end": 1478.16, "text": " Not just have d dot prep we also have a d dot up"}, {"start": 1478.8, "end": 1482.0, "text": " And we know that d was produced by an addition of those two values"}, {"start": 1482.5600000000002, "end": 1485.1200000000001, "text": " And so now we have the full mathematical expression"}, {"start": 1485.8400000000001, "end": 1489.68, "text": " And we're building out this data structure and we know exactly how each value came to be"}, {"start": 1489.68, "end": 1492.0800000000002, "text": " By word expression and from what other values"}, {"start": 1494.8, "end": 1497.3600000000001, "text": " Now because these expressions are about to get quite a bit larger"}, {"start": 1497.76, "end": 1501.76, "text": " We'd like a way to nicely visualize these expressions that we're building out"}, {"start": 1502.16, "end": 1505.52, "text": " So for that I'm going to copy paste a bunch of slightly scary code"}, {"start": 1506.0800000000002, "end": 1509.1200000000001, "text": " That's going to visualize this this expression graphs for us"}, {"start": 1509.68, "end": 1511.6000000000001, "text": " So here's the code and I'll explain it in a bit"}, {"start": 1512.0800000000002, "end": 1514.3200000000002, "text": " But first let me just show you what this code does"}, {"start": 1514.32, "end": 1519.9199999999998, "text": " Basically what it does is it creates a new function draw dot that we can call on some root node"}, {"start": 1520.8799999999999, "end": 1522.6399999999999, "text": " And then it's going to visualize it"}, {"start": 1522.6399999999999, "end": 1527.9199999999998, "text": " So if we call draw dot on d which is this final value here that is a times b plus c"}, {"start": 1529.84, "end": 1534.08, "text": " It creates something like this. So this is d and you see that this is a times b"}, {"start": 1534.72, "end": 1539.04, "text": " Create a value plus c gives us this output node d"}, {"start": 1539.04, "end": 1545.2, "text": " So that's draw dot of d and I'm not going to go through this in complete detail"}, {"start": 1545.52, "end": 1548.0, "text": " You can take a look at graph is and it's api"}, {"start": 1548.0, "end": 1550.8799999999999, "text": " A graph is an open source graph visualization software"}, {"start": 1551.52, "end": 1555.12, "text": " And what we're doing here is we're building out this graph in the graph is api"}, {"start": 1555.92, "end": 1562.24, "text": " And you can basically see that trace is this helper function that enumerates all the nodes and edges in the graph"}, {"start": 1562.8799999999999, "end": 1565.28, "text": " So that's just built a set of all the nodes and edges"}, {"start": 1565.28, "end": 1569.84, "text": " And then we iterate through all the nodes and we create special node objects for them in"}, {"start": 1571.44, "end": 1573.2, "text": " using dot node"}, {"start": 1573.28, "end": 1576.24, "text": " And then we also create edges using dot dot edge"}, {"start": 1576.96, "end": 1582.16, "text": " And the only thing that's like slightly tricky here is you notice that I basically add these fake nodes"}, {"start": 1582.16, "end": 1587.04, "text": " Which are these operation nodes. So for example this node here is just like a plus node"}, {"start": 1587.76, "end": 1590.0, "text": " and I create these"}, {"start": 1590.0, "end": 1596.4, "text": " These special op nodes here and I connect them accordingly"}, {"start": 1596.96, "end": 1601.2, "text": " So these nodes of course are not actual nodes in the original graph"}, {"start": 1601.84, "end": 1606.96, "text": " They're not actually a value object. The only value objects here are the things in squares"}, {"start": 1607.28, "end": 1610.08, "text": " Those are actual value objects or representations thereof"}, {"start": 1610.56, "end": 1614.8, "text": " And these op nodes are just created in this draw dot routine so that it looks nice"}, {"start": 1614.8, "end": 1622.1599999999999, "text": " Let's also add labels to these graphs just so we know what variables are where so let's create a special underscore label"}, {"start": 1622.96, "end": 1628.96, "text": " Um, or let's just do label equals ft by default and save it to each node"}, {"start": 1630.24, "end": 1638.72, "text": " And then here we're going to do label is a label is the label is c"}, {"start": 1640.6399999999999, "end": 1642.6399999999999, "text": " um"}, {"start": 1642.64, "end": 1644.64, "text": " And then"}, {"start": 1644.88, "end": 1646.88, "text": " Let's create a special um"}, {"start": 1647.8400000000001, "end": 1649.8400000000001, "text": " E equals a times b"}, {"start": 1650.88, "end": 1652.88, "text": " And dot label will be"}, {"start": 1654.24, "end": 1655.76, "text": " It's kind of notty"}, {"start": 1655.76, "end": 1657.3600000000001, "text": " And E will be E plus C"}, {"start": 1658.24, "end": 1660.0800000000002, "text": " And a d dot label will be"}, {"start": 1660.88, "end": 1662.88, "text": " B"}, {"start": 1662.72, "end": 1666.0, "text": " Okay, so nothing really changes. I just added this new E function"}, {"start": 1666.64, "end": 1668.64, "text": " A new E variable"}, {"start": 1668.5600000000002, "end": 1671.3600000000001, "text": " And then here when we are printing this"}, {"start": 1671.36, "end": 1678.3999999999999, "text": " I'm going to print the label here. So this will be a percent s bar and this will be end dot label"}, {"start": 1681.36, "end": 1683.36, "text": " And so now"}, {"start": 1683.52, "end": 1688.32, "text": " We have the label on the left here. So it says a b creating e and then E plus C creates d"}, {"start": 1688.8, "end": 1690.8, "text": " Just like we have it here"}, {"start": 1690.9599999999998, "end": 1693.6, "text": " And finally, let's make this expression just one layer deeper"}, {"start": 1694.4799999999998, "end": 1696.4799999999998, "text": " So d will not be the final output node"}, {"start": 1696.48, "end": 1701.04, "text": " Instead after d we are going to create a new value object"}, {"start": 1701.92, "end": 1706.56, "text": " Called f we're going to start running out of variable soon f will be negative 2.0"}, {"start": 1707.44, "end": 1709.44, "text": " And it's label will of course just the f"}, {"start": 1710.64, "end": 1714.88, "text": " And then l will capital L will be the output of our graph"}, {"start": 1715.52, "end": 1717.04, "text": " And l will be p times f"}, {"start": 1718.08, "end": 1720.8, "text": " Okay, so l will be negative eight is the output"}, {"start": 1722.16, "end": 1724.16, "text": " Uh, so"}, {"start": 1724.16, "end": 1727.68, "text": " Now we don't just draw a d draw L"}, {"start": 1730.0800000000002, "end": 1732.0800000000002, "text": " Okay"}, {"start": 1732.0800000000002, "end": 1737.44, "text": " And somehow the label of L is undefined loops. I'll that label as to be explicitly"}, {"start": 1737.76, "end": 1739.76, "text": " So given to it"}, {"start": 1739.76, "end": 1741.76, "text": " There we go. So l is the output"}, {"start": 1741.8400000000001, "end": 1744.16, "text": " So let's quickly recap what we've done so far"}, {"start": 1744.24, "end": 1748.3200000000002, "text": " We are able to build out mathematical expressions using only plus and times so far"}, {"start": 1748.32, "end": 1753.6, "text": " Uh, they are scalar valued along the way and we can do this forward pass"}, {"start": 1753.6, "end": 1756.0, "text": " And build out a mathematical expression"}, {"start": 1756.48, "end": 1759.12, "text": " So we have multiple inputs here a b c and f"}, {"start": 1759.6, "end": 1763.4399999999998, "text": " Going into a mathematical expression that produces a single output L"}, {"start": 1764.08, "end": 1766.8, "text": " And this here is vis-visualizing the forward pass"}, {"start": 1767.36, "end": 1770.72, "text": " So the output of the forward pass is negative eight. That's the value"}, {"start": 1771.76, "end": 1774.8, "text": " Now what we'd like to do next is we'd like to run back propagation"}, {"start": 1774.8, "end": 1778.24, "text": " And in back propagation we are going to start here at the end"}, {"start": 1778.72, "end": 1780.72, "text": " And we're going to reverse"}, {"start": 1781.04, "end": 1784.96, "text": " And calculate the gradient along along all these intermediate values"}, {"start": 1785.6, "end": 1788.32, "text": " And really what we're computing for every single value here"}, {"start": 1788.96, "end": 1794.3999999999999, "text": " Um, we're going to compute the derivative of that node with respect to L"}, {"start": 1795.52, "end": 1799.76, "text": " So the derivative of l with respect to l is just uh one"}, {"start": 1799.76, "end": 1804.72, "text": " And then we're going to derive what is the derivative of l with respect to f with respect to d"}, {"start": 1805.12, "end": 1807.12, "text": " With respect to c with respect to e"}, {"start": 1807.6, "end": 1809.6, "text": " With respect to b and with respect to a"}, {"start": 1810.16, "end": 1811.76, "text": " And in neural network setting"}, {"start": 1812.0, "end": 1816.0, "text": " It'd be very interested in the derivative of basically this loss function l"}, {"start": 1816.8, "end": 1818.8799999999999, "text": " With respect to the weights of a neural network"}, {"start": 1819.44, "end": 1822.16, "text": " And here of course we have just these variables a b c and f"}, {"start": 1822.48, "end": 1825.76, "text": " But some of these will eventually represent the weights of a neural net"}, {"start": 1825.92, "end": 1828.64, "text": " And so we'll need to know how those weights are impacting"}, {"start": 1828.64, "end": 1830.0800000000002, "text": " The loss function"}, {"start": 1830.4, "end": 1832.88, "text": " So we'll be interested basically in the derivative of the output"}, {"start": 1833.0400000000002, "end": 1835.2800000000002, "text": " With respect to some of its leaf nodes"}, {"start": 1835.44, "end": 1837.6000000000001, "text": " And those leaf nodes will be the weights of the neural net"}, {"start": 1838.16, "end": 1840.8000000000002, "text": " And the other leaf nodes of course will be the data itself"}, {"start": 1840.96, "end": 1845.8400000000001, "text": " But usually we will not want or use the derivative of the loss function with respect to data"}, {"start": 1845.92, "end": 1847.2, "text": " Because the data is fixed"}, {"start": 1847.44, "end": 1849.8400000000001, "text": " But the weights will be iterated on"}, {"start": 1850.72, "end": 1852.24, "text": " Using the gradient information"}, {"start": 1852.24, "end": 1855.44, "text": " So next we are going to create a variable inside the value class"}, {"start": 1855.44, "end": 1860.4, "text": " That maintains the derivative of l with respect to that value"}, {"start": 1861.1200000000001, "end": 1863.1200000000001, "text": " And we will call this variable grad"}, {"start": 1863.92, "end": 1866.72, "text": " So there is a dot data and there's a self-adgrad"}, {"start": 1867.3600000000001, "end": 1869.3600000000001, "text": " And initially it will be zero"}, {"start": 1869.52, "end": 1872.88, "text": " And remember that zero is basically means no effect"}, {"start": 1873.04, "end": 1876.56, "text": " So at initialization we're assuming that every value does not impact"}, {"start": 1876.64, "end": 1879.04, "text": " Does not affect the output"}, {"start": 1879.68, "end": 1881.44, "text": " Right because if the gradient is zero"}, {"start": 1881.44, "end": 1885.6000000000001, "text": " That means that changing this variable is not changing the loss function"}, {"start": 1885.8400000000001, "end": 1888.24, "text": " So by default we assume that the gradient is zero"}, {"start": 1888.96, "end": 1894.24, "text": " And then now that we have grad and it's 0.0"}, {"start": 1896.72, "end": 1899.76, "text": " We are going to be able to visualize it here after data"}, {"start": 1899.76, "end": 1901.52, "text": " So here grad is 0.4f"}, {"start": 1902.88, "end": 1904.4, "text": " And this will be in that grad"}, {"start": 1905.8400000000001, "end": 1909.44, "text": " And now we are going to be showing both the data and the grad"}, {"start": 1909.44, "end": 1912.0800000000002, "text": " And initialize that zero"}, {"start": 1913.8400000000001, "end": 1916.88, "text": " And we are just about getting ready to calculate the back propagation"}, {"start": 1917.44, "end": 1919.28, "text": " And of course this grad again as I mentioned"}, {"start": 1919.28, "end": 1922.0800000000002, "text": " Is representing the derivative of the output"}, {"start": 1922.0800000000002, "end": 1924.88, "text": " In this case l with respect to this value"}, {"start": 1925.04, "end": 1926.0800000000002, "text": " So with respect to"}, {"start": 1926.0800000000002, "end": 1928.56, "text": " So this is the derivative of l with respect to f"}, {"start": 1928.56, "end": 1930.0800000000002, "text": " Respect to d and so on"}, {"start": 1930.56, "end": 1932.24, "text": " So let's now fill in those gradients"}, {"start": 1932.24, "end": 1934.3200000000002, "text": " And actually do back propagation manually"}, {"start": 1934.3200000000002, "end": 1935.92, "text": " So let's start filling in these gradients"}, {"start": 1935.92, "end": 1937.92, "text": " And start all the way at the end as I mentioned here"}, {"start": 1937.92, "end": 1941.3600000000001, "text": " First we are interested to fill in this gradient here"}, {"start": 1941.92, "end": 1944.5600000000002, "text": " So what is the derivative of l with respect to l"}, {"start": 1945.3600000000001, "end": 1948.0, "text": " In other words if I change l by a tiny amount of h"}, {"start": 1949.2, "end": 1951.44, "text": " How much does l change?"}, {"start": 1952.48, "end": 1953.92, "text": " It changes by h"}, {"start": 1953.92, "end": 1956.5600000000002, "text": " So it's proportional and therefore the derivative will be 1"}, {"start": 1957.44, "end": 1961.1200000000001, "text": " We can of course measure these or estimate these numerical gradients"}, {"start": 1961.1200000000001, "end": 1963.28, "text": " Numerically just like we've seen before"}, {"start": 1963.28, "end": 1964.8000000000002, "text": " So if I take this expression"}, {"start": 1964.8, "end": 1968.32, "text": " And I create a def lol function here"}, {"start": 1969.36, "end": 1970.56, "text": " And put this here"}, {"start": 1970.56, "end": 1973.52, "text": " Now the reason I'm creating a gating function lol here"}, {"start": 1973.52, "end": 1976.72, "text": " Is because I don't want to pollute or mess up the global scope here"}, {"start": 1976.72, "end": 1978.8799999999999, "text": " This is just kind of like a little staging area"}, {"start": 1978.8799999999999, "end": 1982.48, "text": " And as you know in Python all of these will be local variables to this function"}, {"start": 1982.48, "end": 1984.96, "text": " So I'm not changing any of the global scope here"}, {"start": 1985.84, "end": 1987.6, "text": " So here l1 will be l"}, {"start": 1990.08, "end": 1992.32, "text": " And then copy based on this expression"}, {"start": 1992.32, "end": 1995.04, "text": " We're going to add a small amount h"}, {"start": 1997.4399999999998, "end": 1998.96, "text": " In for example a"}, {"start": 2000.6399999999999, "end": 2004.6399999999999, "text": " Right and this will be measuring the derivative of l with respect to a"}, {"start": 2005.52, "end": 2007.12, "text": " So here this will be l2"}, {"start": 2008.24, "end": 2010.24, "text": " And then we want to print that derivative"}, {"start": 2010.24, "end": 2012.8799999999999, "text": " So print l2 minus l1"}, {"start": 2012.8799999999999, "end": 2014.8799999999999, "text": " Which is how much l changed"}, {"start": 2015.28, "end": 2017.28, "text": " And then normalize it by h"}, {"start": 2017.4399999999998, "end": 2018.96, "text": " So this is the rise over run"}, {"start": 2018.96, "end": 2022.32, "text": " And we have to be careful because l is a valid node"}, {"start": 2022.32, "end": 2023.8400000000001, "text": " So we actually want its data"}, {"start": 2025.28, "end": 2025.52, "text": " Um"}, {"start": 2026.4, "end": 2029.04, "text": " So that these are floats dividing by h"}, {"start": 2029.04, "end": 2032.32, "text": " And this should print the derivative of l with respect to a"}, {"start": 2032.32, "end": 2035.04, "text": " Because a is the one that we bumped a little bit by h"}, {"start": 2035.76, "end": 2038.88, "text": " So what is the derivative of l with respect to a"}, {"start": 2039.44, "end": 2040.24, "text": " It's 6"}, {"start": 2041.1200000000001, "end": 2041.8400000000001, "text": " Okay"}, {"start": 2041.8400000000001, "end": 2042.64, "text": " And obviously"}, {"start": 2043.68, "end": 2045.92, "text": " If we change l by h"}, {"start": 2045.92, "end": 2048.4, "text": " Then that would be"}, {"start": 2050.0, "end": 2051.52, "text": " Here effectively"}, {"start": 2051.52, "end": 2052.32, "text": " Um"}, {"start": 2052.32, "end": 2055.2000000000003, "text": " This looks really awkward but changing l by h"}, {"start": 2056.2400000000002, "end": 2057.76, "text": " You see the derivative here is one"}, {"start": 2058.88, "end": 2060.08, "text": " Um"}, {"start": 2060.96, "end": 2064.2400000000002, "text": " That's kind of like the base case of what we are doing here"}, {"start": 2064.88, "end": 2066.64, "text": " So basically we come out comp here"}, {"start": 2066.64, "end": 2069.76, "text": " And we can manually set l dot grad to one"}, {"start": 2069.76, "end": 2071.2000000000003, "text": " This is our manual back propagation"}, {"start": 2072.08, "end": 2073.36, "text": " l dot grad is one"}, {"start": 2073.36, "end": 2074.32, "text": " And let's redraw"}, {"start": 2074.32, "end": 2077.36, "text": " And we'll see that we filled in"}, {"start": 2077.36, "end": 2078.56, "text": " Grad is one for l"}, {"start": 2079.2000000000003, "end": 2081.28, "text": " We're now going to continue the back propagation"}, {"start": 2081.28, "end": 2084.7200000000003, "text": " So let's here look at the derivatives of l with respect to d and f"}, {"start": 2085.44, "end": 2087.2000000000003, "text": " Uh let's do a d first"}, {"start": 2087.84, "end": 2090.56, "text": " So what we are interested in if I create a mark down on here"}, {"start": 2090.56, "end": 2091.44, "text": " Is we'd like to know"}, {"start": 2092.0, "end": 2093.84, "text": " Basically we have that l is d times f"}, {"start": 2094.4, "end": 2099.04, "text": " And we'd like to know what is uh dl by dd"}, {"start": 2100.32, "end": 2101.1200000000003, "text": " What is that?"}, {"start": 2101.12, "end": 2104.16, "text": " And if you know you're a calculus uh l is d times f"}, {"start": 2104.16, "end": 2106.08, "text": " So what is dl by dd?"}, {"start": 2106.08, "end": 2107.2, "text": " It would be f"}, {"start": 2108.16, "end": 2109.7599999999998, "text": " And if you don't believe me"}, {"start": 2109.7599999999998, "end": 2112.88, "text": " We can also just derive it because the proof would be fairly straightforward"}, {"start": 2113.44, "end": 2116.7999999999997, "text": " Uh we go to the definition of the"}, {"start": 2116.7999999999997, "end": 2117.7599999999998, "text": " A derivative"}, {"start": 2117.7599999999998, "end": 2120.88, "text": " Which is f of x plus h minus f of x divide h"}, {"start": 2122.24, "end": 2122.96, "text": " As a limit"}, {"start": 2122.96, "end": 2125.3599999999997, "text": " Limit of h goes to zero of this kind of expression"}, {"start": 2126.0, "end": 2127.52, "text": " So when we have l is d times f"}, {"start": 2127.52, "end": 2134.56, "text": " Then increasing d by h would give us the output of d plus h times f"}, {"start": 2135.84, "end": 2137.6, "text": " That's basically a full of x plus h, right"}, {"start": 2139.04, "end": 2140.56, "text": " minus d times f"}, {"start": 2142.4, "end": 2143.36, "text": " And then divide h"}, {"start": 2143.92, "end": 2146.0, "text": " And symbolically expanding out here"}, {"start": 2146.0, "end": 2150.96, "text": " We would have basically d times f plus h times f minus d times f"}, {"start": 2151.2, "end": 2151.7599999999998, "text": " divide h"}, {"start": 2152.4, "end": 2154.96, "text": " And then you see how the df minus df cancels"}, {"start": 2154.96, "end": 2157.52, "text": " So you're left with h times f"}, {"start": 2157.52, "end": 2158.32, "text": " divide h"}, {"start": 2158.7200000000003, "end": 2159.28, "text": " Which is f"}, {"start": 2160.08, "end": 2162.48, "text": " So in the limit as h goes to zero of"}, {"start": 2163.36, "end": 2164.56, "text": " You know"}, {"start": 2164.56, "end": 2166.88, "text": " derivative um"}, {"start": 2166.88, "end": 2167.84, "text": " definition"}, {"start": 2167.84, "end": 2170.64, "text": " We just get f in a case of d times f"}, {"start": 2172.4, "end": 2177.36, "text": " So symmetrically dl by d f will just be d"}, {"start": 2178.64, "end": 2180.88, "text": " So what we have is that f dot grad"}, {"start": 2181.44, "end": 2184.56, "text": " We see now is just the value of d"}, {"start": 2184.56, "end": 2186.24, "text": " Which is four"}, {"start": 2188.88, "end": 2193.68, "text": " And we see that d dot grad is just uh the value of f"}, {"start": 2196.96, "end": 2200.08, "text": " And so the value of f is negative two"}, {"start": 2201.36, "end": 2203.36, "text": " So we'll set those manually"}, {"start": 2205.2799999999997, "end": 2208.48, "text": " Let me erase this markdown node and then let's redraw what we have"}, {"start": 2210.96, "end": 2213.84, "text": " Okay, and let's just make sure that these were correct"}, {"start": 2213.84, "end": 2218.32, "text": " So we seem to think that dl by dd is negative two. So let's double check"}, {"start": 2219.6000000000004, "end": 2224.2400000000002, "text": " Um, let me erase this plus h from before and now we want the derivative with respect to f"}, {"start": 2225.2000000000003, "end": 2228.32, "text": " So let's just come here when I create f and let's do a plus h here"}, {"start": 2228.8, "end": 2232.88, "text": " And they should print a derivative of l with respect to f. So we expect to see four"}, {"start": 2234.2400000000002, "end": 2237.76, "text": " Yeah, and this is four up to floating point funquiness"}, {"start": 2239.04, "end": 2242.4, "text": " And then dl by dd should be f"}, {"start": 2242.4, "end": 2244.1600000000003, "text": " Which is negative two"}, {"start": 2245.12, "end": 2246.8, "text": " grad is negative two"}, {"start": 2246.8, "end": 2249.6, "text": " So if we again come here and we change d"}, {"start": 2251.92, "end": 2254.48, "text": " d dot d dot plus equals h right here"}, {"start": 2255.28, "end": 2261.44, "text": " So we expect so we've added a little h and then we see how l changed and we expect to print"}, {"start": 2262.48, "end": 2264.48, "text": " Uh negative two"}, {"start": 2264.8, "end": 2266.0, "text": " There we go"}, {"start": 2266.0, "end": 2272.16, "text": " So we've numerically verified what we're doing here is what kind of like an inline gradient check"}, {"start": 2272.64, "end": 2278.8, "text": " gradient check is when we are deriving this like back propagation and getting the derivative with respect to all the intermediate"}, {"start": 2279.2, "end": 2282.08, "text": " results and then numerical gradient is just you know"}, {"start": 2282.56, "end": 2285.6, "text": " Um, estimating it using small step size"}, {"start": 2286.24, "end": 2289.12, "text": " Now we're going to the crux of back propagation"}, {"start": 2289.28, "end": 2294.72, "text": " So this will be the most important node to understand because if you understand the gradient for this node"}, {"start": 2294.72, "end": 2298.9599999999996, "text": " You understand all of back propagation and all of training on neural nets basically"}, {"start": 2299.68, "end": 2305.9199999999996, "text": " So we need to derive dl by dc in other words the derivative of l with respect to c"}, {"start": 2306.64, "end": 2308.9599999999996, "text": " Because we've computed all these other gradients already"}, {"start": 2309.52, "end": 2312.9599999999996, "text": " Now we're coming here and we're continuing the back propagation manually"}, {"start": 2313.7599999999998, "end": 2317.6, "text": " So we want dl by dc and then we'll also derive dl by dE"}, {"start": 2318.48, "end": 2320.16, "text": " Now here's the problem"}, {"start": 2320.16, "end": 2322.7999999999997, "text": " How do we derive dl by dc?"}, {"start": 2322.8, "end": 2329.44, "text": " We actually know the derivative l with respect to d so we know how l is sensitive to d"}, {"start": 2330.2400000000002, "end": 2336.2400000000002, "text": " But how is l sensitive to c? So if we wiggle c how does that impact l through d?"}, {"start": 2338.1600000000003, "end": 2340.1600000000003, "text": " So we know dl by dc"}, {"start": 2342.0, "end": 2348.5600000000004, "text": " And we also here know how c impacts d and so just very intuitively if you know the impact that c is having on d"}, {"start": 2348.56, "end": 2355.68, "text": " And the impact that d is having on l then you should be able to somehow put that information together to figure out how c impacts l"}, {"start": 2356.56, "end": 2358.56, "text": " And indeed this is what we can actually do"}, {"start": 2359.2799999999997, "end": 2362.4, "text": " So in particular we know just concentrating on d first"}, {"start": 2362.64, "end": 2366.16, "text": " Let's look at how what is the derivative basically of d with respect to c?"}, {"start": 2366.56, "end": 2368.56, "text": " So in other words what is dd by dc"}, {"start": 2371.68, "end": 2375.2, "text": " So here we know that d is c times c plus ee"}, {"start": 2375.2, "end": 2379.04, "text": " That's what we know and our interesting dd by dc"}, {"start": 2379.68, "end": 2384.7999999999997, "text": " If you would just know your calculus again and you remember then differentiating c plus e with respect to c"}, {"start": 2385.04, "end": 2387.3599999999997, "text": " You know that that gives you 1.0"}, {"start": 2388.0, "end": 2394.24, "text": " And we can also go back to the basics and derive this because again we can go to our f of x plus h minus f of x"}, {"start": 2394.3999999999996, "end": 2395.52, "text": " derogh divide by h"}, {"start": 2396.56, "end": 2399.3599999999997, "text": " That's the definition of a derivative as h goes to 0"}, {"start": 2400.16, "end": 2401.68, "text": " And so here"}, {"start": 2401.68, "end": 2410.08, "text": " Focusing on c and its effect on d we can basically do the f of x plus h will be c is incremented by h plus e"}, {"start": 2411.04, "end": 2415.04, "text": " That's the first evaluation of our function minus c plus e"}, {"start": 2416.64, "end": 2419.3599999999997, "text": " And then divide h and so what is this?"}, {"start": 2420.24, "end": 2424.48, "text": " Just expanding the sound this will be c plus h plus e minus c minus e"}, {"start": 2424.48, "end": 2432.48, "text": " And then you see here how c minus c cancels e minus e cancels were left with h over h which is 1.0"}, {"start": 2433.68, "end": 2441.76, "text": " And so by symmetry also dd by dd will be 1.0 as well"}, {"start": 2443.04, "end": 2447.76, "text": " So basically the derivative of a some expression is very simple and this is the local derivative"}, {"start": 2448.4, "end": 2453.28, "text": " So I call this the local derivative because we have the final output value all the way at the end of this graph"}, {"start": 2453.28, "end": 2457.52, "text": " And we're now like a small node here and this is a little plus node"}, {"start": 2458.0800000000004, "end": 2463.1200000000003, "text": " And it the little plus node doesn't know anything about the rest of the graph that it's embedded in"}, {"start": 2463.6000000000004, "end": 2468.6400000000003, "text": " All it knows is that it did it plus it took a c and a e added them and created a d"}, {"start": 2469.2000000000003, "end": 2472.8, "text": " And this plus node also knows the local influence of c on d"}, {"start": 2473.28, "end": 2476.0, "text": " Or rather were rather the derivative of d with respect to c"}, {"start": 2476.2400000000002, "end": 2479.28, "text": " And it also knows the the derivative of d with respect to e"}, {"start": 2479.28, "end": 2482.48, "text": " But that's not what we want. That's just a local derivative"}, {"start": 2482.88, "end": 2488.8, "text": " What we actually want is dl by dc and l could l is here just one step away"}, {"start": 2489.28, "end": 2493.84, "text": " But in a general case this little plus node is could be embedded in like a massive graph"}, {"start": 2494.7200000000003, "end": 2495.92, "text": " So"}, {"start": 2495.92, "end": 2500.48, "text": " Again, we know how l impacts d and now we know how c and e impact d"}, {"start": 2500.88, "end": 2506.96, "text": " How do we put that information together to write dl by dc and the answer of course is the chain rule in calculus"}, {"start": 2506.96, "end": 2508.96, "text": " And so"}, {"start": 2510.16, "end": 2512.16, "text": " I pulled up a chain rule here from capidia"}, {"start": 2513.12, "end": 2516.4, "text": " And I'm going to go through this very briefly. So chain rule"}, {"start": 2517.36, "end": 2523.12, "text": " We capidia sometimes can be very confusing and calculus can can be very confusing like this is the way I"}, {"start": 2523.68, "end": 2525.12, "text": " learned"}, {"start": 2525.2, "end": 2531.28, "text": " Chain rule and was very confusing like what is happening? It's just complicated. So I like this expression much better"}, {"start": 2531.28, "end": 2537.1200000000003, "text": " If a variable z depends on a variable y which itself depends on a variable x"}, {"start": 2538.1600000000003, "end": 2541.84, "text": " Then z depends on x as well obviously through the intermediate variable y"}, {"start": 2542.48, "end": 2547.2000000000003, "text": " And in this case the chain rule is expressed as if you want dz by dx"}, {"start": 2548.2400000000002, "end": 2552.96, "text": " Then you take the dz by dy and you multiply it by dy by dx"}, {"start": 2553.84, "end": 2556.48, "text": " So the chain rule fundamentally is telling you how"}, {"start": 2557.44, "end": 2559.44, "text": " We chain these"}, {"start": 2559.44, "end": 2561.44, "text": " Uh derivatives together"}, {"start": 2561.68, "end": 2562.48, "text": " correctly"}, {"start": 2562.48, "end": 2565.6, "text": " So to differentiate through a function composition"}, {"start": 2566.2400000000002, "end": 2570.56, "text": " We have to apply a multiplication of those derivatives"}, {"start": 2571.84, "end": 2577.28, "text": " So that's really what chain rule is telling us and there's a nice little intuitive explanation here"}, {"start": 2577.28, "end": 2579.28, "text": " Which I also think is kind of cute"}, {"start": 2579.28, "end": 2587.2000000000003, "text": " The chain rule says that knowing the instantaneous rate of change of z with respect to y and y relative to x allows one to calculate the instantaneous rate of change of z relative to x"}, {"start": 2587.2, "end": 2591.9199999999996, "text": " As a product of those two rates of change simply the product of those two"}, {"start": 2592.64, "end": 2594.64, "text": " So here's a good one"}, {"start": 2594.64, "end": 2599.3599999999997, "text": " If a car travels twice as fast as a bicycle and the bicycle is four times as fast as a walking man"}, {"start": 2600.08, "end": 2604.3199999999997, "text": " Then the car travels two times four eight times as fast as a man"}, {"start": 2605.2799999999997, "end": 2610.24, "text": " And so this makes it very clear that the correct thing to do sort of is to multiply"}, {"start": 2611.12, "end": 2616.0, "text": " So cars twice as fast as a bicycle and bicycle is four times as fast as man"}, {"start": 2616.0, "end": 2619.52, "text": " So the car will be eight times as fast as the man"}, {"start": 2620.4, "end": 2622.4, "text": " And so we can take these"}, {"start": 2622.4, "end": 2624.4, "text": " intermediate rates of change if you will and"}, {"start": 2624.88, "end": 2628.0, "text": " Multiply them together and that justifies the"}, {"start": 2628.72, "end": 2636.48, "text": " Chain rule intuitively. So I have a look at chain rule about here. Really what it means for us is there's a very simple recipe for deriving what we want"}, {"start": 2636.56, "end": 2638.56, "text": " Which is dfl by dc"}, {"start": 2639.76, "end": 2642.32, "text": " And what we have so far is we know"}, {"start": 2642.32, "end": 2644.32, "text": " one"}, {"start": 2645.28, "end": 2647.28, "text": " And we know"}, {"start": 2647.6000000000004, "end": 2655.28, "text": " What is the impact of d on l so we know dl by dd the derivative of l respect to dd"}, {"start": 2655.6000000000004, "end": 2658.96, "text": " We know that that's negative two and now because of this local"}, {"start": 2659.52, "end": 2663.36, "text": " Reason that we've done here we know dd by dc"}, {"start": 2664.6400000000003, "end": 2668.96, "text": " So how does c impact d and in particular this is a plus node"}, {"start": 2668.96, "end": 2672.48, "text": " So the local derivative is simply 1.0 is very simple"}, {"start": 2673.36, "end": 2676.64, "text": " And so the chain rule tells us that dl by dc"}, {"start": 2677.44, "end": 2679.44, "text": " going through this intermediate variable"}, {"start": 2680.2400000000002, "end": 2685.36, "text": " We'll just be simply dl by dd times"}, {"start": 2689.36, "end": 2691.36, "text": " dd by dc"}, {"start": 2691.76, "end": 2693.28, "text": " That's chain rule"}, {"start": 2693.36, "end": 2697.28, "text": " So this is identical to what's happening here except"}, {"start": 2697.28, "end": 2702.6400000000003, "text": " Z is rl y is our d and x is our c"}, {"start": 2703.84, "end": 2707.52, "text": " So we literally just have to multiply these and because"}, {"start": 2710.4, "end": 2713.28, "text": " These local derivatives like dd by dc are just one"}, {"start": 2714.8, "end": 2718.7200000000003, "text": " We basically just copy over dl by dd because this is just times one"}, {"start": 2719.6000000000004, "end": 2724.8, "text": " So what is it did so because dl by dd is negative two what is dl by dc?"}, {"start": 2724.8, "end": 2730.7200000000003, "text": " Well, it's the local gradient 1.0 times dl by dd which is negative two"}, {"start": 2731.6000000000004, "end": 2737.04, "text": " So literally what a plus node does you can look at it that way is it literally just routes the gradient"}, {"start": 2737.6000000000004, "end": 2742.8, "text": " Because the plus nodes local derivatives are just one and so in the chain rule one times"}, {"start": 2743.6800000000003, "end": 2745.6800000000003, "text": " dl by dd"}, {"start": 2745.76, "end": 2747.76, "text": " is"}, {"start": 2747.76, "end": 2754.6400000000003, "text": " Is just dl by dd and so that derivative just gets routed to both c and to e in the skates"}, {"start": 2755.6000000000004, "end": 2757.36, "text": " So basically"}, {"start": 2757.36, "end": 2759.36, "text": " We have that e dot grad"}, {"start": 2759.44, "end": 2762.8, "text": " Or what's our good c since that's the one we built that is"}, {"start": 2764.0800000000004, "end": 2767.5200000000004, "text": " negative two times one negative two"}, {"start": 2768.6400000000003, "end": 2773.44, "text": " And in the same way by symmetry e dot grad will be negative two that's the claim"}, {"start": 2774.4, "end": 2776.4, "text": " So we can set those"}, {"start": 2776.4, "end": 2778.4, "text": " We can redraw"}, {"start": 2779.52, "end": 2781.52, "text": " And you see how we just assign negative two negative two"}, {"start": 2782.1600000000003, "end": 2788.2400000000002, "text": " So there's back propagating signal which is carrying the information of like what is the derivative of l with respect to all the intermediate nodes"}, {"start": 2789.04, "end": 2795.92, "text": " We can imagine it almost like flowing backwards through the graph and a plus node will simply distribute the derivative to all the leaf nodes"}, {"start": 2796.08, "end": 2798.08, "text": " Assuring to all the children nodes of it"}, {"start": 2799.2000000000003, "end": 2801.2000000000003, "text": " So this is the claim and now let's verify it"}, {"start": 2802.0, "end": 2804.4, "text": " So let me remove the plus h here from before"}, {"start": 2804.4, "end": 2810.08, "text": " And now instead what we're going to do is we want to increment c so c dot data will be incremented by h"}, {"start": 2810.8, "end": 2812.96, "text": " And when I run this we expect to see negative two"}, {"start": 2814.48, "end": 2817.28, "text": " Negative two and then of course for e"}, {"start": 2818.96, "end": 2822.2400000000002, "text": " So e dot data plus equals h and we expect to see negative two"}, {"start": 2823.2000000000003, "end": 2825.2000000000003, "text": " Simple"}, {"start": 2827.36, "end": 2830.2400000000002, "text": " So those are the derivatives of these internal nodes"}, {"start": 2830.24, "end": 2834.64, "text": " And now we're going to recurse our way backwards again"}, {"start": 2835.6, "end": 2837.6, "text": " And we're again going to apply the chain rule"}, {"start": 2838.0, "end": 2842.64, "text": " So here we go our second application of chain rule and we will apply it all the way through the graph"}, {"start": 2842.72, "end": 2844.72, "text": " Which just happened to only have one more node remaining"}, {"start": 2845.52, "end": 2847.9199999999996, "text": " We have that d l by d e"}, {"start": 2848.64, "end": 2851.6, "text": " As we have just calculated is negative two so we know that"}, {"start": 2852.72, "end": 2854.64, "text": " So we know the derivative of l with respect to e"}, {"start": 2854.64, "end": 2861.12, "text": " And now we want d l by d a"}, {"start": 2861.8399999999997, "end": 2866.4, "text": " Right and the chain rule is telling us that that's just d l by d e"}, {"start": 2869.12, "end": 2871.7599999999998, "text": " Negative two times the local gradient"}, {"start": 2872.4, "end": 2876.3199999999997, "text": " So what is the local gradient basically d e by d a"}, {"start": 2877.12, "end": 2879.12, "text": " We have to look at that"}, {"start": 2880.08, "end": 2882.08, "text": " So I'm a little times node"}, {"start": 2882.08, "end": 2887.84, "text": " Inside a massive graph and I only know that I did a times b and I produced an e"}, {"start": 2889.2, "end": 2895.68, "text": " So now what is d e by d a and d e by d b that's the only thing that I sort of know about that's my local gradient"}, {"start": 2897.2, "end": 2899.6, "text": " So because we have that e is a times b"}, {"start": 2900.48, "end": 2902.64, "text": " We're asking what is d e by d a"}, {"start": 2904.16, "end": 2909.6, "text": " And of course we just did that here we have a times so I'm not going to redrive it"}, {"start": 2909.6, "end": 2915.92, "text": " But if you want to differentiate this with respect to a you'll just get b right the value of b"}, {"start": 2916.72, "end": 2919.04, "text": " Which in this case is negative 3.0"}, {"start": 2921.12, "end": 2924.3199999999997, "text": " So basically we have that d l by d a"}, {"start": 2925.2799999999997, "end": 2927.2, "text": " Well, let me just do it right here"}, {"start": 2927.2, "end": 2930.24, "text": " We have that a dot grad and we are applying chain rule here"}, {"start": 2930.88, "end": 2934.88, "text": " Is d l by d e which we see here is negative two"}, {"start": 2934.88, "end": 2938.88, "text": " times what is d e by d a"}, {"start": 2939.92, "end": 2942.7200000000003, "text": " It's the value of b which is negative three"}, {"start": 2945.04, "end": 2947.04, "text": " That's it"}, {"start": 2948.0, "end": 2952.48, "text": " And then we have b dot grad is again d l by d e which is negative two"}, {"start": 2953.2000000000003, "end": 2955.2000000000003, "text": " Just the same way times"}, {"start": 2955.6800000000003, "end": 2957.44, "text": " What is d e by d"}, {"start": 2958.0, "end": 2962.8, "text": " um d b is the value of a which is 2.0"}, {"start": 2962.8, "end": 2964.8, "text": " That's the value of a"}, {"start": 2965.76, "end": 2968.0800000000004, "text": " So these are our claimed derivatives"}, {"start": 2968.88, "end": 2970.6400000000003, "text": " Let's"}, {"start": 2970.6400000000003, "end": 2973.1200000000003, "text": " read draw and we see here that"}, {"start": 2973.92, "end": 2977.84, "text": " a dot grad turns out to be six because that is negative two times negative three"}, {"start": 2978.5600000000004, "end": 2980.32, "text": " And b dot grad is negative four"}, {"start": 2981.2000000000003, "end": 2984.32, "text": " times sorry is negative two times two which is negative four"}, {"start": 2985.6000000000004, "end": 2988.5600000000004, "text": " So those are our claims let's delete this and let's verify them"}, {"start": 2990.6400000000003, "end": 2992.6400000000003, "text": " We have"}, {"start": 2992.64, "end": 2995.44, "text": " a here a dot data plus equals h"}, {"start": 2997.6, "end": 3002.56, "text": " So the claim is that a dot grad is six. Let's verify"}, {"start": 3003.6, "end": 3008.16, "text": " six and we have b dot data plus equals h"}, {"start": 3008.8799999999997, "end": 3012.48, "text": " So nudging b by h and looking at what happens"}, {"start": 3013.12, "end": 3014.4, "text": " We claim it's negative four"}, {"start": 3015.2799999999997, "end": 3019.6, "text": " And indeed it's negative four plus minus again float oddness"}, {"start": 3020.64, "end": 3021.92, "text": " um"}, {"start": 3021.92, "end": 3023.44, "text": " And uh"}, {"start": 3023.44, "end": 3026.0, "text": " That's it. This that was the manual"}, {"start": 3026.88, "end": 3028.64, "text": " backpropagation"}, {"start": 3028.64, "end": 3032.96, "text": " All the way from here to all the leaf notes and I've done it piece by piece"}, {"start": 3033.44, "end": 3038.0, "text": " And really all we've done is as you saw we iterated through all the nodes one by one"}, {"start": 3038.0, "end": 3040.0, "text": " And locally applied the chain rule"}, {"start": 3040.0, "end": 3044.0, "text": " We always know what is the derivative of l with respect to this little output"}, {"start": 3044.32, "end": 3046.56, "text": " And then we look at how this output was produced"}, {"start": 3046.56, "end": 3049.12, "text": " This output was produced through some operation"}, {"start": 3049.12, "end": 3052.16, "text": " And we have the pointers to the children nodes of this operation"}, {"start": 3052.7999999999997, "end": 3056.0, "text": " And so in this little operation we know what the local derivatives are"}, {"start": 3056.56, "end": 3059.04, "text": " And we just multiply them onto the derivative always"}, {"start": 3059.7599999999998, "end": 3064.16, "text": " So we just go through and recursively multiply on the local derivatives"}, {"start": 3064.16, "end": 3068.24, "text": " And that's what backpropagation is is just a recursive application of chain rule"}, {"start": 3068.24, "end": 3069.7599999999998, "text": " Backwards through the computation graph"}, {"start": 3070.64, "end": 3073.68, "text": " Let's see this power in action just very briefly"}, {"start": 3073.68, "end": 3075.12, "text": " What we're good to do is we're going to"}, {"start": 3075.12, "end": 3078.96, "text": " Uh, nudge our inputs to try to make l go up"}, {"start": 3079.8399999999997, "end": 3083.6, "text": " So in particular what we're doing is we want a dot data. We're going to change it"}, {"start": 3084.3199999999997, "end": 3088.64, "text": " And if we want l to go up that means we just have to go in the direction of the gradient"}, {"start": 3088.96, "end": 3090.96, "text": " So a"}, {"start": 3091.12, "end": 3095.92, "text": " Should increase in the direction of gradient by like some small step amount. This is the step size"}, {"start": 3096.64, "end": 3098.96, "text": " And we don't just want this for b, but also for b"}, {"start": 3101.6, "end": 3103.6, "text": " Also for c"}, {"start": 3103.6, "end": 3105.6, "text": " Also for f"}, {"start": 3106.3199999999997, "end": 3110.08, "text": " Those are leaf nodes which we usually have control over"}, {"start": 3110.7999999999997, "end": 3115.36, "text": " And if we nudge in direction of the gradient we expect a positive influence on l"}, {"start": 3115.92, "end": 3119.2, "text": " So we expect l to go up positively"}, {"start": 3119.7599999999998, "end": 3124.88, "text": " Uh, so it should become less negative it should go up to say negative, you know, six or something like that"}, {"start": 3126.16, "end": 3130.72, "text": " Uh, it's hard to tell exactly and we have to reroute the forward path. So let me just um"}, {"start": 3130.72, "end": 3134.3199999999997, "text": " Do that here um"}, {"start": 3136.64, "end": 3141.4399999999996, "text": " This would be the forward pass f would be unchanged. This is effectively the forward pass"}, {"start": 3142.0, "end": 3144.0, "text": " And now if we print l dot data"}, {"start": 3144.7999999999997, "end": 3149.2799999999997, "text": " We expect because we nudge all the values or the inputs in the rational gradient"}, {"start": 3149.2799999999997, "end": 3152.16, "text": " We expect it less negative l. We expect it to go up"}, {"start": 3152.9599999999996, "end": 3155.6, "text": " So maybe it's negative six or so. Let's see what happens"}, {"start": 3156.7999999999997, "end": 3158.48, "text": " Okay negative seven"}, {"start": 3158.48, "end": 3163.68, "text": " And uh, this is basically one step of an optimization that will end up running"}, {"start": 3164.0, "end": 3168.88, "text": " And really this gradient just give us some power because we know how to influence the final outcome"}, {"start": 3169.44, "end": 3172.08, "text": " And this will be extremely useful for training. You know, that's as we'll see"}, {"start": 3173.04, "end": 3181.84, "text": " So now I would like to do one more uh example of manual back propagation using a bit more complex and uh useful example"}, {"start": 3182.4, "end": 3184.64, "text": " We are going to back propagate through a neuron"}, {"start": 3185.52, "end": 3187.52, "text": " so"}, {"start": 3187.52, "end": 3192.96, "text": " We want to eventually build out neuron that works in an as simplest cases are multi-layer perceptrons as they're called"}, {"start": 3193.04, "end": 3195.04, "text": " So this is a two layer neuron that"}, {"start": 3195.7599999999998, "end": 3199.52, "text": " And it's got these hidden layers made up of neurons and these neurons are fully connected to each other"}, {"start": 3200.08, "end": 3202.88, "text": " Now biologically neurons are very complicated devices"}, {"start": 3202.96, "end": 3205.52, "text": " But we have very simple mathematical models of them"}, {"start": 3206.16, "end": 3210.96, "text": " And so this is a very simple mathematical model of a neuron. You have some inputs xs"}, {"start": 3211.84, "end": 3215.04, "text": " And then you have these synapses that have weights on them"}, {"start": 3215.04, "end": 3218.16, "text": " So um, the w's are weights"}, {"start": 3219.04, "end": 3223.2799999999997, "text": " Um, and then the synapse interacts with the input to this neuron"}, {"start": 3223.6, "end": 3224.64, "text": " multiplicatively"}, {"start": 3224.64, "end": 3226.64, "text": " So what flows to the cell body"}, {"start": 3227.2799999999997, "end": 3229.44, "text": " Of this neuron is w times x"}, {"start": 3230.08, "end": 3234.24, "text": " But there's multiple inputs. There's many w times x is flowing to the cell body"}, {"start": 3235.12, "end": 3237.44, "text": " The cell body then has also like some bias"}, {"start": 3238.08, "end": 3239.84, "text": " So this is kind of like the"}, {"start": 3239.84, "end": 3243.2, "text": " In in their innate sort of trigger happiness of this neuron"}, {"start": 3243.2, "end": 3247.9199999999996, "text": " So this bias can make it a bit more trigger happy or a little less trigger happy regardless of the input"}, {"start": 3248.56, "end": 3250.7999999999997, "text": " But basically we're taking all the w times x"}, {"start": 3251.4399999999996, "end": 3256.08, "text": " Of all the inputs adding the bias and then we take it through an activation function"}, {"start": 3257.12, "end": 3259.9199999999996, "text": " And this activation function is usually some kind of a squashing function"}, {"start": 3260.48, "end": 3264.16, "text": " Like a sigmoid or 10-h or something like that. So as an example"}, {"start": 3264.7999999999997, "end": 3266.7999999999997, "text": " We're going to use the 10-h in this example"}, {"start": 3267.6, "end": 3269.04, "text": " um, numpy has a"}, {"start": 3269.8399999999997, "end": 3271.3599999999997, "text": " NP.10-h"}, {"start": 3271.36, "end": 3274.48, "text": " So um, we can call it on a range"}, {"start": 3275.04, "end": 3276.7200000000003, "text": " Then we can plot it"}, {"start": 3276.7200000000003, "end": 3280.48, "text": " This is the 10-h function and you see that the inputs as they come in"}, {"start": 3281.2000000000003, "end": 3284.6400000000003, "text": " Get squashed on the wipe coordinate here. So um"}, {"start": 3285.44, "end": 3290.1600000000003, "text": " Right at zero. We're going to get exactly zero and then as you go more positive in the input"}, {"start": 3290.96, "end": 3294.88, "text": " Then you'll see that the function will only go up to one and then plateau out"}, {"start": 3295.92, "end": 3300.48, "text": " And so if you pass in very positive inputs, we're gonna cap it smoothly at one"}, {"start": 3300.48, "end": 3303.68, "text": " And on the negative side we're gonna cap it smoothly to negative one"}, {"start": 3304.64, "end": 3309.44, "text": " So that's 10-h and that's the squashing function or an activation function"}, {"start": 3309.92, "end": 3316.16, "text": " And what comes out of this neuron is just the activation function applied to the dot product of the weights and the"}, {"start": 3316.88, "end": 3318.16, "text": " inputs"}, {"start": 3318.2400000000002, "end": 3320.2400000000002, "text": " So let's write one out"}, {"start": 3321.12, "end": 3322.64, "text": " um"}, {"start": 3322.64, "end": 3324.64, "text": " I'm going to copy paste because"}, {"start": 3327.44, "end": 3329.12, "text": " I don't want to type too much"}, {"start": 3329.12, "end": 3331.12, "text": " But okay, so here we have the inputs"}, {"start": 3331.6, "end": 3335.04, "text": " x1 x2 so this is a two-dimensional neuron. So two inputs are gonna come in"}, {"start": 3336.0, "end": 3338.24, "text": " These are thought out as the weights of the neuron"}, {"start": 3338.96, "end": 3344.3199999999997, "text": " weights w1 w2 and these weights again are the synaptic strengths for each input"}, {"start": 3345.2799999999997, "end": 3347.2799999999997, "text": " And this is the bias of the neuron"}, {"start": 3347.6, "end": 3349.2799999999997, "text": " b"}, {"start": 3349.2799999999997, "end": 3354.88, "text": " And now we want to do is according to this model we need to multiply x1 times w1"}, {"start": 3355.68, "end": 3357.68, "text": " and x2 times w2"}, {"start": 3357.68, "end": 3360.64, "text": " And then we need to add bias on top of it"}, {"start": 3361.52, "end": 3367.44, "text": " And it gets a little messy here, but all we are trying to do is x1 w1 plus x2 w2 plus b"}, {"start": 3368.0, "end": 3370.0, "text": " And these are multiply here"}, {"start": 3370.0, "end": 3374.8799999999997, "text": " Except I'm doing it in small steps so that we actually have pointers to all these intermediate nodes"}, {"start": 3375.2, "end": 3380.48, "text": " So we have x1 w1 variable x times x2 w2 variable and I'm also labeling them"}, {"start": 3381.8399999999997, "end": 3383.8399999999997, "text": " So n is now"}, {"start": 3383.8399999999997, "end": 3385.8399999999997, "text": " the cell body raw"}, {"start": 3385.84, "end": 3389.76, "text": " activation without the activation function for now"}, {"start": 3390.6400000000003, "end": 3394.7200000000003, "text": " And this should be enough to basically plot it so draw out of n"}, {"start": 3397.92, "end": 3400.96, "text": " Gives us x1 times w1 x2 times w2"}, {"start": 3402.0, "end": 3408.56, "text": " Being added then the bias gets added on top of this and this n is this sum"}, {"start": 3409.44, "end": 3411.84, "text": " So we're now going to take it through an activation function"}, {"start": 3411.84, "end": 3416.0, "text": " And let's say we use the 10h so that we produce the output"}, {"start": 3416.6400000000003, "end": 3421.84, "text": " So what we'd like to do here is we'd like to do the output and I'll call it O is"}, {"start": 3423.2000000000003, "end": 3425.2000000000003, "text": " n dot 10h"}, {"start": 3425.28, "end": 3427.6800000000003, "text": " Okay, but we haven't yet written the 10h"}, {"start": 3428.4, "end": 3432.32, "text": " Now the reason that we need to implement another 10h function here is that"}, {"start": 3433.04, "end": 3434.88, "text": " 10h is a"}, {"start": 3434.88, "end": 3441.36, "text": " hyperbolic function and we've only so far implemented plus and at times and you can't make a 10h out of just pluses and times"}, {"start": 3442.08, "end": 3443.92, "text": " You also need explanation"}, {"start": 3443.92, "end": 3446.32, "text": " So 10h is this kind of a formula here"}, {"start": 3447.28, "end": 3449.44, "text": " You can use either one of these and you see that there is"}, {"start": 3449.6800000000003, "end": 3454.2400000000002, "text": " Explanation involved which we have not implemented yet for our little value node here"}, {"start": 3454.48, "end": 3458.32, "text": " So we're not going to be able to produce 10h yet and we have to go back up and implement something like it"}, {"start": 3459.2000000000003, "end": 3461.2000000000003, "text": " now one option here"}, {"start": 3461.2, "end": 3463.2, "text": " is"}, {"start": 3463.2, "end": 3465.2, "text": " We could actually implement"}, {"start": 3465.2, "end": 3467.2, "text": " X-managementation"}, {"start": 3467.2, "end": 3471.8399999999997, "text": " And we could return the X-off value instead of a 10h of a value"}, {"start": 3472.48, "end": 3475.52, "text": " Because if we had X then we have everything else that we need"}, {"start": 3476.08, "end": 3479.4399999999996, "text": " So because we know how to add and we know how to"}, {"start": 3480.3999999999996, "end": 3481.4399999999996, "text": " um"}, {"start": 3481.4399999999996, "end": 3483.3599999999997, "text": " We know how to add and we know how to multiply"}, {"start": 3483.3599999999997, "end": 3486.64, "text": " So we'd be able to create 10h if we knew how to X-off"}, {"start": 3486.96, "end": 3489.3599999999997, "text": " But for the purposes of this example, I specifically wanted to"}, {"start": 3489.36, "end": 3491.1200000000003, "text": " Show you"}, {"start": 3491.1200000000003, "end": 3495.52, "text": " That we don't necessarily need to have the most atomic pieces in"}, {"start": 3496.1600000000003, "end": 3496.96, "text": " um"}, {"start": 3496.96, "end": 3501.36, "text": " In this value object we can actually like create functions at arbitrary"}, {"start": 3503.2000000000003, "end": 3505.92, "text": " Points of abstraction they can be complicated functions"}, {"start": 3506.0, "end": 3509.6800000000003, "text": " But they can be also very very simple functions like a plus and it's totally up to us"}, {"start": 3510.08, "end": 3513.84, "text": " The only thing that matters is that we know how to differentiate through any one function"}, {"start": 3513.84, "end": 3519.36, "text": " So we take some inputs and we make an output the only thing that matters can be arbitrarily complex function"}, {"start": 3519.92, "end": 3522.96, "text": " As long as you know how to create the local derivative"}, {"start": 3523.04, "end": 3526.88, "text": " If you know the local derivative of how the inputs impact the output then that's all you need"}, {"start": 3527.36, "end": 3530.96, "text": " So we're going to cluster up all of this expression"}, {"start": 3531.1200000000003, "end": 3533.28, "text": " And we're not going to break it down to its atomic pieces"}, {"start": 3533.28, "end": 3535.28, "text": " We're just going to directly implement 10h"}, {"start": 3535.6000000000004, "end": 3537.6000000000004, "text": " So let's do that"}, {"start": 3537.28, "end": 3539.28, "text": " depth 10h"}, {"start": 3539.28, "end": 3541.28, "text": " And then out will be a value"}, {"start": 3541.28, "end": 3543.28, "text": " Of"}, {"start": 3543.6800000000003, "end": 3545.6800000000003, "text": " And we need this expression here. So"}, {"start": 3546.0800000000004, "end": 3548.0800000000004, "text": " um"}, {"start": 3548.5600000000004, "end": 3551.0400000000004, "text": " Let me actually copy-based"}, {"start": 3554.2400000000002, "end": 3560.0800000000004, "text": " What's graph n which is a solid data and then this I believe is the 10h"}, {"start": 3561.6000000000004, "end": 3563.6000000000004, "text": " Math dot x off"}, {"start": 3564.7200000000003, "end": 3566.0800000000004, "text": " two"}, {"start": 3566.0800000000004, "end": 3567.36, "text": " No n that"}, {"start": 3567.36, "end": 3571.76, "text": " m minus one over two n plus one maybe I can call this x"}, {"start": 3573.1200000000003, "end": 3575.1200000000003, "text": " Just that it matches exactly"}, {"start": 3575.76, "end": 3578.7200000000003, "text": " Okay, and now this will be t"}, {"start": 3580.4, "end": 3583.36, "text": " And uh children of this node they're just one child"}, {"start": 3584.1600000000003, "end": 3587.76, "text": " And I'm wrapping it in a tuple so this is a couple of one object just self"}, {"start": 3588.7200000000003, "end": 3591.36, "text": " And here the name of this operation will be 10h"}, {"start": 3592.32, "end": 3594.32, "text": " And we're going to return that"}, {"start": 3594.32, "end": 3596.32, "text": " Okay"}, {"start": 3598.6400000000003, "end": 3603.76, "text": " So now values should be implementing 10h and now it's rolled away down here"}, {"start": 3604.6400000000003, "end": 3608.48, "text": " And we can actually do n dot 10h and that's going to return the 10h"}, {"start": 3609.28, "end": 3614.1600000000003, "text": " Output of n and now we should be able to draw that of oh not of n"}, {"start": 3614.88, "end": 3616.88, "text": " So let's see how that worked"}, {"start": 3618.8, "end": 3621.04, "text": " There we go and went through 10h"}, {"start": 3621.04, "end": 3623.36, "text": " To produce this up it"}, {"start": 3624.32, "end": 3626.32, "text": " So now 10h is a"}, {"start": 3626.32, "end": 3627.92, "text": " sort of"}, {"start": 3627.92, "end": 3631.2799999999997, "text": " Our little micrograt supported node here as an operation"}, {"start": 3633.2799999999997, "end": 3635.68, "text": " And as long as we know derivative of 10h"}, {"start": 3636.32, "end": 3638.32, "text": " Then we'll be able to back propagate through it"}, {"start": 3638.32, "end": 3640.32, "text": " Now let's see this 10h in action"}, {"start": 3640.4, "end": 3644.24, "text": " Currently, it's not squashing too much because the input to it is pretty low"}, {"start": 3644.8, "end": 3647.36, "text": " So if the bias was increased to say eight"}, {"start": 3647.36, "end": 3652.32, "text": " Then we'll see that what's flowing into the 10h now is"}, {"start": 3653.2000000000003, "end": 3654.48, "text": " two"}, {"start": 3654.48, "end": 3657.04, "text": " And 10h is squashing to point nine six"}, {"start": 3657.04, "end": 3659.92, "text": " So we're already hitting the tail of this 10h"}, {"start": 3659.92, "end": 3662.88, "text": " And it will sort of smoothly go up to one and then plateau out over there"}, {"start": 3663.44, "end": 3665.52, "text": " Okay, so now I'm going to do something slightly strange"}, {"start": 3665.92, "end": 3669.2000000000003, "text": " I'm going to change this bias from eight to this number"}, {"start": 3669.84, "end": 3671.84, "text": " 6.88 etc"}, {"start": 3671.92, "end": 3676.48, "text": " And I'm going to do this for specific reasons because we're about to start back propagation"}, {"start": 3676.48, "end": 3679.92, "text": " And I want to make sure that our numbers come out nice"}, {"start": 3680.16, "end": 3684.32, "text": " They're not like very crazy numbers. They're nice numbers that we can sort of understand in our head"}, {"start": 3684.8, "end": 3686.8, "text": " Let me also add pose label"}, {"start": 3687.04, "end": 3689.04, "text": " Oh, it's short for output here"}, {"start": 3690.16, "end": 3691.68, "text": " So that's the R"}, {"start": 3691.68, "end": 3692.88, "text": " Okay, so"}, {"start": 3692.88, "end": 3695.2, "text": " 28 flows into 10h comes up point seven"}, {"start": 3695.2, "end": 3699.52, "text": " So now we're going to do back propagation and we're going to fill in all the gradients"}, {"start": 3700.4, "end": 3705.76, "text": " So what is the derivative or with respect to all the inputs here"}, {"start": 3705.76, "end": 3710.8, "text": " And of course in a typical neural network setting what we really care about the most is the derivative of"}, {"start": 3711.44, "end": 3719.0400000000004, "text": " These neurons on the weights specifically the W2 and W1 because those are the weights that we're going to be changing part of the optimization"}, {"start": 3719.76, "end": 3722.8, "text": " And the other thing that we have to remember is here we have only single neuron"}, {"start": 3722.96, "end": 3725.5200000000004, "text": " But in the neural net you typically have many neurons and they're connected"}, {"start": 3727.2000000000003, "end": 3732.0, "text": " So this is only like a one small neuron a piece of a much bigger puzzle and eventually there's a loss function"}, {"start": 3732.0, "end": 3737.84, "text": " That sort of measures the accuracy of the neural net and we're back propagating with respect to that accuracy and trying to increase it"}, {"start": 3739.28, "end": 3741.76, "text": " So let's start off back propagation here and and"}, {"start": 3742.64, "end": 3749.68, "text": " What is the derivative? Oh with respect to oh the base case sort of we know always is that the gradient is just one point there"}, {"start": 3750.56, "end": 3758.32, "text": " So let me fill it in and then let me split out the drawing function"}, {"start": 3758.32, "end": 3760.48, "text": " um here"}, {"start": 3763.76, "end": 3765.76, "text": " And then here sell"}, {"start": 3767.36, "end": 3769.36, "text": " Clear this output here, okay"}, {"start": 3770.2400000000002, "end": 3773.36, "text": " So now when we draw oh we'll see that oh that grad is one"}, {"start": 3774.0800000000004, "end": 3776.0800000000004, "text": " So now we're going to back propagate through the 10 H"}, {"start": 3776.7200000000003, "end": 3780.6400000000003, "text": " So to back propagate through 10 H we need to know the local derivative of 10 H"}, {"start": 3780.64, "end": 3787.3599999999997, "text": " So if we have that oh is 10 H of n"}, {"start": 3788.56, "end": 3791.12, "text": " Then what is d oh by d n?"}, {"start": 3792.08, "end": 3798.48, "text": " Now what you could do is you could come here and you could take this expression and you could do your calculus derivative taking"}, {"start": 3799.2, "end": 3800.64, "text": " Um, and that would work"}, {"start": 3800.8799999999997, "end": 3803.2, "text": " But we can also just scroll down with the pd i here"}, {"start": 3803.8399999999997, "end": 3807.68, "text": " Into a section that hopefully tells us that derivative"}, {"start": 3807.68, "end": 3809.12, "text": " uh"}, {"start": 3809.12, "end": 3810.8799999999997, "text": " d by dx of 10 H of x is"}, {"start": 3811.8399999999997, "end": 3814.56, "text": " Any of these I like this one one minus 10 H square of x"}, {"start": 3815.3599999999997, "end": 3818.8799999999997, "text": " So this is one minus 10 H of x squared"}, {"start": 3819.52, "end": 3826.24, "text": " So basically what this is saying is that d oh by d n is one minus 10 H"}, {"start": 3827.6, "end": 3829.12, "text": " often"}, {"start": 3829.12, "end": 3831.12, "text": " squared"}, {"start": 3831.12, "end": 3833.68, "text": " And we already have 10 H of n. It's just oh"}, {"start": 3833.68, "end": 3840.48, "text": " So it's one minus oh squared. So it was the output here. So the output is this number"}, {"start": 3841.9199999999996, "end": 3847.04, "text": " Odadega is this number and then"}, {"start": 3848.24, "end": 3854.96, "text": " What this is saying is that d oh by d n is one minus this squared. So one minus Odadega squared"}, {"start": 3856.56, "end": 3858.56, "text": " It's point five conveniently"}, {"start": 3858.56, "end": 3863.36, "text": " So the local derivative of this 10 H operation here is point five and"}, {"start": 3864.4, "end": 3870.72, "text": " uh, so that would be d oh by d n so we can fill in that n dot grad"}, {"start": 3873.44, "end": 3875.44, "text": " Is point five. We'll just fill in"}, {"start": 3882.56, "end": 3884.56, "text": " So this is exactly point five one half"}, {"start": 3884.56, "end": 3887.7599999999998, "text": " So now we're going to continue the back propagation"}, {"start": 3889.36, "end": 3891.2, "text": " This is point five and this is a plus node"}, {"start": 3892.24, "end": 3896.08, "text": " So how is back prop going to what is back prop going to do here"}, {"start": 3896.72, "end": 3901.2799999999997, "text": " And if you remember our previous example a plus is just a distributor of gradient"}, {"start": 3901.84, "end": 3904.7999999999997, "text": " So this gradient will simply flow to both of these equally"}, {"start": 3905.36, "end": 3910.16, "text": " And that's because the local derivative of this operation is one for every one of its nodes"}, {"start": 3910.4, "end": 3912.24, "text": " So one times point five is point five"}, {"start": 3912.24, "end": 3917.52, "text": " So therefore we know that this node here which we called this"}, {"start": 3918.7999999999997, "end": 3923.68, "text": " It's grad is just point five and we know that b dot grad is also point five"}, {"start": 3924.9599999999996, "end": 3926.9599999999996, "text": " So let's set those and let's draw"}, {"start": 3928.9599999999996, "end": 3934.3199999999997, "text": " So those are point five continuing. We have another plus point five again. We'll just distribute you"}, {"start": 3934.32, "end": 3941.84, "text": " So point five will flow to both of these so we can set theirs"}, {"start": 3943.76, "end": 3946.88, "text": " x2w2 as well that grad is point five"}, {"start": 3948.0800000000004, "end": 3950.32, "text": " And let's read wrong. Pluses are my favorite"}, {"start": 3950.7200000000003, "end": 3952.48, "text": " operations to back propagate through because"}, {"start": 3953.28, "end": 3955.04, "text": " it's very simple"}, {"start": 3955.2000000000003, "end": 3957.84, "text": " So now it's flowing into these expressions is point five"}, {"start": 3957.92, "end": 3962.4, "text": " And so really again keep in mind what the derivative is telling us at every point in time along here"}, {"start": 3962.4, "end": 3966.64, "text": " This is saying that if we want the output of this neuron to increase"}, {"start": 3968.1600000000003, "end": 3973.84, "text": " Then the influence on these expressions is positive on the output both of them are positive"}, {"start": 3977.12, "end": 3979.12, "text": " Contribution to the output"}, {"start": 3980.64, "end": 3983.6, "text": " So now back propagating to x2 and w2 first"}, {"start": 3984.32, "end": 3988.4, "text": " This is a times node so we know that the local derivative is no the other term"}, {"start": 3988.4, "end": 3991.6, "text": " So if we want to calculate x2 dot grad"}, {"start": 3992.88, "end": 3995.28, "text": " Then can you think through what it's going to be"}, {"start": 4001.04, "end": 4004.96, "text": " So x2 dot grad will be w2 dot data times"}, {"start": 4006.4, "end": 4009.36, "text": " This x2w2 dot grad right"}, {"start": 4011.2000000000003, "end": 4013.84, "text": " And w2 dot grad will be"}, {"start": 4013.84, "end": 4018.96, "text": " x2 dot data times x2w2 dot grad"}, {"start": 4021.44, "end": 4024.2400000000002, "text": " Right, so that's the little local piece of chain rule"}, {"start": 4027.1200000000003, "end": 4029.1200000000003, "text": " Let's set them and let's redraw"}, {"start": 4029.92, "end": 4035.04, "text": " So here we see that the gradient on our weight two is zero because x2's data was zero"}, {"start": 4035.76, "end": 4040.0, "text": " Right, but x2 will have the gradient point five because data here was one"}, {"start": 4040.0, "end": 4044.96, "text": " And so what's interesting here, right is because the input x2 was zero"}, {"start": 4045.44, "end": 4047.44, "text": " And because of the way the times works"}, {"start": 4048.16, "end": 4052.0, "text": " Um, of course this gradient will be zero and think about intuitively why that is"}, {"start": 4053.2, "end": 4055.6, "text": " Derbit it always tells us the influence of"}, {"start": 4056.16, "end": 4061.04, "text": " This on the final output if I will w2 how is the output changing?"}, {"start": 4061.6, "end": 4063.84, "text": " It's not changing because we're multiplying by zero"}, {"start": 4064.48, "end": 4068.08, "text": " So because it's not changing there is no derivative and zero is the correct answer"}, {"start": 4068.08, "end": 4070.88, "text": " Because we're multiplying or swashing with that zero"}, {"start": 4072.3199999999997, "end": 4077.04, "text": " And let's do it here point five should come here and flow through this times"}, {"start": 4077.84, "end": 4080.64, "text": " And so we'll have that x1 dot grad is"}, {"start": 4082.08, "end": 4085.12, "text": " Can you think through a little bit what what this should be"}, {"start": 4087.44, "end": 4091.84, "text": " The local derivative of times with respect to x1 is going to be w1"}, {"start": 4092.72, "end": 4094.72, "text": " So w1's data times"}, {"start": 4094.72, "end": 4097.679999999999, "text": " x1 w1 dot grad"}, {"start": 4098.96, "end": 4102.48, "text": " And w1 dot grad will be x1 dot data times"}, {"start": 4103.679999999999, "end": 4105.679999999999, "text": " x1 w2 w1 dot grad"}, {"start": 4107.36, "end": 4109.36, "text": " Let's see what those came out to be"}, {"start": 4109.36, "end": 4113.76, "text": " So this is point five. So this would be negative 1.5 and this would be one"}, {"start": 4114.719999999999, "end": 4118.88, "text": " And we backpropagated through this expression these are the actual final derivatives"}, {"start": 4119.2, "end": 4122.5599999999995, "text": " So if we want this neurons output to increase"}, {"start": 4122.56, "end": 4125.52, "text": " We know that what's necessary is that"}, {"start": 4126.56, "end": 4126.56, "text": " uh"}, {"start": 4127.200000000001, "end": 4128.72, "text": " W2 we have no gradient"}, {"start": 4128.72, "end": 4131.120000000001, "text": " W2 doesn't actually matter to this neuron right now"}, {"start": 4131.6, "end": 4134.56, "text": " But this neuron this weight should uh go up"}, {"start": 4135.200000000001, "end": 4138.96, "text": " So if this weight goes up then this neuron's output would have gone up"}, {"start": 4139.52, "end": 4141.84, "text": " And proportionally because the gradient is one"}, {"start": 4141.92, "end": 4145.120000000001, "text": " Okay, so during the backpropagation manual is obviously ridiculous"}, {"start": 4145.200000000001, "end": 4147.76, "text": " So we are now going to put an end to this suffering"}, {"start": 4147.76, "end": 4149.76, "text": " And we're going to see how we can implement"}, {"start": 4149.76, "end": 4154.400000000001, "text": " uh the backward pass a bit more automatically. We're not going to be doing all of it manually out here"}, {"start": 4154.96, "end": 4159.6, "text": " It's now pretty obvious to us by example how these pluses and times are backpropagated ingredients"}, {"start": 4160.0, "end": 4162.0, "text": " So let's go up to the value"}, {"start": 4162.24, "end": 4164.56, "text": " object and we're going to start"}, {"start": 4164.64, "end": 4168.16, "text": " codifying what we've seen uh in the examples below"}, {"start": 4169.6, "end": 4172.8, "text": " So we're going to do this by storing a special self-doubt backward"}, {"start": 4174.96, "end": 4177.6, "text": " And uh underscore backward and this will be a function"}, {"start": 4177.6, "end": 4184.400000000001, "text": " Which is going to do that little piece of chain rule at each little node that complete that took inputs and produced output"}, {"start": 4184.8, "end": 4186.160000000001, "text": " Uh we're going to store"}, {"start": 4186.8, "end": 4191.6, "text": " How we are going to chain the the outputs gradient into the inputs gradients"}, {"start": 4192.400000000001, "end": 4194.0, "text": " So by default"}, {"start": 4194.160000000001, "end": 4197.360000000001, "text": " This will be a function that uh doesn't do anything"}, {"start": 4198.08, "end": 4199.120000000001, "text": " Uh so um"}, {"start": 4200.0, "end": 4202.240000000001, "text": " And you can also see that here in the value in micrograd"}, {"start": 4203.360000000001, "end": 4205.52, "text": " So with this backward function"}, {"start": 4205.52, "end": 4207.52, "text": " And by default doesn't do anything"}, {"start": 4208.56, "end": 4209.52, "text": " This is a function"}, {"start": 4210.240000000001, "end": 4214.080000000001, "text": " And that would be sort of the case for example for leaf node for leaf node. There's nothing to do"}, {"start": 4215.92, "end": 4219.040000000001, "text": " But now if when we're creating these out values"}, {"start": 4219.280000000001, "end": 4223.280000000001, "text": " These out values are an addition of self and other"}, {"start": 4224.320000000001, "end": 4226.72, "text": " And so we're going to want to self set"}, {"start": 4227.280000000001, "end": 4231.84, "text": " Out's backward to be the function that propagates the gradient"}, {"start": 4231.84, "end": 4233.4400000000005, "text": " So"}, {"start": 4234.24, "end": 4235.4400000000005, "text": " So"}, {"start": 4235.4400000000005, "end": 4237.4400000000005, "text": " Let's define what should happen"}, {"start": 4240.56, "end": 4244.400000000001, "text": " And we're going to store it in a closure. Let's define what should happen when we call"}, {"start": 4245.28, "end": 4247.28, "text": " Out's grad"}, {"start": 4247.92, "end": 4249.92, "text": " for an addition"}, {"start": 4250.16, "end": 4252.16, "text": " Our job is to take"}, {"start": 4252.16, "end": 4256.64, "text": " Out's grad and propagate it into self-scrad and other dot grad"}, {"start": 4257.12, "end": 4259.360000000001, "text": " So basically we want to self-self grad to something"}, {"start": 4259.36, "end": 4261.36, "text": " And"}, {"start": 4261.36, "end": 4263.36, "text": " We want to set others that grad to something"}, {"start": 4264.639999999999, "end": 4265.92, "text": " Okay"}, {"start": 4265.92, "end": 4269.04, "text": " And the way we saw below how chain rule works"}, {"start": 4269.04, "end": 4274.08, "text": " We want to take the local derivative times the um sort of global derivative"}, {"start": 4274.08, "end": 4279.759999999999, "text": " I should call it which is the derivative of the final output of the expression with respect to out's data"}, {"start": 4281.28, "end": 4283.04, "text": " Respect to out"}, {"start": 4283.04, "end": 4285.04, "text": " so"}, {"start": 4285.04, "end": 4289.76, "text": " The local derivative of self in an addition is 1.0"}, {"start": 4289.76, "end": 4293.04, "text": " So it's just 1.0 times out's grad"}, {"start": 4294.48, "end": 4296.0, "text": " That's the chain rule"}, {"start": 4296.0, "end": 4298.64, "text": " And others that grad will be 1.0 times out grad"}, {"start": 4299.28, "end": 4302.16, "text": " And what you basically what you're seeing here is that out's grad"}, {"start": 4302.8, "end": 4309.04, "text": " Will simply be copied onto self-scrad and others grad as we saw happens for an addition operation"}, {"start": 4309.04, "end": 4314.56, "text": " So we're going to later call this function to propagate the gradient having done an addition"}, {"start": 4315.92, "end": 4319.36, "text": " Let's not do multiplication we're going to also define a dot backward"}, {"start": 4322.4, "end": 4325.36, "text": " And we're going to set its backward to be backward"}, {"start": 4327.92, "end": 4332.0, "text": " And we want to chain out grad into self-scrad"}, {"start": 4334.4, "end": 4336.4, "text": " And others that grad"}, {"start": 4336.4, "end": 4339.759999999999, "text": " And this will be a little piece of chain rule for multiplication"}, {"start": 4340.48, "end": 4342.5599999999995, "text": " So we'll have so what should it be?"}, {"start": 4343.36, "end": 4345.36, "text": " Can you think through"}, {"start": 4348.719999999999, "end": 4350.16, "text": " So what is the local derivative?"}, {"start": 4350.879999999999, "end": 4353.36, "text": " Here the local derivative was others that data"}, {"start": 4355.5199999999995, "end": 4356.719999999999, "text": " And then"}, {"start": 4356.719999999999, "end": 4361.04, "text": " There's other stuff data and then times out that grad that's chain rule"}, {"start": 4362.639999999999, "end": 4365.2, "text": " And here we have self-that data times out that grad"}, {"start": 4365.2, "end": 4366.5599999999995, "text": " That's what we've been doing"}, {"start": 4369.679999999999, "end": 4372.4, "text": " And finally here for 10h that backward"}, {"start": 4374.96, "end": 4378.32, "text": " And then we want to set out backwards to be just backward"}, {"start": 4380.639999999999, "end": 4382.0, "text": " And here we need to"}, {"start": 4382.72, "end": 4387.599999999999, "text": " Back-propagate we have out that grad and we want to chain it into self-that grad"}, {"start": 4389.76, "end": 4391.04, "text": " And self-that grad will be"}, {"start": 4391.04, "end": 4395.44, "text": " The local derivative of this operation that we've done here which is 10h"}, {"start": 4396.24, "end": 4402.16, "text": " And so we saw that the local gradient is 1 minus the 10h of x squared which here is t"}, {"start": 4403.76, "end": 4407.12, "text": " That's the local derivative because that's t is the output of this 10h"}, {"start": 4407.5199999999995, "end": 4409.5199999999995, "text": " So 1 minus t squared is the local derivative"}, {"start": 4410.08, "end": 4412.0, "text": " And then gradient"}, {"start": 4412.56, "end": 4414.32, "text": " Has to be multiplied because of the chain rule"}, {"start": 4414.96, "end": 4418.0, "text": " So out grad is chained through the local gradient into self-that grad"}, {"start": 4418.0, "end": 4421.28, "text": " And that should be basically it"}, {"start": 4421.28, "end": 4424.0, "text": " So we're going to redefine our value node"}, {"start": 4424.96, "end": 4426.56, "text": " We're going to swing all the way down here"}, {"start": 4428.16, "end": 4432.0, "text": " And we're going to redefine our expression"}, {"start": 4432.72, "end": 4434.4, "text": " Make sure that all the grads are zero"}, {"start": 4435.44, "end": 4438.48, "text": " Okay, but now we don't have to do this manually anymore"}, {"start": 4439.84, "end": 4443.12, "text": " We are going to basically be calling the dot backward in the right order"}, {"start": 4444.16, "end": 4447.2, "text": " So first we want to call oaths"}, {"start": 4447.2, "end": 4449.2, "text": " dot backward"}, {"start": 4454.08, "end": 4456.639999999999, "text": " So o was the outcome of 10h"}, {"start": 4458.0, "end": 4461.04, "text": " Right so column oaths that back those those backward"}, {"start": 4462.32, "end": 4465.04, "text": " Will be this function. This is what it will do"}, {"start": 4466.08, "end": 4468.08, "text": " Now we have to be careful"}, {"start": 4468.08, "end": 4474.72, "text": " Because there's times out that grad and out that grad remember is initialized to zero"}, {"start": 4478.96, "end": 4481.76, "text": " So here we see grad zero. So as a base case"}, {"start": 4482.32, "end": 4485.44, "text": " We need to set oath dot grad to 1.0"}, {"start": 4486.72, "end": 4488.72, "text": " To initialize this with one"}, {"start": 4493.6, "end": 4495.28, "text": " And then once this is one"}, {"start": 4495.28, "end": 4501.599999999999, "text": " We can call o dot backward and what that should do is it should propagate this grad through 10h"}, {"start": 4502.24, "end": 4508.8, "text": " So the local derivative times the global derivative which is initialize at one. So this should"}, {"start": 4511.28, "end": 4513.28, "text": " Um"}, {"start": 4515.84, "end": 4517.2, "text": " Uh, no"}, {"start": 4517.2, "end": 4521.679999999999, "text": " So I thought about redoing it but I figured I should just leave the error in here because it's pretty funny"}, {"start": 4522.16, "end": 4524.16, "text": " Why is not I object not collable"}, {"start": 4524.16, "end": 4526.16, "text": " Uh, it's because"}, {"start": 4527.2, "end": 4532.639999999999, "text": " I screwed up we're trying to save these functions. So this is correct this here"}, {"start": 4533.44, "end": 4537.28, "text": " We don't want to call the function because that returns none these functions return none"}, {"start": 4537.68, "end": 4539.68, "text": " Which is want to store the function"}, {"start": 4539.76, "end": 4541.76, "text": " So let me redefine the value object"}, {"start": 4542.32, "end": 4545.76, "text": " And then we're going to come back and redefine the expression draw dot"}, {"start": 4546.72, "end": 4549.04, "text": " Everything is great. O dot grad is one"}, {"start": 4550.24, "end": 4552.24, "text": " O dot grad is one and now"}, {"start": 4552.24, "end": 4554.639999999999, "text": " Now this should work of course"}, {"start": 4555.84, "end": 4557.84, "text": " Okay, so all that backward should have"}, {"start": 4558.639999999999, "end": 4562.639999999999, "text": " This grad should now be point five if we withdraw and everything was correctly"}, {"start": 4563.28, "end": 4565.28, "text": " point five yay"}, {"start": 4565.36, "end": 4567.92, "text": " Okay, so now we need to call ns dot grad"}, {"start": 4570.4, "end": 4572.4, "text": " ns dot backward sorry"}, {"start": 4573.2, "end": 4574.639999999999, "text": " ns backward"}, {"start": 4574.719999999999, "end": 4576.719999999999, "text": " So that seems to have worked"}, {"start": 4577.92, "end": 4579.599999999999, "text": " So ns dot backward"}, {"start": 4579.6, "end": 4583.200000000001, "text": " Rapted the gradient to both of these so this is looking great"}, {"start": 4584.72, "end": 4586.96, "text": " Now we can of course call b dot grad"}, {"start": 4587.68, "end": 4589.68, "text": " Be the backwards or"}, {"start": 4590.320000000001, "end": 4592.08, "text": " What's going to happen?"}, {"start": 4592.160000000001, "end": 4594.320000000001, "text": " Well b doesn't have it backward"}, {"start": 4594.320000000001, "end": 4596.88, "text": " Bees backward because b is a leaf node"}, {"start": 4597.68, "end": 4600.88, "text": " Bees backward is by initialization the empty function"}, {"start": 4601.68, "end": 4605.120000000001, "text": " So nothing would happen but we can call call it on it"}, {"start": 4606.0, "end": 4608.0, "text": " But when we call"}, {"start": 4608.0, "end": 4610.0, "text": " This one"}, {"start": 4610.32, "end": 4612.32, "text": " backwards"}, {"start": 4613.6, "end": 4616.72, "text": " Then we expect this point five to get further routed"}, {"start": 4617.6, "end": 4619.92, "text": " Right, so there we go point five point five"}, {"start": 4621.04, "end": 4622.8, "text": " And then finally"}, {"start": 4622.8, "end": 4627.12, "text": " We want to call it here on x2w2"}, {"start": 4630.4, "end": 4632.4, "text": " And on x1w1"}, {"start": 4632.4, "end": 4638.639999999999, "text": " Let's do both of those and there we go"}, {"start": 4639.839999999999, "end": 4645.04, "text": " So we get 0.5 negative 1.5 and 1 exactly as we did before"}, {"start": 4645.5199999999995, "end": 4649.5199999999995, "text": " But now we've done it through calling that backward"}, {"start": 4651.12, "end": 4652.5599999999995, "text": " Sir manually"}, {"start": 4652.5599999999995, "end": 4658.0, "text": " So we have one last piece to get rid of which is us calling underscore backward manually"}, {"start": 4658.24, "end": 4660.24, "text": " So let's think through what we are actually doing"}, {"start": 4660.24, "end": 4661.44, "text": " Um"}, {"start": 4661.44, "end": 4665.92, "text": " We've laid out a mathematical expression and now we're trying to go backwards through that expression"}, {"start": 4666.5599999999995, "end": 4673.2, "text": " Um, so going backwards through the expression just means that we never want to call a dot backward for any node"}, {"start": 4674.16, "end": 4677.28, "text": " Before we've done sort of um"}, {"start": 4678.0, "end": 4680.0, "text": " Everything after it"}, {"start": 4680.0, "end": 4684.0, "text": " So we have to do everything after it before ever going to call dot backward on any one node"}, {"start": 4684.0, "end": 4687.92, "text": " We have to get all of its full dependencies everything that it depends on has to"}, {"start": 4687.92, "end": 4691.36, "text": " Propagate to it before we can continue that propagation"}, {"start": 4692.4, "end": 4697.28, "text": " So this ordering of graphs can be achieved using something called topological sort"}, {"start": 4698.0, "end": 4700.16, "text": " So topological sort"}, {"start": 4700.16, "end": 4702.64, "text": " Is basically a laying out of a graph"}, {"start": 4703.2, "end": 4706.24, "text": " Such that all the edges go only from left to right basically"}, {"start": 4706.8, "end": 4708.32, "text": " So here we have a graph"}, {"start": 4708.32, "end": 4710.96, "text": " So direction as such like a graph a dag"}, {"start": 4711.68, "end": 4715.2, "text": " And this is two different topological orders of it, I believe"}, {"start": 4715.2, "end": 4721.12, "text": " Where basically you'll see that it's a laying out of the nodes such that all the edges go only one way from left to right"}, {"start": 4722.08, "end": 4727.5199999999995, "text": " And implementing topological sort you can look in Wikipedia and so on. I'm not going to go through it in detail"}, {"start": 4728.96, "end": 4732.5599999999995, "text": " But basically this is what builds a topological graph"}, {"start": 4733.28, "end": 4737.76, "text": " Um, we maintain a set of visited nodes and then we are"}, {"start": 4738.4, "end": 4739.599999999999, "text": " um"}, {"start": 4739.6, "end": 4745.120000000001, "text": " Going through starting at some root node which for us is oh, that's what we want to start the top logical sort"}, {"start": 4745.76, "end": 4751.280000000001, "text": " And starting at oh we go through all of its children and we need to lay them out from left to right"}, {"start": 4752.8, "end": 4760.0, "text": " And basically this starts at oh if it's not visited then it marks it as visited and then it iterates through all of its children"}, {"start": 4760.88, "end": 4763.360000000001, "text": " And calls build topological on them"}, {"start": 4764.320000000001, "end": 4767.76, "text": " And then uh after it's gone through all the children it adds itself"}, {"start": 4767.76, "end": 4769.2, "text": " So basically"}, {"start": 4770.08, "end": 4776.16, "text": " This node that we're going to call it on like say oh is only going to add itself to the topical list"}, {"start": 4776.64, "end": 4781.280000000001, "text": " After all of the children have been processed and that's how this function is guaranteeing"}, {"start": 4781.84, "end": 4787.52, "text": " That you're only going to be in the list once all your children are in the list and that's the invariant that is being maintained"}, {"start": 4788.0, "end": 4791.280000000001, "text": " So if we built up on oh and then inspect this list"}, {"start": 4792.24, "end": 4795.76, "text": " We're going to see that it ordered our value objects"}, {"start": 4795.76, "end": 4800.96, "text": " And the last one is the value of 0.7 which is the output"}, {"start": 4801.92, "end": 4808.08, "text": " So this is oh and then this is n and then all the other nodes get laid out before it"}, {"start": 4809.84, "end": 4819.12, "text": " So that built the topological graph and really what we're doing now is we're just calling that underscore backward on all of the nodes in a topological order"}, {"start": 4819.92, "end": 4822.8, "text": " So if we just reset the gradients they're all zero"}, {"start": 4822.8, "end": 4830.4800000000005, "text": " So what did we do? We started by setting o.grad to be one"}, {"start": 4831.6, "end": 4833.6, "text": " That's that base case"}, {"start": 4833.76, "end": 4836.08, "text": " Then we built the topological order"}, {"start": 4838.56, "end": 4844.72, "text": " And then we went for node in reversed octopo"}, {"start": 4844.72, "end": 4853.2, "text": " Now in the reverse order because this list goes from you know we need to go through it in reverse order"}, {"start": 4854.08, "end": 4861.6, "text": " So starting at o node dot backward and this should be it"}, {"start": 4863.52, "end": 4864.72, "text": " There we go"}, {"start": 4865.68, "end": 4869.68, "text": " Those are the correct derivatives finally we are going to hide this functionality"}, {"start": 4869.68, "end": 4877.360000000001, "text": " So I'm going to copy this and we're going to hide it inside the value class because we don't want to have all that code lying around"}, {"start": 4878.320000000001, "end": 4884.400000000001, "text": " So instead of an underscore backward we're now going to define an actual backward so that backward without the underscore"}, {"start": 4886.320000000001, "end": 4888.400000000001, "text": " And that's going to do all the stuff that we just derived"}, {"start": 4889.12, "end": 4891.280000000001, "text": " So let me just clean this up a little bit. So"}, {"start": 4892.56, "end": 4894.56, "text": " We're first going to"}, {"start": 4894.56, "end": 4898.320000000001, "text": " Build the topological graph"}, {"start": 4899.120000000001, "end": 4901.120000000001, "text": " Starting at self"}, {"start": 4901.360000000001, "end": 4903.360000000001, "text": " So build topo of self"}, {"start": 4904.240000000001, "end": 4908.64, "text": " Will populate the topological order into the topo list which is a local variable"}, {"start": 4909.4400000000005, "end": 4911.6, "text": " Then we set self-adgrad to be one"}, {"start": 4913.120000000001, "end": 4918.320000000001, "text": " And then for each node in the reversed list so starting at us and going to all the children"}, {"start": 4919.280000000001, "end": 4921.120000000001, "text": " Uh, underscore backward"}, {"start": 4921.12, "end": 4926.48, "text": " And um, that should be it. So save"}, {"start": 4928.08, "end": 4930.08, "text": " Come down here we define"}, {"start": 4931.28, "end": 4933.28, "text": " Okay, all the grand are zero"}, {"start": 4933.76, "end": 4937.5199999999995, "text": " And now what we can do is oh, down backward without the underscore and"}, {"start": 4941.5199999999995, "end": 4945.12, "text": " There we go and that's uh, that's back propagation"}, {"start": 4945.12, "end": 4952.72, "text": " Please for one euro now we shouldn't be too happy with ourselves actually because we have a bad bug"}, {"start": 4953.12, "end": 4958.8, "text": " Um, and we have not surfaced the bug because of some specific conditions that we are have we have to think about right now"}, {"start": 4960.0, "end": 4962.48, "text": " So here's the simplest case that shows the bug"}, {"start": 4964.08, "end": 4966.08, "text": " Say I create a single node a"}, {"start": 4968.16, "end": 4970.16, "text": " And then I create a b that is e plus a"}, {"start": 4970.16, "end": 4974.16, "text": " And then I call it backward"}, {"start": 4974.96, "end": 4981.68, "text": " So what's gonna happen is a is three and then a is b is a plus a so there's two arrows on top of each other here"}, {"start": 4983.84, "end": 4989.28, "text": " Then we can see that b is of course the forward pass works b is just a plus a which is six"}, {"start": 4990.08, "end": 4992.08, "text": " But the gradient here is not actually correct"}, {"start": 4992.72, "end": 4994.72, "text": " That we calculate it automatically"}, {"start": 4996.08, "end": 4998.08, "text": " And that's because um"}, {"start": 4998.08, "end": 4999.28, "text": " You"}, {"start": 4999.28, "end": 5003.92, "text": " Of course, uh, just doing calculus in your head the derivative of b with respect to a"}, {"start": 5004.72, "end": 5006.72, "text": " should be uh two"}, {"start": 5007.76, "end": 5009.76, "text": " One plus one it's not one"}, {"start": 5010.96, "end": 5015.84, "text": " And totally what's happening here, right? So b is the result of a plus a and then we call backward on it"}, {"start": 5016.72, "end": 5019.2, "text": " So let's go up and see what that does"}, {"start": 5019.2, "end": 5026.5599999999995, "text": " um, b is a result of addition so out as b"}, {"start": 5028.0, "end": 5030.32, "text": " And then when we call backward what happened is"}, {"start": 5031.04, "end": 5033.84, "text": " self that grad was set to one"}, {"start": 5034.48, "end": 5036.48, "text": " And then other that grad was set to one"}, {"start": 5037.36, "end": 5042.639999999999, "text": " But because we're doing a plus a self and other are actually these as a object"}, {"start": 5042.64, "end": 5050.64, "text": " So we are overriding the gradient we are setting it to one and then we are setting it again to one and that's why it stays at"}, {"start": 5052.0, "end": 5053.200000000001, "text": " one"}, {"start": 5053.200000000001, "end": 5054.72, "text": " So that's a problem"}, {"start": 5054.72, "end": 5058.08, "text": " There's another way to see this in a little bit more complicated expression"}, {"start": 5061.6, "end": 5066.400000000001, "text": " So here we have a and b and then"}, {"start": 5066.4, "end": 5072.5599999999995, "text": " uh, d will be the multiplication of the two and he will be the addition of the two and um"}, {"start": 5073.12, "end": 5076.719999999999, "text": " Then we multiply times d to get f and then we call it f that backward"}, {"start": 5077.759999999999, "end": 5080.16, "text": " And these gradients if you check will be incorrect"}, {"start": 5080.879999999999, "end": 5084.4, "text": " So fundamentally what's happening here again is um"}, {"start": 5085.2, "end": 5088.639999999999, "text": " Basically, we're going to see an issue anytime we use a variable more than once"}, {"start": 5089.28, "end": 5094.24, "text": " Until now in these expressions above every variable is used exactly once so we didn't see the issue"}, {"start": 5094.24, "end": 5098.5599999999995, "text": " But here if a variable is used more than once what's going to happen during backward pass"}, {"start": 5099.2, "end": 5102.48, "text": " We're backpropagating from f to e to d so far so good"}, {"start": 5102.96, "end": 5107.12, "text": " But now e calls it backward and it deposits its gradients to a and b"}, {"start": 5107.5199999999995, "end": 5113.04, "text": " But then we come back to d and call backward and it overrides those gradients at a and b"}, {"start": 5114.5599999999995, "end": 5116.5599999999995, "text": " So that's obviously a problem"}, {"start": 5117.2, "end": 5123.44, "text": " And the solution here if you look at the multi-variate case of the chain rule and its generalization there"}, {"start": 5123.44, "end": 5128.96, "text": " The solution there is basically that we have to accumulate these gradients these gradients add"}, {"start": 5130.16, "end": 5132.879999999999, "text": " And so instead of setting those gradients"}, {"start": 5134.719999999999, "end": 5140.719999999999, "text": " We can simply do plus equals we need to accumulate those gradients plus equals plus equals"}, {"start": 5141.839999999999, "end": 5143.839999999999, "text": " plus equals"}, {"start": 5144.799999999999, "end": 5146.48, "text": " plus equals"}, {"start": 5146.48, "end": 5152.48, "text": " And this will be okay remember because we are initializing them at zero. So they started zero and then any"}, {"start": 5153.5199999999995, "end": 5156.0, "text": " contribution that flows backwards"}, {"start": 5157.28, "end": 5158.879999999999, "text": " Will simply add"}, {"start": 5158.879999999999, "end": 5160.879999999999, "text": " So now if we redefine"}, {"start": 5161.44, "end": 5163.44, "text": " this one"}, {"start": 5163.839999999999, "end": 5169.36, "text": " Because the plus equals this now works because a dot grad started at zero and we called b dot backward"}, {"start": 5169.679999999999, "end": 5174.32, "text": " We deposit one and then we deposit one again and now this is two which is correct"}, {"start": 5174.32, "end": 5178.0, "text": " And here this will also work and we'll get correct gradients"}, {"start": 5178.48, "end": 5182.24, "text": " Because when we call e dot backward we will deposit the gradients from this branch"}, {"start": 5182.48, "end": 5186.5599999999995, "text": " And then we get to back to d dot backward it will deposit its own gradients"}, {"start": 5187.04, "end": 5189.679999999999, "text": " And then those gradients simply add on top of each other"}, {"start": 5190.16, "end": 5192.88, "text": " And so we just accumulate those gradients and that fixes the issue"}, {"start": 5193.44, "end": 5198.32, "text": " Okay, now before we move on let me actually do a bit of cleanup here and delete some of these"}, {"start": 5198.96, "end": 5200.32, "text": " some of the intermediate work"}, {"start": 5200.32, "end": 5201.44, "text": " So"}, {"start": 5201.44, "end": 5204.0, "text": " I'm not going to need any of this now that we've derived all of it"}, {"start": 5204.96, "end": 5205.759999999999, "text": " um"}, {"start": 5205.839999999999, "end": 5208.799999999999, "text": " We are going to keep this because I want to come back to it"}, {"start": 5209.839999999999, "end": 5211.44, "text": " Delete the 10h"}, {"start": 5211.44, "end": 5213.44, "text": " delete arm when you can example"}, {"start": 5214.0, "end": 5216.0, "text": " delete the step"}, {"start": 5216.0, "end": 5218.719999999999, "text": " delete this keep the code that draws"}, {"start": 5219.839999999999, "end": 5224.24, "text": " And then delete this example and leave behind only the definition of value"}, {"start": 5225.44, "end": 5228.799999999999, "text": " And now let's come back to this non-linearity here that we implemented the 10h"}, {"start": 5228.8, "end": 5234.24, "text": " Now I told you that we could have broken down 10h into its explicit atoms"}, {"start": 5234.72, "end": 5237.28, "text": " In terms of other expressions if we had the x function"}, {"start": 5237.76, "end": 5239.6, "text": " So if you remember 10h is defined like this"}, {"start": 5240.16, "end": 5242.72, "text": " And we chose to develop 10h as a single function"}, {"start": 5242.96, "end": 5246.400000000001, "text": " And we can do that because we know it's derivative and we can back propagate through it"}, {"start": 5247.04, "end": 5250.8, "text": " But we can also break down 10h into an expressive function of x"}, {"start": 5251.28, "end": 5255.84, "text": " And I would like to do that now because I want to prove to you that you can all the same results and all the same gradients"}, {"start": 5255.84, "end": 5259.76, "text": " Um, but also because it forces us to implement a few more expressions"}, {"start": 5260.0, "end": 5261.360000000001, "text": " It forces us to do"}, {"start": 5261.52, "end": 5265.2, "text": " Accumentation, addition, subtraction, division and things like that"}, {"start": 5265.2, "end": 5267.4400000000005, "text": " And I think it's a good exercise to go through a few more of these"}, {"start": 5268.16, "end": 5270.16, "text": " Okay, so let's scroll up"}, {"start": 5270.16, "end": 5272.16, "text": " To the definition of value"}, {"start": 5272.24, "end": 5277.4400000000005, "text": " And here one thing that we currently can't do is we can do like a value of say 2.0"}, {"start": 5278.400000000001, "end": 5284.0, "text": " But we can't do you know here for example we want to add a constant one and we can't do something like this"}, {"start": 5284.0, "end": 5288.24, "text": " And we can't do it because it's just into object has no attribute data"}, {"start": 5288.64, "end": 5291.44, "text": " That's because a plus one comes right here to add"}, {"start": 5292.16, "end": 5297.44, "text": " And then other is the integer one and then here Python is trying to access one dot data"}, {"start": 5297.44, "end": 5298.72, "text": " And that's not a thing"}, {"start": 5298.72, "end": 5303.36, "text": " And that's because basically one is not a value object and we only have addition from value objects"}, {"start": 5303.92, "end": 5308.56, "text": " So as a matter of convenience so that we can create expressions like this and make them make sense"}, {"start": 5309.2, "end": 5311.2, "text": " We can simply do something like this"}, {"start": 5311.2, "end": 5317.12, "text": " Basically we let other alone if other is an instance of value"}, {"start": 5317.12, "end": 5320.96, "text": " But if it's not an instance of value we're going to assume that it's a number like an integer or float"}, {"start": 5320.96, "end": 5324.0, "text": " And we're going to simply wrap it in in value"}, {"start": 5324.16, "end": 5329.12, "text": " And then other will just become value of other and then other will have a data attribute and this should work"}, {"start": 5329.28, "end": 5332.72, "text": " So if I just say this read the farm value then this should work"}, {"start": 5333.36, "end": 5334.32, "text": " There we go"}, {"start": 5334.32, "end": 5338.16, "text": " Okay, now let's do the exact same thing for multiply because we can't do something like this"}, {"start": 5338.16, "end": 5344.4, "text": " Again for the exact same reason so we just have to go to mall and if other is"}, {"start": 5345.04, "end": 5347.04, "text": " Not a value then let's wrap it in value"}, {"start": 5347.68, "end": 5349.84, "text": " Let's redefine value and now this works"}, {"start": 5350.5599999999995, "end": 5353.599999999999, "text": " Now here's a kind of unfortunate and not obvious part"}, {"start": 5354.0, "end": 5358.4, "text": " A times two works we saw that but two times a is that going to work"}, {"start": 5359.68, "end": 5362.48, "text": " You'd expect it to write but actually it will not"}, {"start": 5363.04, "end": 5365.44, "text": " And the reason it won't is because Python doesn't know"}, {"start": 5365.44, "end": 5367.679999999999, "text": " Like when when you do a times two"}, {"start": 5368.5599999999995, "end": 5374.24, "text": " Basically um, so a times two Python will go and it will basically do something like a dot mall"}, {"start": 5374.799999999999, "end": 5381.28, "text": " Of two that's basically what we'll call but to it two times a is the same as two dot mall of a"}, {"start": 5382.0, "end": 5384.16, "text": " And it doesn't two can't multiply"}, {"start": 5385.12, "end": 5387.12, "text": " Value and so it's really confused about that"}, {"start": 5387.599999999999, "end": 5393.28, "text": " So instead what happens is in Python the way this works is you are free to define something called the armol"}, {"start": 5393.28, "end": 5394.48, "text": " and"}, {"start": 5394.48, "end": 5397.2, "text": " And armol is kind of like a fallback"}, {"start": 5397.36, "end": 5401.2, "text": " So if the Python can't do two times a it will check if"}, {"start": 5401.5199999999995, "end": 5402.719999999999, "text": " um"}, {"start": 5402.5599999999995, "end": 5407.5199999999995, "text": " If by any chance a knows how to multiply it too and that will be called into armol"}, {"start": 5408.88, "end": 5416.0, "text": " So because Python can't do two times a it will check is there an armol in value and because there is it will now call that"}, {"start": 5416.88, "end": 5420.08, "text": " And what we'll do here is we will swap the order of the operands"}, {"start": 5420.08, "end": 5427.28, "text": " So basically two times a will redirect to armol and armol will basically call it times two and that's how that will work"}, {"start": 5428.4, "end": 5432.16, "text": " So redefining that with armol two times a becomes four"}, {"start": 5432.64, "end": 5436.8, "text": " Okay, now looking at the other elements that we still need we need to know how to exponentiate and how to divide"}, {"start": 5437.2, "end": 5442.4, "text": " So let's first the explanation to the explanation part. We're going to introduce a single"}, {"start": 5443.12, "end": 5445.12, "text": " function x here"}, {"start": 5445.12, "end": 5452.64, "text": " And x is going to mirror 10h in the sense that it's a single single function that transform a single scalar value and outputs a single scalar value"}, {"start": 5453.2, "end": 5455.2, "text": " So we pop out the Python number"}, {"start": 5455.76, "end": 5458.72, "text": " We use method x to x-manitiate it create a new value object"}, {"start": 5459.36, "end": 5461.04, "text": " Everything that we've seen before"}, {"start": 5461.04, "end": 5463.92, "text": " The tricky part of course is how do you back propagate through e to the x?"}, {"start": 5464.88, "end": 5470.08, "text": " And uh, so here you can potentially pause the video and think about what should go here"}, {"start": 5470.08, "end": 5478.24, "text": " Okay, so basically we need to know what is the local derivative of e to the x"}, {"start": 5478.5599999999995, "end": 5481.92, "text": " So d by dx of e to the x is famously just e to the x"}, {"start": 5482.32, "end": 5486.24, "text": " And we've already just calculated e to the x and it's inside out that data"}, {"start": 5486.64, "end": 5488.16, "text": " So we can do about that data times"}, {"start": 5489.04, "end": 5491.12, "text": " And out that grad that's the chain"}, {"start": 5492.16, "end": 5494.48, "text": " So we're just chaining on to the current run grad"}, {"start": 5495.36, "end": 5498.24, "text": " And this is what the expression looks like it looks a little confusing"}, {"start": 5498.24, "end": 5500.88, "text": " But uh, this is what it is and that's the explanation"}, {"start": 5501.92, "end": 5504.8, "text": " So redefining we should not be able to call a dot x"}, {"start": 5505.5199999999995, "end": 5508.16, "text": " And uh, hopefully the backward pass works as well"}, {"start": 5508.32, "end": 5511.5199999999995, "text": " Okay, and the last thing we'd like to do of course is if we'd like to be able to divide"}, {"start": 5512.32, "end": 5518.16, "text": " Now I actually will implement something slightly more powerful than division because division is just a special case of"}, {"start": 5518.48, "end": 5520.08, "text": " Something a bit more powerful"}, {"start": 5520.08, "end": 5522.24, "text": " So in particular just by rearranging"}, {"start": 5522.8, "end": 5524.4, "text": " If we have some kind of a b equals"}, {"start": 5524.4, "end": 5530.719999999999, "text": " Uh, value of 4.0 here we'd like to basically be able to do a divide b and we'd like this to be able to give us 0.5"}, {"start": 5531.759999999999, "end": 5534.799999999999, "text": " Now division actually can be reshoffled as follows"}, {"start": 5535.44, "end": 5536.96, "text": " If we have a divide b"}, {"start": 5536.96, "end": 5539.2, "text": " That's actually the same as a multiplying 1 over b"}, {"start": 5539.92, "end": 5543.36, "text": " And that's the same as a multiplying b to the power of negative 1"}, {"start": 5544.4, "end": 5550.4, "text": " And so what I'd like to do instead is I basically like to implement the operation of x to the k for some constant"}, {"start": 5550.4, "end": 5557.36, "text": " uh, k so it's an integer or a float um, and we would like to be able to differentiate this and then as a special case"}, {"start": 5557.839999999999, "end": 5560.08, "text": " Uh, negative 1 will be division"}, {"start": 5560.96, "end": 5566.0, "text": " And so I'm doing that just because uh, it's more general and um, yeah, you might as well do it that way"}, {"start": 5566.48, "end": 5570.4, "text": " So basically what I'm saying is we can redefine uh, division"}, {"start": 5571.36, "end": 5573.36, "text": " Which we will put here somewhere"}, {"start": 5574.639999999999, "end": 5576.4, "text": " You know, we can put this here somewhere"}, {"start": 5576.4, "end": 5580.5599999999995, "text": " What I'm saying is that we can redefine division so self-divided other"}, {"start": 5580.879999999999, "end": 5584.879999999999, "text": " It can actually be rewritten as self times other to the power of negative 1"}, {"start": 5585.839999999999, "end": 5587.599999999999, "text": " And now"}, {"start": 5587.759999999999, "end": 5590.719999999999, "text": " Value raised to the power of negative 1 we had to now define that"}, {"start": 5591.759999999999, "end": 5592.96, "text": " So here's"}, {"start": 5593.679999999999, "end": 5595.5199999999995, "text": " So we need to implement the pow function"}, {"start": 5596.08, "end": 5598.639999999999, "text": " Where am I going to put the pow function maybe here somewhere"}, {"start": 5600.16, "end": 5602.16, "text": " This is this call it from Fort"}, {"start": 5602.16, "end": 5608.0, "text": " So this function will be called when we try to raise a value to some power and other will be that power"}, {"start": 5608.8, "end": 5615.28, "text": " Now I'd like to make sure that other is only an int or a float usually other is some kind of a different value object"}, {"start": 5615.5199999999995, "end": 5619.84, "text": " But here other will be forced to be an int or a float otherwise the math"}, {"start": 5620.5599999999995, "end": 5624.16, "text": " Uh, won't work for for we're trying to achieve in the specific case"}, {"start": 5624.5599999999995, "end": 5628.639999999999, "text": " That would be a different derivative expression if we wanted other to be a value"}, {"start": 5628.64, "end": 5636.56, "text": " So here we create the up the value which is just uh, you know, this data raised to the power of other and other here could be for example negative 1"}, {"start": 5636.72, "end": 5638.72, "text": " That's what we are hoping to achieve"}, {"start": 5639.4400000000005, "end": 5647.6, "text": " And then uh, this is the backward stub and this is the fun part which is what is the uh chain rule expression here for back"}, {"start": 5648.08, "end": 5649.68, "text": " for um"}, {"start": 5649.68, "end": 5655.12, "text": " Backpropagating through the power function where the power is to the power of some kind of a constant"}, {"start": 5655.12, "end": 5660.8, "text": " So this is the exercise and maybe pause the video here and see if you can figure it out yourself as to what we should put here"}, {"start": 5667.04, "end": 5672.32, "text": " Okay, so um, you can actually go here and look at the derivative rules as an example"}, {"start": 5672.72, "end": 5678.4, "text": " And we see lots of the derivatives that you can hopefully know from calculus in particular what we're looking for is the power rule"}, {"start": 5679.28, "end": 5683.84, "text": " Because that's telling us that if we're trying to take d by dx of x to the n which is what we're doing here"}, {"start": 5683.84, "end": 5688.72, "text": " Then that is just n times x to the n minus 1 right"}, {"start": 5689.6, "end": 5690.96, "text": " Okay"}, {"start": 5690.96, "end": 5695.12, "text": " So that's telling us about the local derivative of this power operation"}, {"start": 5696.08, "end": 5698.08, "text": " So all we want here"}, {"start": 5698.4800000000005, "end": 5705.2, "text": " Basically n is now other and self that data is x and so this now becomes"}, {"start": 5706.16, "end": 5708.16, "text": " Other which is n times"}, {"start": 5708.72, "end": 5710.24, "text": " self that data"}, {"start": 5710.400000000001, "end": 5712.400000000001, "text": " Which is now a python in to reflote"}, {"start": 5713.04, "end": 5715.599999999999, "text": " Uh, it's not a valley object. We're accessing the data attribute"}, {"start": 5716.24, "end": 5717.44, "text": " raised"}, {"start": 5717.44, "end": 5720.32, "text": " To the power of other minus one or n minus one"}, {"start": 5721.28, "end": 5724.32, "text": " I can put brackets around this, but this doesn't matter because um"}, {"start": 5725.36, "end": 5728.879999999999, "text": " Power takes precedence over multiply and pi hell so that would have been okay"}, {"start": 5729.599999999999, "end": 5735.2, "text": " And that's the local derivative only but now we have to chain it and we chain it just as simply by multiplying by on thought graph"}, {"start": 5735.5199999999995, "end": 5739.2, "text": " That's chain rule and this should uh technically work"}, {"start": 5739.2, "end": 5745.679999999999, "text": " And we're gonna find out soon, but now if we do this this should now work"}, {"start": 5746.8, "end": 5750.639999999999, "text": " And we get one five so the forward pass works, but thus the backward pass work"}, {"start": 5751.2, "end": 5756.88, "text": " And I realized that we actually also have to know how to subtract so right now a minus b will not work"}, {"start": 5757.44, "end": 5762.08, "text": " To make it work. We need one more piece of code here and"}, {"start": 5763.12, "end": 5765.12, "text": " basically this is the"}, {"start": 5765.12, "end": 5772.16, "text": " Subtraction and the way we're gonna implement subtraction is we're gonna implement it by addition of an negation and then to implement negation"}, {"start": 5772.16, "end": 5773.84, "text": " We're gonna multiply by negative one"}, {"start": 5773.84, "end": 5778.48, "text": " So just again using the stuff we've already built and just um expressing it in terms of what we have"}, {"start": 5779.04, "end": 5781.12, "text": " And a minus b does not work in"}, {"start": 5781.12, "end": 5784.32, "text": " Okay, so now let's scroll again to this expression here for this neuron"}, {"start": 5785.28, "end": 5786.96, "text": " And let's just"}, {"start": 5786.96, "end": 5791.12, "text": " compute the backward pass here once we've defined oh and let's draw it"}, {"start": 5791.12, "end": 5797.68, "text": " So here's the gradients for all these lead nodes for this two dimensional neuron that has a 10H that we've seen before"}, {"start": 5798.48, "end": 5803.84, "text": " So now what I'd like to do is I'd like to break up this 10H into this expression here"}, {"start": 5804.48, "end": 5806.64, "text": " So let me copy paste this here"}, {"start": 5807.5199999999995, "end": 5812.88, "text": " And now instead of we'll preserve the label and we will change how we define oh"}, {"start": 5813.76, "end": 5816.24, "text": " So in particular we're going to implement this formula here"}, {"start": 5816.88, "end": 5820.4, "text": " So we need each of the two x minus one over each of the x plus one"}, {"start": 5820.4, "end": 5825.839999999999, "text": " So e to the two x we need to take two times m and we need to explain it"}, {"start": 5826.48, "end": 5829.36, "text": " That's e to the two x and then because we're using it twice"}, {"start": 5829.839999999999, "end": 5831.839999999999, "text": " Let's create an intermediate variable e"}, {"start": 5832.719999999999, "end": 5838.32, "text": " And then define oh as e plus one over e minus one over e plus one"}, {"start": 5839.36, "end": 5841.92, "text": " e minus one over e plus one"}, {"start": 5842.879999999999, "end": 5846.32, "text": " And that should be it and then we should be able to draw dot of oh"}, {"start": 5846.32, "end": 5850.24, "text": " So now before I run this what do we expect to see"}, {"start": 5850.96, "end": 5853.12, "text": " Number one we're expecting to see a much longer"}, {"start": 5853.84, "end": 5857.04, "text": " graph here because we've broken up 10H into a bunch of other operations"}, {"start": 5857.759999999999, "end": 5862.08, "text": " But those operations are mathematically equivalent and so what we're expecting to see is number one"}, {"start": 5862.5599999999995, "end": 5868.32, "text": " The same result here. So the forward pass works and number two because of that mathematical equivalence"}, {"start": 5868.719999999999, "end": 5874.5599999999995, "text": " We expect to see the same backward pass and the same gradients on these lead nodes. So these gradients should be identical"}, {"start": 5874.56, "end": 5876.56, "text": " So let's run this"}, {"start": 5878.160000000001, "end": 5883.76, "text": " So number one let's verify that instead of a single 10H node we have now x and we have"}, {"start": 5884.400000000001, "end": 5888.0, "text": " Plus we have times negative one. This is the division"}, {"start": 5888.8, "end": 5895.4400000000005, "text": " And we end up with the same forward pass here and then the gradients we have to be careful because they're in slightly different order potentially"}, {"start": 5896.320000000001, "end": 5898.96, "text": " The gradients for w2 x2 should be 0 and 0.5"}, {"start": 5898.96, "end": 5904.56, "text": " W2 and x2 are 0 and 0.5 and w1 x1 are 1 and negative 1.5"}, {"start": 5905.36, "end": 5907.36, "text": " 1 and negative 1.5"}, {"start": 5907.36, "end": 5913.44, "text": " So that means that both our forward passes and backward passes were correct because this turned out to be equivalent to"}, {"start": 5914.16, "end": 5915.84, "text": " 10H before"}, {"start": 5915.84, "end": 5921.2, "text": " And so the reason I wanted to go through this exercises number one we got to practice a few more operations and"}, {"start": 5921.76, "end": 5926.4800000000005, "text": " Writing more backwards passes and number two. I wanted to illustrate the point that the"}, {"start": 5926.48, "end": 5927.839999999999, "text": " um"}, {"start": 5927.839999999999, "end": 5933.5199999999995, "text": " The level at which you implement your operations is totally up to you. You can implement backward passes for tiny"}, {"start": 5933.599999999999, "end": 5938.16, "text": " Expressions like a single individual plus or a single times or you can implement them for say"}, {"start": 5939.12, "end": 5940.08, "text": " 10H"}, {"start": 5940.08, "end": 5945.919999999999, "text": " Which is a kind of a potentially you can see it as a composite operation because it's made up of all these more atomic operations"}, {"start": 5946.4, "end": 5951.44, "text": " But really all of this is kind of like a fake concept all that matters is we have some kind of inputs and some kind of an output"}, {"start": 5951.44, "end": 5956.96, "text": " And this output is a function of the inputs in some way and as long as you can do forward pass and the backward pass of that"}, {"start": 5957.12, "end": 5958.4, "text": " little operation"}, {"start": 5958.4, "end": 5962.48, "text": " It doesn't matter what that operation is um and how composite it is"}, {"start": 5963.12, "end": 5967.2, "text": " If you can write the local gradients you can chain the gradient and you can continue back propagation"}, {"start": 5967.5199999999995, "end": 5970.96, "text": " So the design of what those functions are is completely up to you"}, {"start": 5972.08, "end": 5974.799999999999, "text": " So now I would like to show you how you can do the exact same thing"}, {"start": 5974.879999999999, "end": 5978.32, "text": " But using a modern deep neural network library like for example PyTorch"}, {"start": 5978.32, "end": 5981.04, "text": " Which I've roughly modeled micrograd"}, {"start": 5981.92, "end": 5988.0, "text": " By and so PyTorch is something you would use in production and I'll show you how you can do the exact same thing"}, {"start": 5988.0, "end": 5989.5199999999995, "text": " But in PyTorch API"}, {"start": 5989.679999999999, "end": 5993.5199999999995, "text": " So I'm just going to copy-paste it in and walk you through it a little bit. This is what it looks like"}, {"start": 5994.799999999999, "end": 5998.4, "text": " So we're going to import PyTorch and then we need to define these"}, {"start": 5999.679999999999, "end": 6004.0, "text": " Value objects like we have here now micrograd is a scalar valued"}, {"start": 6004.0, "end": 6008.4, "text": " um engine so we only have scalar values like 2.0"}, {"start": 6008.96, "end": 6013.28, "text": " But in PyTorch everything is based around tensors and like I mentioned tensors are just"}, {"start": 6013.28, "end": 6015.28, "text": " Indimensional arrays of scalars"}, {"start": 6015.84, "end": 6021.2, "text": " So that's why things get a little bit more complicated here. I just need a scalar valued tensor"}, {"start": 6021.28, "end": 6023.28, "text": " A tensor with just a single element"}, {"start": 6023.52, "end": 6026.24, "text": " But by default when you work with PyTorch you would use"}, {"start": 6026.24, "end": 6031.679999999999, "text": " um more complicated tensors like this so if I import PyTorch"}, {"start": 6034.0, "end": 6039.2, "text": " Then I can create tensors like this and this tensor for example is a 2 by 3 array"}, {"start": 6039.84, "end": 6041.84, "text": " Of scalar scalars"}, {"start": 6041.84, "end": 6048.719999999999, "text": " Um in a single compact representation. So you can check it shape. We see that it's a 2 by 3 array and so"}, {"start": 6048.72, "end": 6055.92, "text": " So this is usually what you would work with um in the actual libraries. So here I'm creating a tensor"}, {"start": 6056.4800000000005, "end": 6058.88, "text": " That has only a single element 2.0"}, {"start": 6060.56, "end": 6063.04, "text": " And then I'm casting it to be double"}, {"start": 6063.6, "end": 6067.92, "text": " Because PyTorch is by default using double precision force floating point numbers"}, {"start": 6068.0, "end": 6074.16, "text": " So I like everything to be identical by default the data type of these tensors will be float 32"}, {"start": 6074.64, "end": 6076.64, "text": " So it's only using a single precision float"}, {"start": 6076.64, "end": 6078.64, "text": " So I'm casting it to double"}, {"start": 6079.12, "end": 6081.76, "text": " So that we have float 64 just like in Python"}, {"start": 6082.72, "end": 6087.4400000000005, "text": " So I'm casting to double and then we get something similar to value of 2"}, {"start": 6088.08, "end": 6093.52, "text": " The next thing I have to do is because these are leaf nodes by default PyTorch assumes that they do not require gradients"}, {"start": 6093.92, "end": 6097.4400000000005, "text": " So I need to explicitly say that all of these nodes require gradients"}, {"start": 6098.08, "end": 6102.72, "text": " Okay, so this is going to construct scalar valued one element tensors"}, {"start": 6103.360000000001, "end": 6105.52, "text": " Make sure that PyTorch knows that they require gradients"}, {"start": 6105.52, "end": 6109.84, "text": " Now by default these are said to false by the way because of efficiency reasons"}, {"start": 6110.160000000001, "end": 6112.8, "text": " Because usually you would not want gradients for leaf nodes"}, {"start": 6113.52, "end": 6118.320000000001, "text": " Like the inputs to the network and this is just trying to be efficient in the most common cases"}, {"start": 6119.280000000001, "end": 6122.320000000001, "text": " So once we've defined all of our values in PyTorch land"}, {"start": 6122.64, "end": 6125.6, "text": " We can perform arithmetic just like we can here in micrograd land"}, {"start": 6125.92, "end": 6128.96, "text": " So this will just work and then there's a torshtot 10h also"}, {"start": 6129.76, "end": 6131.76, "text": " And when we get back is a tensor again"}, {"start": 6131.76, "end": 6137.68, "text": " And we can just like in micrograd it's got a data attribute and it's got grad attributes"}, {"start": 6138.320000000001, "end": 6142.16, "text": " So these tensor objects just like in micrograd have a dot data and a dot grad"}, {"start": 6142.8, "end": 6146.16, "text": " And the only difference here is that we need to call a dot item"}, {"start": 6146.72, "end": 6149.2, "text": " because otherwise um PyTorch"}, {"start": 6150.4800000000005, "end": 6156.8, "text": " Dot item basically takes a single tensor of one element and it just returns that element stripping out the tensor"}, {"start": 6156.8, "end": 6162.08, "text": " So let me just run this and hopefully we are going to get this is going to print the forward pass"}, {"start": 6162.56, "end": 6167.6, "text": " Which is 0.707 and this will be the gradients which hopefully are"}, {"start": 6168.4800000000005, "end": 6170.64, "text": " 0.5 to 0 negative 0.5 and 1"}, {"start": 6171.28, "end": 6173.28, "text": " So if we just run this"}, {"start": 6174.0, "end": 6175.04, "text": " There we go"}, {"start": 6175.04, "end": 6179.76, "text": " 0.7 so the forward pass agrees and then 0.5 0, 80, 0.5 and 1"}, {"start": 6180.8, "end": 6182.8, "text": " So PyTorch agrees with us"}, {"start": 6182.8, "end": 6184.320000000001, "text": " And just to show you here basically oh"}, {"start": 6184.32, "end": 6186.88, "text": " Here's a tensor with a single element"}, {"start": 6188.32, "end": 6193.5199999999995, "text": " And it's a double and we can call that item on it to just get the single number out"}, {"start": 6194.32, "end": 6201.04, "text": " So that's what item does and oh is a tensor object like I mentioned and it's got a backward function just like we've implemented"}, {"start": 6202.16, "end": 6207.12, "text": " And then all of these also have a dot grad so like x2 for example on the grad and it's a tensor"}, {"start": 6207.36, "end": 6209.92, "text": " And we can pop out the individual number with dot actum"}, {"start": 6209.92, "end": 6213.52, "text": " So basically torches"}, {"start": 6214.08, "end": 6219.92, "text": " Torch can do what we did in micrograd as a special case when your tensors are all single element tensors"}, {"start": 6220.4800000000005, "end": 6223.92, "text": " But the big deal with PyTorch is that everything is significantly more efficient"}, {"start": 6224.16, "end": 6229.92, "text": " Because we are working with these tensor objects and we can do lots of operations in parallel on all of these tensors"}, {"start": 6231.6, "end": 6235.12, "text": " But otherwise what we've built very much agrees with the API of PyTorch"}, {"start": 6235.12, "end": 6239.5199999999995, "text": " Okay, so now that we have some machinery to build out pretty complicated mathematical expressions"}, {"start": 6239.92, "end": 6246.16, "text": " We can also start building up neural nets and as I mentioned neural nets are just a specific class of mathematical expressions"}, {"start": 6247.12, "end": 6252.32, "text": " So we're going to start building out a neural net piece by piece and eventually we'll build out a two layer multi layer"}, {"start": 6252.32, "end": 6255.5199999999995, "text": " Layer perceptron as it's called and I'll show you exactly what that means"}, {"start": 6256.0, "end": 6258.0, "text": " Let's start with a single individual neuron"}, {"start": 6258.0, "end": 6259.68, "text": " We've implemented one here"}, {"start": 6259.68, "end": 6266.8, "text": " But here I'm going to implement one that also subscribes to the PyTorch API and how it designs its neural network modules"}, {"start": 6267.360000000001, "end": 6270.64, "text": " So just like we saw that we can like matched API of PyTorch"}, {"start": 6271.360000000001, "end": 6273.04, "text": " On the autograd side"}, {"start": 6273.04, "end": 6275.280000000001, "text": " We're going to try to do that on the neural network modules"}, {"start": 6275.92, "end": 6277.92, "text": " So here's class neuron"}, {"start": 6278.4800000000005, "end": 6280.88, "text": " And just for the sake of efficiency"}, {"start": 6280.88, "end": 6284.240000000001, "text": " I'm going to copy-base some sections that are relatively straightforward"}, {"start": 6284.24, "end": 6292.08, "text": " So the constructor will take a number of inputs to this neuron which is how many inputs come to a neuron"}, {"start": 6292.48, "end": 6294.48, "text": " So this one for example is three inputs"}, {"start": 6295.2, "end": 6300.719999999999, "text": " And then it's going to create a weight that is some random number between negative one and one for every one of those inputs"}, {"start": 6301.36, "end": 6305.12, "text": " And a bias that controls the overall trigger happiness of this neuron"}, {"start": 6306.639999999999, "end": 6309.84, "text": " And then we're going to implement a depth underscore underscore call"}, {"start": 6309.84, "end": 6313.6, "text": " Of self and x, sum input x"}, {"start": 6314.16, "end": 6316.8, "text": " And really what we're not going to do here is w times x plus b"}, {"start": 6317.52, "end": 6320.32, "text": " We're w times x here because they dot power specifically"}, {"start": 6321.4400000000005, "end": 6323.2, "text": " Now if you haven't seen call"}, {"start": 6324.16, "end": 6326.16, "text": " Let me just return 0.0 here from now"}, {"start": 6326.72, "end": 6330.400000000001, "text": " The way this works now is we can have an x which is say like 2.0 3.0"}, {"start": 6330.88, "end": 6333.28, "text": " Then we can initialize a neuron that is two-dimensional"}, {"start": 6334.0, "end": 6335.52, "text": " Because these are two numbers"}, {"start": 6335.52, "end": 6339.68, "text": " And then we can feed those two numbers into that neuron to again and output"}, {"start": 6339.92, "end": 6344.080000000001, "text": " And so when you use this notation n of x python will use call"}, {"start": 6345.120000000001, "end": 6347.040000000001, "text": " So currently call just returns 0.0"}, {"start": 6350.160000000001, "end": 6354.080000000001, "text": " Now we'd like to actually do the forward pass of this neuron instead"}, {"start": 6354.96, "end": 6362.64, "text": " So what we're going to do here first is we need to basically multiply all of the elements of w with all of the elements of x pairwise"}, {"start": 6362.64, "end": 6364.400000000001, "text": " We need to multiply them"}, {"start": 6364.4, "end": 6366.639999999999, "text": " So the first thing we're going to do is we're going to zip up"}, {"start": 6367.28, "end": 6368.48, "text": " sultan w and x"}, {"start": 6369.2, "end": 6372.08, "text": " And in python zip takes two iterators"}, {"start": 6372.719999999999, "end": 6377.12, "text": " And it creates a new iterator that iterates over the tuples of their corresponding entries"}, {"start": 6378.0, "end": 6381.44, "text": " So for example just to show you we can print this list"}, {"start": 6382.16, "end": 6383.92, "text": " And still returns 0.0 here"}, {"start": 6390.879999999999, "end": 6392.879999999999, "text": " Sorry"}, {"start": 6392.88, "end": 6398.32, "text": " So we see that these w's are paired up with the x's w with x"}, {"start": 6401.6, "end": 6402.8, "text": " And now what we're going to do is"}, {"start": 6406.96, "end": 6409.6, "text": " For wixi in"}, {"start": 6410.72, "end": 6414.08, "text": " We want to multiply w times wixi"}, {"start": 6414.96, "end": 6418.88, "text": " And then we want to sum all of that together to come up with an activation"}, {"start": 6419.68, "end": 6421.4400000000005, "text": " And add also sultan b on top"}, {"start": 6421.44, "end": 6423.44, "text": " So that's the real activation"}, {"start": 6423.919999999999, "end": 6426.4, "text": " And then of course we need to pass that through a non-mejority"}, {"start": 6426.719999999999, "end": 6429.12, "text": " So what we're going to be returning is act.h"}, {"start": 6429.919999999999, "end": 6431.12, "text": " And here's out"}, {"start": 6432.32, "end": 6435.12, "text": " So now we see that we are getting some outputs"}, {"start": 6435.5199999999995, "end": 6440.4, "text": " And we get a different output from neuron each time because we are initializing different weights and biases"}, {"start": 6441.28, "end": 6443.5199999999995, "text": " And then to be a bit more efficient here actually"}, {"start": 6443.5199999999995, "end": 6448.32, "text": " Some by the way takes a second optional parameter which is the start"}, {"start": 6448.32, "end": 6451.36, "text": " And by default the start is 0"}, {"start": 6451.599999999999, "end": 6455.5199999999995, "text": " So these elements of this sum will be added on top of 0 to begin with"}, {"start": 6455.759999999999, "end": 6457.5199999999995, "text": " But actually we can just start with sultan b"}, {"start": 6458.5599999999995, "end": 6460.08, "text": " And then we just have an expression like this"}, {"start": 6465.5199999999995, "end": 6468.5599999999995, "text": " And then the generator expression here must be parenthesisized by the line"}, {"start": 6469.5199999999995, "end": 6471.5199999999995, "text": " There we go"}, {"start": 6473.84, "end": 6476.16, "text": " Yep, so now we can forward a single neuron"}, {"start": 6476.16, "end": 6479.12, "text": " And next up we're going to define a layer of neurons"}, {"start": 6479.36, "end": 6481.92, "text": " So here we have a schematic for a mlp"}, {"start": 6482.5599999999995, "end": 6486.16, "text": " So we see that these mlp's each layer, this is one layer"}, {"start": 6486.4, "end": 6487.84, "text": " Has actually a number of neurons"}, {"start": 6487.84, "end": 6490.72, "text": " And they're not connected to each other but all of them are fully connected to the input"}, {"start": 6491.36, "end": 6493.12, "text": " So what is a layer of neurons?"}, {"start": 6493.12, "end": 6495.84, "text": " It's just a set of neurons evaluated independently"}, {"start": 6496.8, "end": 6501.76, "text": " So in the interest of time I'm going to do something fairly straightforward here"}, {"start": 6503.2, "end": 6505.2, "text": " It's um"}, {"start": 6505.2, "end": 6508.32, "text": " Literally a layer is just a list of neurons"}, {"start": 6509.12, "end": 6510.72, "text": " And then how many neurons do we have?"}, {"start": 6510.72, "end": 6512.639999999999, "text": " We take that as an input argument here"}, {"start": 6512.639999999999, "end": 6514.16, "text": " How many neurons do you want in your layer?"}, {"start": 6514.16, "end": 6515.76, "text": " A number of outputs in this layer"}, {"start": 6516.639999999999, "end": 6519.44, "text": " And so we just initialize completely independent neurons"}, {"start": 6519.44, "end": 6521.36, "text": " With this given dimensionality"}, {"start": 6521.36, "end": 6525.679999999999, "text": " And when we call on it we just independently evaluate them"}, {"start": 6526.4, "end": 6529.599999999999, "text": " So now instead of a neuron we can make a layer of neurons"}, {"start": 6529.599999999999, "end": 6531.76, "text": " They are two dimensional neurons and let's say three of them"}, {"start": 6531.76, "end": 6536.72, "text": " And now we see that we have three independent evaluations of three different neurons"}, {"start": 6536.72, "end": 6537.52, "text": " Right?"}, {"start": 6538.96, "end": 6540.96, "text": " Okay and finally let's complete this picture"}, {"start": 6540.96, "end": 6544.0, "text": " And define an entire multilayer perception or mlp"}, {"start": 6544.64, "end": 6546.4800000000005, "text": " And as we can see here in an mlp"}, {"start": 6546.4800000000005, "end": 6548.56, "text": " These layers just speed into each other sequentially"}, {"start": 6549.280000000001, "end": 6551.04, "text": " So let's come here and I'm just going to"}, {"start": 6551.68, "end": 6553.52, "text": " Copy the code here in interest of time"}, {"start": 6554.4800000000005, "end": 6556.16, "text": " So an mlp is very similar"}, {"start": 6556.8, "end": 6559.52, "text": " We're taking the number of inputs as before"}, {"start": 6559.52, "end": 6563.52, "text": " But now instead of taking a single n out which is number of neurons and a single layer"}, {"start": 6563.92, "end": 6566.0, "text": " We're going to take a list of n outs"}, {"start": 6566.0, "end": 6569.68, "text": " And this list defines the sizes of all the layers that we want in our mlp"}, {"start": 6570.400000000001, "end": 6572.0, "text": " So here we just put them all together"}, {"start": 6572.0, "end": 6574.0, "text": " And then iterate over consecutive pairs"}, {"start": 6574.4800000000005, "end": 6577.200000000001, "text": " Of these sizes and create a layer objects for them"}, {"start": 6577.92, "end": 6580.64, "text": " And then in the call function we are just calling them sequentially"}, {"start": 6580.64, "end": 6582.160000000001, "text": " So that's an mlp really"}, {"start": 6582.88, "end": 6584.64, "text": " And let's actually re-implement this picture"}, {"start": 6584.64, "end": 6586.64, "text": " So we want three input neurons"}, {"start": 6586.64, "end": 6589.68, "text": " And then two layers of four and an output unit"}, {"start": 6589.84, "end": 6591.200000000001, "text": " So we want"}, {"start": 6592.56, "end": 6593.68, "text": " Three dimensional input"}, {"start": 6593.68, "end": 6595.360000000001, "text": " Say this is an example input"}, {"start": 6595.360000000001, "end": 6600.56, "text": " We want three inputs into two layers of four and one output"}, {"start": 6600.56, "end": 6602.160000000001, "text": " And this of course is an mlp"}, {"start": 6603.76, "end": 6604.56, "text": " And there we go"}, {"start": 6604.56, "end": 6606.08, "text": " That's a forward passive in mlp"}, {"start": 6606.8, "end": 6608.240000000001, "text": " To make this a little bit nicer"}, {"start": 6608.240000000001, "end": 6609.84, "text": " You see how we have just a single element"}, {"start": 6609.84, "end": 6611.04, "text": " But it's wrapped in a list"}, {"start": 6611.04, "end": 6612.96, "text": " Because layer always returns lists"}, {"start": 6612.96, "end": 6616.96, "text": " So for convenience return out at zero if"}, {"start": 6616.96, "end": 6620.0, "text": " Lend out is exactly a single element"}, {"start": 6620.0, "end": 6622.0, "text": " Else return fullest"}, {"start": 6622.0, "end": 6624.96, "text": " And this will allow us to just get a single value out"}, {"start": 6624.96, "end": 6626.96, "text": " At the last layer that only has a single neuron"}, {"start": 6626.96, "end": 6630.96, "text": " And finally we should be able to prod out of n of x"}, {"start": 6630.96, "end": 6632.96, "text": " And as you might imagine"}, {"start": 6632.96, "end": 6634.96, "text": " These expressions are now getting"}, {"start": 6634.96, "end": 6636.96, "text": " Relatively involved"}, {"start": 6636.96, "end": 6640.96, "text": " So this is an entire mlp that we're defining now"}, {"start": 6640.96, "end": 6646.96, "text": " All the way until a single output"}, {"start": 6646.96, "end": 6648.96, "text": " Okay"}, {"start": 6648.96, "end": 6652.96, "text": " And so obviously you would never differentiate on pen and paper"}, {"start": 6652.96, "end": 6654.96, "text": " These expressions but with micrograd"}, {"start": 6654.96, "end": 6656.96, "text": " We will be able to back propagate all the way through this"}, {"start": 6656.96, "end": 6658.96, "text": " And back propagate"}, {"start": 6658.96, "end": 6662.96, "text": " Into these weights of all these neurons"}, {"start": 6662.96, "end": 6664.96, "text": " So let's see how that works"}, {"start": 6664.96, "end": 6666.96, "text": " Okay, so let's create ourselves a very simple"}, {"start": 6666.96, "end": 6668.96, "text": " Example data set here"}, {"start": 6668.96, "end": 6670.96, "text": " So this data set has four examples"}, {"start": 6670.96, "end": 6674.96, "text": " And so we have four possible inputs into the neural net"}, {"start": 6674.96, "end": 6676.96, "text": " And we have four desired targets"}, {"start": 6676.96, "end": 6678.96, "text": " So we'd like the neural net to"}, {"start": 6678.96, "end": 6680.96, "text": " Assign"}, {"start": 6680.96, "end": 6684.96, "text": " Or output 1.0 when it's fed this example"}, {"start": 6684.96, "end": 6686.96, "text": " Negative 1 when it's fed these examples"}, {"start": 6686.96, "end": 6688.96, "text": " And 1 when it's fed this example"}, {"start": 6688.96, "end": 6690.96, "text": " So it's a very simple binary classifier neural net"}, {"start": 6690.96, "end": 6692.96, "text": " Basically that we would like here"}, {"start": 6692.96, "end": 6694.96, "text": " Now let's think what the neural net currently thinks about these four examples"}, {"start": 6694.96, "end": 6696.96, "text": " We can just get their predictions"}, {"start": 6696.96, "end": 6700.96, "text": " Basically we can just call n of x for x in axis"}, {"start": 6700.96, "end": 6702.96, "text": " And then we can print"}, {"start": 6702.96, "end": 6706.96, "text": " So these are the outputs of the neural net on those four examples"}, {"start": 6706.96, "end": 6710.96, "text": " So the first one is 0.91"}, {"start": 6710.96, "end": 6712.96, "text": " But we'd like it to be 1"}, {"start": 6712.96, "end": 6714.96, "text": " So we should push this one higher"}, {"start": 6714.96, "end": 6716.96, "text": " This one we want to be higher"}, {"start": 6716.96, "end": 6718.96, "text": " This one says 0.88"}, {"start": 6718.96, "end": 6720.96, "text": " And we want this to be negative 1"}, {"start": 6720.96, "end": 6722.96, "text": " This is 0.88"}, {"start": 6722.96, "end": 6724.96, "text": " We want it to be negative 1"}, {"start": 6724.96, "end": 6726.96, "text": " And this one is 0.88 we want it to be 1"}, {"start": 6726.96, "end": 6728.96, "text": " So how do we make the neural net"}, {"start": 6728.96, "end": 6730.96, "text": " And how do we tune the weights"}, {"start": 6730.96, "end": 6734.96, "text": " To better predict the desired targets"}, {"start": 6734.96, "end": 6738.96, "text": " And the trick used in deep learning to achieve this"}, {"start": 6738.96, "end": 6740.96, "text": " Is to calculate a single number"}, {"start": 6740.96, "end": 6744.96, "text": " That somehow measures the total performance of your neural net"}, {"start": 6744.96, "end": 6746.96, "text": " And we call this single number the loss"}, {"start": 6746.96, "end": 6748.96, "text": " So the loss"}, {"start": 6748.96, "end": 6750.96, "text": " First is a single number"}, {"start": 6750.96, "end": 6752.96, "text": " That we're going to define"}, {"start": 6752.96, "end": 6754.96, "text": " That basically measures how well the neural net is performing"}, {"start": 6754.96, "end": 6756.96, "text": " Right now we have the intuitive sense"}, {"start": 6756.96, "end": 6758.96, "text": " That it's not performing very well"}, {"start": 6758.96, "end": 6760.96, "text": " Because we're not very much close to this"}, {"start": 6760.96, "end": 6762.96, "text": " So the loss will be high"}, {"start": 6762.96, "end": 6764.96, "text": " And we'll want to minimize the loss"}, {"start": 6764.96, "end": 6766.96, "text": " So in particular in this case what we're going to do"}, {"start": 6766.96, "end": 6768.96, "text": " Is we're going to implement the mean squared error loss"}, {"start": 6768.96, "end": 6770.96, "text": " So what this is doing"}, {"start": 6770.96, "end": 6772.96, "text": " Is we're going to basically iterate"}, {"start": 6772.96, "end": 6774.96, "text": " For y ground truth"}, {"start": 6774.96, "end": 6778.96, "text": " And y output can zip off"}, {"start": 6778.96, "end": 6780.96, "text": " Ys and y bread"}, {"start": 6780.96, "end": 6782.96, "text": " So we're going to pair up the"}, {"start": 6782.96, "end": 6784.96, "text": " Ground truth with the predictions"}, {"start": 6784.96, "end": 6786.96, "text": " And this zip iterates over tuples of them"}, {"start": 6786.96, "end": 6788.96, "text": " And for each"}, {"start": 6788.96, "end": 6792.96, "text": " Y ground truth and y output"}, {"start": 6792.96, "end": 6794.96, "text": " We're going to subtract them"}, {"start": 6794.96, "end": 6796.96, "text": " And it's squared"}, {"start": 6796.96, "end": 6798.96, "text": " So let's first see what these losses are"}, {"start": 6798.96, "end": 6800.96, "text": " These are individual loss components"}, {"start": 6800.96, "end": 6802.96, "text": " And so basically for each"}, {"start": 6802.96, "end": 6804.96, "text": " One of the four"}, {"start": 6804.96, "end": 6806.96, "text": " We are taking the prediction"}, {"start": 6806.96, "end": 6808.96, "text": " And the ground truth"}, {"start": 6808.96, "end": 6810.96, "text": " We are subtracting them"}, {"start": 6810.96, "end": 6812.96, "text": " And squaring them"}, {"start": 6812.96, "end": 6814.96, "text": " So because this one is so close to its target"}, {"start": 6814.96, "end": 6816.96, "text": " 0.91 is almost one"}, {"start": 6816.96, "end": 6818.96, "text": " Subtracting them gives a very small number"}, {"start": 6818.96, "end": 6820.96, "text": " So here we would get like a negative point one"}, {"start": 6820.96, "end": 6822.96, "text": " And then squaring it"}, {"start": 6822.96, "end": 6824.96, "text": " Just makes sure"}, {"start": 6824.96, "end": 6826.96, "text": " That regardless of whether we are more negative"}, {"start": 6826.96, "end": 6828.96, "text": " Or more positive"}, {"start": 6828.96, "end": 6830.96, "text": " We always get a positive number"}, {"start": 6830.96, "end": 6832.96, "text": " Instead of squaring the Schrold"}, {"start": 6832.96, "end": 6834.96, "text": " We could also take for example the absolute value"}, {"start": 6834.96, "end": 6836.96, "text": " We need to discard the sign"}, {"start": 6836.96, "end": 6838.96, "text": " And so you see that the expression is"}, {"start": 6838.96, "end": 6840.96, "text": " Ranged so that you only get 0 exactly"}, {"start": 6840.96, "end": 6842.96, "text": " When y out is equal to y ground truth"}, {"start": 6842.96, "end": 6844.96, "text": " When those two are equal"}, {"start": 6844.96, "end": 6846.96, "text": " So your prediction is exactly the target"}, {"start": 6846.96, "end": 6848.96, "text": " You are going to get 0"}, {"start": 6848.96, "end": 6850.96, "text": " And if your prediction is not the target"}, {"start": 6850.96, "end": 6852.96, "text": " You are going to get some other number"}, {"start": 6852.96, "end": 6854.96, "text": " So here for example we are way off"}, {"start": 6854.96, "end": 6856.96, "text": " And so that's why the loss is quite high"}, {"start": 6856.96, "end": 6858.96, "text": " And the more off we are"}, {"start": 6858.96, "end": 6860.96, "text": " The greater the loss will be"}, {"start": 6860.96, "end": 6862.96, "text": " So we don't want high loss"}, {"start": 6862.96, "end": 6864.96, "text": " We want low loss"}, {"start": 6864.96, "end": 6866.96, "text": " And so the final loss here"}, {"start": 6866.96, "end": 6868.96, "text": " Will be just the sum"}, {"start": 6868.96, "end": 6870.96, "text": " Of all of these numbers"}, {"start": 6870.96, "end": 6872.96, "text": " So you see that this should be"}, {"start": 6872.96, "end": 6874.96, "text": " 0 roughly plus 0 roughly"}, {"start": 6874.96, "end": 6876.96, "text": " But plus"}, {"start": 6876.96, "end": 6876.96, "text": " 7"}, {"start": 6876.96, "end": 6878.96, "text": " So loss should be about"}, {"start": 6878.96, "end": 6880.96, "text": " 7 here"}, {"start": 6880.96, "end": 6882.96, "text": " And now we want to minimize the loss"}, {"start": 6882.96, "end": 6884.96, "text": " We want the loss to be low"}, {"start": 6884.96, "end": 6886.96, "text": " Because the loss is low"}, {"start": 6886.96, "end": 6888.96, "text": " Then every one of the predictions"}, {"start": 6888.96, "end": 6890.96, "text": " Is equal to 0"}, {"start": 6890.96, "end": 6894.96, "text": " Then every one of the predictions is equal to its target"}, {"start": 6894.96, "end": 6898.96, "text": " So the loss, the loss, the loss, it can be 0"}, {"start": 6898.96, "end": 6900.96, "text": " And the greater it is"}, {"start": 6900.96, "end": 6902.96, "text": " The worse off the neural net is predicting"}, {"start": 6902.96, "end": 6904.96, "text": " So now of course if we do"}, {"start": 6904.96, "end": 6906.96, "text": " Loss that backward"}, {"start": 6906.96, "end": 6908.96, "text": " Something magical happened when I hit enter"}, {"start": 6908.96, "end": 6912.96, "text": " And the magical thing of course that happened"}, {"start": 6912.96, "end": 6914.96, "text": " Is that we can look at endout layers"}, {"start": 6914.96, "end": 6916.96, "text": " That neuron, endout layers"}, {"start": 6916.96, "end": 6918.96, "text": " At say like the first layer"}, {"start": 6918.96, "end": 6920.96, "text": " That neurons at 0"}, {"start": 6920.96, "end": 6922.96, "text": " Because remember that"}, {"start": 6922.96, "end": 6924.96, "text": " MLP has layers which is a list"}, {"start": 6924.96, "end": 6926.96, "text": " And each layer has a neurons which is a list"}, {"start": 6926.96, "end": 6928.96, "text": " And that gives us individual neuron"}, {"start": 6928.96, "end": 6930.96, "text": " And then it's got some weights"}, {"start": 6930.96, "end": 6932.96, "text": " And so we can for example"}, {"start": 6932.96, "end": 6934.96, "text": " Look at the weights at 0"}, {"start": 6934.96, "end": 6938.96, "text": " Oops, it's not cold weights"}, {"start": 6938.96, "end": 6940.96, "text": " It's called W"}, {"start": 6940.96, "end": 6942.96, "text": " And that's a value"}, {"start": 6942.96, "end": 6944.96, "text": " But now this value also has a graph"}, {"start": 6944.96, "end": 6946.96, "text": " Because of the backward values"}, {"start": 6946.96, "end": 6950.96, "text": " And so we see that because this gradient here"}, {"start": 6950.96, "end": 6952.96, "text": " On this particular weight of this particular neuron"}, {"start": 6952.96, "end": 6954.96, "text": " Of this particular layer is negative"}, {"start": 6954.96, "end": 6956.96, "text": " We see that its influence on the loss"}, {"start": 6956.96, "end": 6958.96, "text": " Is also negative"}, {"start": 6958.96, "end": 6960.96, "text": " So slightly increasing this particular weight"}, {"start": 6960.96, "end": 6962.96, "text": " Of this neuron of this layer"}, {"start": 6962.96, "end": 6964.96, "text": " Would make the loss go down"}, {"start": 6964.96, "end": 6966.96, "text": " And we actually have this information"}, {"start": 6966.96, "end": 6968.96, "text": " For every single one of our neurons"}, {"start": 6968.96, "end": 6970.96, "text": " And all of their parameters"}, {"start": 6970.96, "end": 6972.96, "text": " Actually it's worth looking at"}, {"start": 6972.96, "end": 6974.96, "text": " Also the draw dot loss"}, {"start": 6974.96, "end": 6976.96, "text": " So the draw dot loss by the way"}, {"start": 6976.96, "end": 6978.96, "text": " So previously we looked at the draw dot"}, {"start": 6978.96, "end": 6980.96, "text": " Of a single neuron"}, {"start": 6980.96, "end": 6982.96, "text": " Neuralin forward pass"}, {"start": 6982.96, "end": 6984.96, "text": " And that was already a large expression"}, {"start": 6984.96, "end": 6986.96, "text": " But what is this expression?"}, {"start": 6986.96, "end": 6988.96, "text": " We actually forwarded"}, {"start": 6988.96, "end": 6990.96, "text": " Every one of those four examples"}, {"start": 6990.96, "end": 6992.96, "text": " And then we have the loss in top with them"}, {"start": 6992.96, "end": 6994.96, "text": " With the mean squared error"}, {"start": 6994.96, "end": 6996.96, "text": " And so this is a really massive graph"}, {"start": 6996.96, "end": 6998.96, "text": " Because this graph that we built up now"}, {"start": 6998.96, "end": 7000.96, "text": " Oh my gosh"}, {"start": 7000.96, "end": 7002.96, "text": " This graph that we built up now"}, {"start": 7002.96, "end": 7004.96, "text": " Which is kind of excessive"}, {"start": 7004.96, "end": 7006.96, "text": " It's excessive because it has four forward passes"}, {"start": 7006.96, "end": 7008.96, "text": " Of a neural net for every one of the examples"}, {"start": 7008.96, "end": 7010.96, "text": " And then it has the loss on top"}, {"start": 7010.96, "end": 7012.96, "text": " And it ends with the value of the loss"}, {"start": 7012.96, "end": 7014.96, "text": " Which for 7.1.2"}, {"start": 7014.96, "end": 7016.96, "text": " And this loss will now back propagate"}, {"start": 7016.96, "end": 7018.96, "text": " Through all the forward passes"}, {"start": 7018.96, "end": 7020.96, "text": " All the way through just every single"}, {"start": 7020.96, "end": 7022.96, "text": " Intermediate value of the neural net"}, {"start": 7022.96, "end": 7024.96, "text": " All the way back to"}, {"start": 7024.96, "end": 7026.96, "text": " Of course the parameters of the weights"}, {"start": 7026.96, "end": 7028.96, "text": " Which are the input"}, {"start": 7028.96, "end": 7030.96, "text": " So these weight parameters here"}, {"start": 7030.96, "end": 7032.96, "text": " To this neural net"}, {"start": 7032.96, "end": 7034.96, "text": " And these numbers here"}, {"start": 7034.96, "end": 7036.96, "text": " These scalars are inputs to the neural net"}, {"start": 7036.96, "end": 7038.96, "text": " So if we went around here"}, {"start": 7038.96, "end": 7040.96, "text": " We will probably find"}, {"start": 7040.96, "end": 7042.96, "text": " Some of these examples"}, {"start": 7042.96, "end": 7044.96, "text": " This 1.0 potentially maybe this 1.0"}, {"start": 7044.96, "end": 7046.96, "text": " Or you know some of the others"}, {"start": 7046.96, "end": 7048.96, "text": " And you'll see that they all have gradients as well"}, {"start": 7048.96, "end": 7050.96, "text": " The thing is these gradients on the input data"}, {"start": 7050.96, "end": 7052.96, "text": " Are not that useful to us"}, {"start": 7052.96, "end": 7054.96, "text": " And that's because"}, {"start": 7054.96, "end": 7056.96, "text": " The input data seems to be"}, {"start": 7056.96, "end": 7058.96, "text": " Not changeable"}, {"start": 7058.96, "end": 7060.96, "text": " And it's not given to the problem"}, {"start": 7060.96, "end": 7062.96, "text": " And so it's a fixed input"}, {"start": 7062.96, "end": 7064.96, "text": " We're not going to be changing it or messing with it"}, {"start": 7064.96, "end": 7066.96, "text": " Even though we do have gradients for it"}, {"start": 7066.96, "end": 7068.96, "text": " But some of these gradients here"}, {"start": 7068.96, "end": 7070.96, "text": " Will be for the neural network parameters"}, {"start": 7070.96, "end": 7072.96, "text": " The W's and the B's"}, {"start": 7072.96, "end": 7074.96, "text": " And those we of course we want to change"}, {"start": 7074.96, "end": 7076.96, "text": " Okay so now we're going to"}, {"start": 7076.96, "end": 7078.96, "text": " Want some convenience code"}, {"start": 7078.96, "end": 7080.96, "text": " To gather up all of the parameters of the neural net"}, {"start": 7080.96, "end": 7084.96, "text": " So that we can operate on on all of them simultaneously"}, {"start": 7084.96, "end": 7086.96, "text": " And every one of them"}, {"start": 7086.96, "end": 7088.96, "text": " A tiny amount"}, {"start": 7088.96, "end": 7090.96, "text": " Based on the gradient depermission"}, {"start": 7090.96, "end": 7094.96, "text": " So let's collect the parameters of the neural net all in one array"}, {"start": 7094.96, "end": 7096.96, "text": " So let's create a parameters of self"}, {"start": 7096.96, "end": 7098.96, "text": " That just returns"}, {"start": 7098.96, "end": 7100.96, "text": " Salt that W which is a list"}, {"start": 7100.96, "end": 7102.96, "text": " Concatenated with"}, {"start": 7102.96, "end": 7104.96, "text": " A list of"}, {"start": 7104.96, "end": 7106.96, "text": " Salt that B"}, {"start": 7106.96, "end": 7108.96, "text": " So this will just return a list"}, {"start": 7108.96, "end": 7110.96, "text": " List plus list just"}, {"start": 7110.96, "end": 7112.96, "text": " You know gives you a list"}, {"start": 7112.96, "end": 7114.96, "text": " So that's parameters of neural"}, {"start": 7114.96, "end": 7116.96, "text": " And I'm calling it this way because also"}, {"start": 7116.96, "end": 7118.96, "text": " Pipe Torch has a parameters on every single"}, {"start": 7118.96, "end": 7120.96, "text": " And in module"}, {"start": 7120.96, "end": 7122.96, "text": " And it does exactly what we're doing here"}, {"start": 7122.96, "end": 7124.96, "text": " It just returns the"}, {"start": 7124.96, "end": 7126.96, "text": " Parameter tensors for us is the primary scalers"}, {"start": 7126.96, "end": 7130.96, "text": " Now layer is also a module"}, {"start": 7130.96, "end": 7132.96, "text": " So it will have parameters"}, {"start": 7132.96, "end": 7134.96, "text": " Self"}, {"start": 7134.96, "end": 7136.96, "text": " And basically what we want to do here is"}, {"start": 7136.96, "end": 7138.96, "text": " Something like this like"}, {"start": 7138.96, "end": 7140.96, "text": " Param's is here"}, {"start": 7140.96, "end": 7142.96, "text": " And then for"}, {"start": 7142.96, "end": 7144.96, "text": " Neuron in salt that neurons"}, {"start": 7144.96, "end": 7148.96, "text": " We want to get neuron that parameters"}, {"start": 7148.96, "end": 7150.96, "text": " And we want to"}, {"start": 7150.96, "end": 7152.96, "text": " Param's that extend"}, {"start": 7152.96, "end": 7154.96, "text": " Right so these are the"}, {"start": 7154.96, "end": 7156.96, "text": " Parameters of this neuron"}, {"start": 7156.96, "end": 7158.96, "text": " And then we want to put them on top of"}, {"start": 7158.96, "end": 7159.96, "text": " Param's so"}, {"start": 7159.96, "end": 7160.96, "text": " Param's that extend of"}, {"start": 7160.96, "end": 7162.96, "text": " Peace"}, {"start": 7162.96, "end": 7163.96, "text": " And then we want to return"}, {"start": 7163.96, "end": 7164.96, "text": " Param's"}, {"start": 7164.96, "end": 7166.96, "text": " So this is way too much code"}, {"start": 7166.96, "end": 7168.96, "text": " So actually there's a way to simplify this"}, {"start": 7168.96, "end": 7170.96, "text": " Which is"}, {"start": 7170.96, "end": 7172.96, "text": " Return"}, {"start": 7172.96, "end": 7174.96, "text": " P"}, {"start": 7174.96, "end": 7176.96, "text": " For neuron in self"}, {"start": 7176.96, "end": 7178.96, "text": " That neurons"}, {"start": 7178.96, "end": 7180.96, "text": " For"}, {"start": 7180.96, "end": 7184.96, "text": " P in neuron dot parameters"}, {"start": 7184.96, "end": 7186.96, "text": " So it's a single list comprehension"}, {"start": 7186.96, "end": 7188.96, "text": " In python you can sort of nest"}, {"start": 7188.96, "end": 7190.96, "text": " Then like this and you can"}, {"start": 7190.96, "end": 7192.96, "text": " Then create"}, {"start": 7192.96, "end": 7194.96, "text": " The desired array"}, {"start": 7194.96, "end": 7196.96, "text": " So these are identical"}, {"start": 7196.96, "end": 7198.96, "text": " We can take this out"}, {"start": 7198.96, "end": 7200.96, "text": " And then let's do the same here"}, {"start": 7200.96, "end": 7204.96, "text": " Deframeters"}, {"start": 7204.96, "end": 7206.96, "text": " Self"}, {"start": 7206.96, "end": 7208.96, "text": " And return"}, {"start": 7208.96, "end": 7210.96, "text": " A parameter for layer"}, {"start": 7210.96, "end": 7212.96, "text": " In self dot layers"}, {"start": 7212.96, "end": 7214.96, "text": " For"}, {"start": 7214.96, "end": 7216.96, "text": " P in layer dot parameters"}, {"start": 7216.96, "end": 7220.96, "text": " And that should be good"}, {"start": 7220.96, "end": 7224.96, "text": " Now let me pop out this"}, {"start": 7224.96, "end": 7226.96, "text": " So we don't re-initialize our network"}, {"start": 7226.96, "end": 7228.96, "text": " Because we need to re-initialize our"}, {"start": 7228.96, "end": 7230.96, "text": " Okay so unfortunately we"}, {"start": 7230.96, "end": 7232.96, "text": " Will have to probably re-initialize"}, {"start": 7232.96, "end": 7234.96, "text": " Network because we just had"}, {"start": 7234.96, "end": 7236.96, "text": " Functionality because this class"}, {"start": 7236.96, "end": 7238.96, "text": " Of course we i want to get"}, {"start": 7238.96, "end": 7240.96, "text": " All the end dot parameters"}, {"start": 7240.96, "end": 7242.96, "text": " But that's not going to work because this is the old class"}, {"start": 7242.96, "end": 7246.96, "text": " Okay so unfortunately we do have to re-initialize the network"}, {"start": 7246.96, "end": 7248.96, "text": " Which will change some of the numbers"}, {"start": 7248.96, "end": 7250.96, "text": " But let me do that so"}, {"start": 7250.96, "end": 7252.96, "text": " That we can do that"}, {"start": 7252.96, "end": 7256.96, "text": " So that we pick up the new API"}, {"start": 7256.96, "end": 7258.96, "text": " We can now do end dot parameters"}, {"start": 7258.96, "end": 7260.96, "text": " And these are all the weights and biases"}, {"start": 7260.96, "end": 7262.96, "text": " Inside the entire neural net"}, {"start": 7262.96, "end": 7268.96, "text": " So in total this MLP has 41 parameters"}, {"start": 7268.96, "end": 7272.96, "text": " And now we'll be able to change them"}, {"start": 7272.96, "end": 7276.96, "text": " If we recalculate the loss here"}, {"start": 7276.96, "end": 7278.96, "text": " We see that unfortunately we have slightly different"}, {"start": 7278.96, "end": 7280.96, "text": " Predictionality"}, {"start": 7280.96, "end": 7284.96, "text": " Predictions and slightly different loss"}, {"start": 7284.96, "end": 7286.96, "text": " But that's okay"}, {"start": 7286.96, "end": 7290.96, "text": " Okay so we see that this neuron's"}, {"start": 7290.96, "end": 7292.96, "text": " Gradient is slightly negative"}, {"start": 7292.96, "end": 7294.96, "text": " We can also look at its data right now"}, {"start": 7294.96, "end": 7296.96, "text": " Which is 0.85"}, {"start": 7296.96, "end": 7298.96, "text": " So this is the current value of this neuron"}, {"start": 7298.96, "end": 7300.96, "text": " And this is its gradient on the loss"}, {"start": 7300.96, "end": 7304.96, "text": " So what we want to do now is"}, {"start": 7304.96, "end": 7306.96, "text": " We want to iterate for every P"}, {"start": 7306.96, "end": 7308.96, "text": " N end dot parameters"}, {"start": 7308.96, "end": 7310.96, "text": " So for all the 41 parameters of this neuron net"}, {"start": 7310.96, "end": 7312.96, "text": " We actually want to change"}, {"start": 7312.96, "end": 7314.96, "text": " P data data"}, {"start": 7314.96, "end": 7318.96, "text": " Slightly according to the gradient information"}, {"start": 7318.96, "end": 7320.96, "text": " Okay so data to do here"}, {"start": 7320.96, "end": 7324.96, "text": " But this will be basically a tiny update"}, {"start": 7324.96, "end": 7328.96, "text": " In this gradient descent scheme"}, {"start": 7328.96, "end": 7330.96, "text": " And gradient descent we are thinking of the gradient"}, {"start": 7330.96, "end": 7336.96, "text": " As a vector pointing in the direction of increased loss"}, {"start": 7336.96, "end": 7340.96, "text": " And so in gradient descent"}, {"start": 7340.96, "end": 7342.96, "text": " We are modifying P dot data"}, {"start": 7342.96, "end": 7346.96, "text": " By a small step size in the direction of the gradient"}, {"start": 7346.96, "end": 7348.96, "text": " So the step size as an example could be"}, {"start": 7348.96, "end": 7350.96, "text": " Like a very small number of 0.01 is the step size"}, {"start": 7350.96, "end": 7352.96, "text": " Times P dot grad"}, {"start": 7352.96, "end": 7354.96, "text": " Right"}, {"start": 7354.96, "end": 7358.96, "text": " But we have to think through some of the signs here"}, {"start": 7358.96, "end": 7360.96, "text": " So in particular"}, {"start": 7360.96, "end": 7362.96, "text": " Working with this specific example here"}, {"start": 7362.96, "end": 7364.96, "text": " We see that if we just left it like this"}, {"start": 7364.96, "end": 7366.96, "text": " Then this neurons value"}, {"start": 7366.96, "end": 7370.96, "text": " Would be currently increased by a tiny amount of the gradient"}, {"start": 7370.96, "end": 7372.96, "text": " The gradient is negative"}, {"start": 7372.96, "end": 7376.96, "text": " So this value of this neuron would go slightly down"}, {"start": 7376.96, "end": 7378.96, "text": " It would become like 0.8"}, {"start": 7378.96, "end": 7380.96, "text": " You know, 4 or something like that"}, {"start": 7380.96, "end": 7384.96, "text": " But if this neurons value goes lower"}, {"start": 7384.96, "end": 7388.96, "text": " That would actually increase the loss"}, {"start": 7388.96, "end": 7392.96, "text": " That because the derivative of this neuron is negative"}, {"start": 7392.96, "end": 7396.96, "text": " So increasing this makes the loss go down"}, {"start": 7396.96, "end": 7400.96, "text": " So increasing it is what we want to do instead of decreasing it"}, {"start": 7400.96, "end": 7402.96, "text": " So basically what we are missing here is"}, {"start": 7402.96, "end": 7404.96, "text": " We are actually missing a negative sign"}, {"start": 7404.96, "end": 7406.96, "text": " And again this other interpretation"}, {"start": 7406.96, "end": 7408.96, "text": " And that's because we want to minimize the loss"}, {"start": 7408.96, "end": 7410.96, "text": " We don't want to maximize the loss"}, {"start": 7410.96, "end": 7412.96, "text": " We want to decrease it"}, {"start": 7412.96, "end": 7414.96, "text": " And the other interpretation as I mentioned"}, {"start": 7414.96, "end": 7416.96, "text": " Is you can think of the gradient vector"}, {"start": 7416.96, "end": 7418.96, "text": " So basically just the vector of all the gradients"}, {"start": 7418.96, "end": 7422.96, "text": " As pointing in the direction of increasing the loss"}, {"start": 7422.96, "end": 7424.96, "text": " But then we want to decrease it"}, {"start": 7424.96, "end": 7426.96, "text": " So we actually want to go in the opposite direction"}, {"start": 7426.96, "end": 7428.96, "text": " And so you can convince yourself that this"}, {"start": 7428.96, "end": 7430.96, "text": " Or like thus the right thing here with the negative"}, {"start": 7430.96, "end": 7432.96, "text": " Because we want to minimize the loss"}, {"start": 7432.96, "end": 7436.96, "text": " So if we notch all the parameters by a tiny amount"}, {"start": 7436.96, "end": 7442.96, "text": " Then we'll see that this data will change a little bit"}, {"start": 7442.96, "end": 7446.96, "text": " So now this neuron is a tiny amount creator"}, {"start": 7446.96, "end": 7448.96, "text": " A tiny amount creator"}, {"start": 7448.96, "end": 7450.96, "text": " Value"}, {"start": 7450.96, "end": 7452.96, "text": " So 0.854, once it's 0.857"}, {"start": 7452.96, "end": 7454.96, "text": " And that's a good thing"}, {"start": 7454.96, "end": 7456.96, "text": " Because slightly increasing this neuron"}, {"start": 7456.96, "end": 7460.96, "text": " Data makes the loss go down"}, {"start": 7460.96, "end": 7462.96, "text": " According to the gradient"}, {"start": 7462.96, "end": 7464.96, "text": " And so the correct thing has happened signwise"}, {"start": 7464.96, "end": 7468.96, "text": " And so now what we would expect of course is that"}, {"start": 7468.96, "end": 7470.96, "text": " Because we've changed all these parameters"}, {"start": 7470.96, "end": 7474.96, "text": " We expect that the loss should have gone down a bit"}, {"start": 7474.96, "end": 7476.96, "text": " So we want to re-evaluate the loss"}, {"start": 7476.96, "end": 7478.96, "text": " Let me basically"}, {"start": 7478.96, "end": 7482.96, "text": " This is just a data definition that hasn't changed"}, {"start": 7482.96, "end": 7484.96, "text": " But the forward pass here"}, {"start": 7484.96, "end": 7486.96, "text": " Of the network we can recalculate"}, {"start": 7488.96, "end": 7490.96, "text": " And actually let me do it outside here"}, {"start": 7490.96, "end": 7492.96, "text": " So that we can compare the two loss values"}, {"start": 7492.96, "end": 7496.96, "text": " So here if I recalculate the loss"}, {"start": 7496.96, "end": 7500.96, "text": " We'd expect the new loss now to be slightly lower than this number"}, {"start": 7500.96, "end": 7502.96, "text": " So hopefully what we're getting now"}, {"start": 7502.96, "end": 7504.96, "text": " Is a tiny bit lower than 4.854"}, {"start": 7506.96, "end": 7508.96, "text": " 4.36"}, {"start": 7508.96, "end": 7510.96, "text": " And remember the way we've arranged this"}, {"start": 7510.96, "end": 7514.96, "text": " Is that low loss means that our predictions are matching the targets"}, {"start": 7514.96, "end": 7518.96, "text": " So our predictions now are probably slightly closer to the targets"}, {"start": 7520.96, "end": 7524.96, "text": " And now all we have to do is we have to iterate this process"}, {"start": 7524.96, "end": 7526.96, "text": " So again we've done the forward pass"}, {"start": 7526.96, "end": 7528.96, "text": " And this is the loss"}, {"start": 7528.96, "end": 7530.96, "text": " Now we can lost that backward"}, {"start": 7530.96, "end": 7532.96, "text": " And we can do a step size"}, {"start": 7532.96, "end": 7536.96, "text": " And now we should have a slightly lower loss"}, {"start": 7536.96, "end": 7538.96, "text": " 4.36 goes to 3.9"}, {"start": 7538.96, "end": 7540.96, "text": " And okay so"}, {"start": 7540.96, "end": 7542.96, "text": " We've done the forward pass"}, {"start": 7542.96, "end": 7544.96, "text": " Here's the backward pass"}, {"start": 7544.96, "end": 7546.96, "text": " And now the loss is 3.66"}, {"start": 7546.96, "end": 7550.96, "text": " 3.47"}, {"start": 7550.96, "end": 7552.96, "text": " And you get the idea"}, {"start": 7552.96, "end": 7554.96, "text": " We just continue doing this"}, {"start": 7554.96, "end": 7556.96, "text": " And this is gradient descent"}, {"start": 7556.96, "end": 7558.96, "text": " We're just iteratively doing forward pass"}, {"start": 7558.96, "end": 7560.96, "text": " Backward pass"}, {"start": 7560.96, "end": 7562.96, "text": " Updates"}, {"start": 7562.96, "end": 7564.96, "text": " Forward pass, backward pass, update"}, {"start": 7564.96, "end": 7566.96, "text": " And the neural net is improving its predictions"}, {"start": 7566.96, "end": 7568.96, "text": " So here if we look at wide-pred now"}, {"start": 7568.96, "end": 7570.96, "text": " Wide-pred"}, {"start": 7570.96, "end": 7572.96, "text": " We see that"}, {"start": 7572.96, "end": 7576.96, "text": " This value should be getting closer to 1"}, {"start": 7576.96, "end": 7578.96, "text": " So this value should be getting more positive"}, {"start": 7578.96, "end": 7580.96, "text": " These should be getting more negative"}, {"start": 7580.96, "end": 7582.96, "text": " And this one should be also getting more positive"}, {"start": 7582.96, "end": 7584.96, "text": " So if we just iterate this"}, {"start": 7584.96, "end": 7586.96, "text": " A few more times"}, {"start": 7586.96, "end": 7588.96, "text": " Actually we'll be able to afford to go a bit faster"}, {"start": 7588.96, "end": 7590.96, "text": " Let's try a slightly higher learning rate"}, {"start": 7590.96, "end": 7594.96, "text": " Whoops, okay"}, {"start": 7594.96, "end": 7596.96, "text": " There we go, so now we're at 0.31"}, {"start": 7596.96, "end": 7600.96, "text": " If you go too fast by the way"}, {"start": 7600.96, "end": 7602.96, "text": " If you try to make it too big of a step"}, {"start": 7602.96, "end": 7604.96, "text": " You may actually overstep"}, {"start": 7604.96, "end": 7608.96, "text": " It's overconfidence"}, {"start": 7608.96, "end": 7610.96, "text": " Because again remember we don't actually know exactly about the loss function"}, {"start": 7610.96, "end": 7612.96, "text": " The loss function has all kinds of structure"}, {"start": 7612.96, "end": 7614.96, "text": " And we only know about the very local"}, {"start": 7614.96, "end": 7616.96, "text": " Dependence of all these parameters on the loss"}, {"start": 7616.96, "end": 7618.96, "text": " But if we step too far"}, {"start": 7618.96, "end": 7620.96, "text": " We may step into you know"}, {"start": 7620.96, "end": 7622.96, "text": " A part of the loss that is completely different"}, {"start": 7622.96, "end": 7624.96, "text": " And that can destabilize training"}, {"start": 7624.96, "end": 7626.96, "text": " And make your loss actually blow up even"}, {"start": 7626.96, "end": 7628.96, "text": " So the loss is now 0.04"}, {"start": 7628.96, "end": 7632.96, "text": " So actually the predictions should be really quite close"}, {"start": 7632.96, "end": 7634.96, "text": " Let's take a look"}, {"start": 7634.96, "end": 7636.96, "text": " So you see how this is almost 1"}, {"start": 7636.96, "end": 7638.96, "text": " Almost negative 1, almost 1"}, {"start": 7638.96, "end": 7640.96, "text": " We can continue going"}, {"start": 7640.96, "end": 7642.96, "text": " So, yep, backward, update"}, {"start": 7642.96, "end": 7644.96, "text": " Up, there we go"}, {"start": 7644.96, "end": 7646.96, "text": " So we went way too fast"}, {"start": 7646.96, "end": 7648.96, "text": " And we actually overstepped"}, {"start": 7648.96, "end": 7650.96, "text": " So we got to 2, 2 eager"}, {"start": 7650.96, "end": 7652.96, "text": " Where are we now?"}, {"start": 7652.96, "end": 7654.96, "text": " Oops"}, {"start": 7654.96, "end": 7656.96, "text": " Okay, 7 in negative 9"}, {"start": 7656.96, "end": 7658.96, "text": " So this is very, very low loss"}, {"start": 7658.96, "end": 7662.96, "text": " And the predictions are basically perfect"}, {"start": 7662.96, "end": 7666.96, "text": " So somehow we were doing way too big updates"}, {"start": 7666.96, "end": 7668.96, "text": " And we briefly explored it"}, {"start": 7668.96, "end": 7670.96, "text": " But then somehow we ended up getting into a really good spot"}, {"start": 7670.96, "end": 7674.96, "text": " So usually this learning rate and the tuning of it is a subtle art"}, {"start": 7674.96, "end": 7676.96, "text": " You want to set your learning rate"}, {"start": 7676.96, "end": 7678.96, "text": " If it's too low, you're going to take way too long to converge"}, {"start": 7678.96, "end": 7680.96, "text": " But if it's too high, the whole thing gets unstable"}, {"start": 7680.96, "end": 7682.96, "text": " And you might actually even explode the loss"}, {"start": 7682.96, "end": 7684.96, "text": " Depending on your loss function"}, {"start": 7684.96, "end": 7686.96, "text": " So finding the step size to be just right"}, {"start": 7686.96, "end": 7688.96, "text": " It's a pretty subtle art sometimes"}, {"start": 7688.96, "end": 7690.96, "text": " When you're using sort of a null-accradient descent"}, {"start": 7690.96, "end": 7692.96, "text": " But we happen to get into a good spot"}, {"start": 7692.96, "end": 7694.96, "text": " We can look at end dot parameters"}, {"start": 7694.96, "end": 7696.96, "text": " And we can see that the result is really good"}, {"start": 7696.96, "end": 7700.96, "text": " So we can see that we have a very good spot"}, {"start": 7700.96, "end": 7702.96, "text": " We can look at end dot parameters"}, {"start": 7702.96, "end": 7706.96, "text": " So this is the setting of weights and biases"}, {"start": 7706.96, "end": 7712.96, "text": " That makes our network predict the desired targets very, very close"}, {"start": 7712.96, "end": 7718.96, "text": " And basically we successfully trained in neural nut"}, {"start": 7718.96, "end": 7720.96, "text": " Okay, let's make this a tiny bit more respectable"}, {"start": 7720.96, "end": 7722.96, "text": " And implement an actual training loop"}, {"start": 7722.96, "end": 7724.96, "text": " And what that looks like"}, {"start": 7724.96, "end": 7726.96, "text": " is the initialization of that state"}, {"start": 7726.96, "end": 7728.96, "text": " This is the forward pass"}, {"start": 7728.96, "end": 7730.96, "text": " So for K in range"}, {"start": 7730.96, "end": 7734.96, "text": " We're going to take a bunch of steps"}, {"start": 7736.96, "end": 7738.96, "text": " First, you do the forward pass"}, {"start": 7738.96, "end": 7740.96, "text": " We evaluate the loss"}, {"start": 7742.96, "end": 7744.96, "text": " Let's re-initialize the neural nut from scratch"}, {"start": 7744.96, "end": 7746.96, "text": " And here's the data"}, {"start": 7746.96, "end": 7750.96, "text": " And we first do forward pass"}, {"start": 7750.96, "end": 7752.96, "text": " Then we do the backward pass"}, {"start": 7752.96, "end": 7754.96, "text": " And then we do an update"}, {"start": 7754.96, "end": 7756.96, "text": " That's gradient descent"}, {"start": 7758.96, "end": 7760.96, "text": " And then we do an update"}, {"start": 7760.96, "end": 7762.96, "text": " That's gradient descent"}, {"start": 7766.96, "end": 7768.96, "text": " And then we should be able to iterate this"}, {"start": 7768.96, "end": 7770.96, "text": " And we should be able to print the current step"}, {"start": 7770.96, "end": 7772.96, "text": " The current loss"}, {"start": 7772.96, "end": 7774.96, "text": " Let's just print the sort of"}, {"start": 7774.96, "end": 7776.96, "text": " Number of the loss"}, {"start": 7776.96, "end": 7778.96, "text": " And that should be it"}, {"start": 7778.96, "end": 7780.96, "text": " And then the learning rate"}, {"start": 7780.96, "end": 7782.96, "text": " 0.01 is a little too small"}, {"start": 7782.96, "end": 7784.96, "text": " 0.1 we saw is like a little bit dangerous if you buy"}, {"start": 7784.96, "end": 7786.96, "text": " Let's go somewhere between"}, {"start": 7786.96, "end": 7788.96, "text": " And we'll optimize this for"}, {"start": 7788.96, "end": 7790.96, "text": " Not 10 steps"}, {"start": 7790.96, "end": 7792.96, "text": " But let's go for say 20 steps"}, {"start": 7792.96, "end": 7796.96, "text": " Let me erase all of this junk"}, {"start": 7796.96, "end": 7800.96, "text": " And let's run the optimization"}, {"start": 7800.96, "end": 7804.96, "text": " And you see how we've actually converged slower"}, {"start": 7804.96, "end": 7806.96, "text": " In a more controlled manner"}, {"start": 7806.96, "end": 7808.96, "text": " And got through a loss that is very low"}, {"start": 7808.96, "end": 7810.96, "text": " So I expect wide-pred to be quite good"}, {"start": 7810.96, "end": 7812.96, "text": " There we go"}, {"start": 7816.96, "end": 7818.96, "text": " And that's it"}, {"start": 7818.96, "end": 7820.96, "text": " Okay, so this is kind of embarrassing"}, {"start": 7820.96, "end": 7822.96, "text": " But we actually have a really terrible bug"}, {"start": 7822.96, "end": 7824.96, "text": " In here"}, {"start": 7824.96, "end": 7826.96, "text": " And it's a subtle bug"}, {"start": 7826.96, "end": 7828.96, "text": " And it's a very common bug"}, {"start": 7828.96, "end": 7830.96, "text": " And I can't believe I've done it"}, {"start": 7830.96, "end": 7832.96, "text": " For the 20th time in my life"}, {"start": 7832.96, "end": 7834.96, "text": " Especially on camera"}, {"start": 7834.96, "end": 7836.96, "text": " And I could have reshot the bug"}, {"start": 7836.96, "end": 7838.96, "text": " And I could have reshot the whole thing"}, {"start": 7838.96, "end": 7840.96, "text": " But I think it's pretty funny"}, {"start": 7840.96, "end": 7842.96, "text": " And you know, you get to appreciate a bit"}, {"start": 7842.96, "end": 7844.96, "text": " What working with neural nuts maybe"}, {"start": 7844.96, "end": 7846.96, "text": " Is like sometimes"}, {"start": 7846.96, "end": 7848.96, "text": " We are guilty of"}, {"start": 7848.96, "end": 7850.96, "text": " A common bug"}, {"start": 7850.96, "end": 7852.96, "text": " I've actually tweeted"}, {"start": 7852.96, "end": 7854.96, "text": " The most common neural nut mistakes"}, {"start": 7854.96, "end": 7856.96, "text": " A long time ago now"}, {"start": 7856.96, "end": 7858.96, "text": " And I'm not really"}, {"start": 7858.96, "end": 7860.96, "text": " Gonna explain any of these"}, {"start": 7860.96, "end": 7862.96, "text": " Except for we are guilty of number three"}, {"start": 7862.96, "end": 7864.96, "text": " You forgot to zero grad"}, {"start": 7864.96, "end": 7866.96, "text": " Before dot backward"}, {"start": 7866.96, "end": 7868.96, "text": " What is that?"}, {"start": 7868.96, "end": 7870.96, "text": " Basically what's happening"}, {"start": 7870.96, "end": 7872.96, "text": " And it's a subtle bug and I'm not sure if you saw it"}, {"start": 7872.96, "end": 7874.96, "text": " Is that"}, {"start": 7874.96, "end": 7876.96, "text": " All of these"}, {"start": 7876.96, "end": 7878.96, "text": " Wates here have a dot data and a dot grad"}, {"start": 7878.96, "end": 7880.96, "text": " And dot grad starts at zero"}, {"start": 7880.96, "end": 7882.96, "text": " And then we do backward"}, {"start": 7882.96, "end": 7884.96, "text": " And we fill in the gradients"}, {"start": 7884.96, "end": 7886.96, "text": " And then we do an update on the data"}, {"start": 7886.96, "end": 7888.96, "text": " But we don't flush the grad"}, {"start": 7888.96, "end": 7890.96, "text": " It stays there"}, {"start": 7890.96, "end": 7892.96, "text": " So when we do the second"}, {"start": 7892.96, "end": 7894.96, "text": " Forward pass and we do backward again"}, {"start": 7894.96, "end": 7896.96, "text": " Remember that all the backward operations"}, {"start": 7896.96, "end": 7898.96, "text": " Do a plus equals on the grad"}, {"start": 7898.96, "end": 7900.96, "text": " And so these gradients"}, {"start": 7900.96, "end": 7904.96, "text": " Just add up and they never get reset to zero"}, {"start": 7904.96, "end": 7906.96, "text": " So basically we didn't zero grad"}, {"start": 7906.96, "end": 7908.96, "text": " So here's how we zero grad"}, {"start": 7908.96, "end": 7910.96, "text": " Before backward"}, {"start": 7910.96, "end": 7912.96, "text": " We need to iterate over all the parameters"}, {"start": 7912.96, "end": 7914.96, "text": " And we need to make sure that"}, {"start": 7914.96, "end": 7916.96, "text": " P dot grad is set to zero"}, {"start": 7916.96, "end": 7918.96, "text": " We need to reset it to zero"}, {"start": 7918.96, "end": 7920.96, "text": " Just like it is in the constructor"}, {"start": 7920.96, "end": 7922.96, "text": " So remember all the way here"}, {"start": 7922.96, "end": 7924.96, "text": " For all these value nodes"}, {"start": 7924.96, "end": 7926.96, "text": " Grad is reset to zero"}, {"start": 7926.96, "end": 7930.96, "text": " And then all these backward passes do a plus equals on that grad"}, {"start": 7930.96, "end": 7932.96, "text": " But we need to make sure that"}, {"start": 7932.96, "end": 7934.96, "text": " We reset these grads to zero"}, {"start": 7934.96, "end": 7936.96, "text": " So that when we do backward"}, {"start": 7936.96, "end": 7938.96, "text": " All of them start at zero"}, {"start": 7938.96, "end": 7940.96, "text": " And the actual backward pass accumulates"}, {"start": 7940.96, "end": 7944.96, "text": " The loss derivatives into the grads"}, {"start": 7944.96, "end": 7946.96, "text": " So this is zero grad in PyTorch"}, {"start": 7946.96, "end": 7948.96, "text": " And we will slightly"}, {"start": 7948.96, "end": 7952.96, "text": " We will get a slightly different optimization"}, {"start": 7952.96, "end": 7954.96, "text": " Let's reset the neural net"}, {"start": 7954.96, "end": 7956.96, "text": " The data is the same"}, {"start": 7956.96, "end": 7958.96, "text": " This is now I think correct"}, {"start": 7958.96, "end": 7960.96, "text": " And we get a much more"}, {"start": 7960.96, "end": 7962.96, "text": " You know we get a much more slower descent"}, {"start": 7962.96, "end": 7964.96, "text": " We still end up with pretty good results"}, {"start": 7964.96, "end": 7966.96, "text": " And we can continue this a bit more"}, {"start": 7966.96, "end": 7968.96, "text": " To get down lower"}, {"start": 7968.96, "end": 7970.96, "text": " And lower"}, {"start": 7970.96, "end": 7972.96, "text": " And lower"}, {"start": 7972.96, "end": 7974.96, "text": " Yeah"}, {"start": 7974.96, "end": 7976.96, "text": " So the only reason that the previous thing worked"}, {"start": 7976.96, "end": 7978.96, "text": " It's extremely buggy"}, {"start": 7978.96, "end": 7980.96, "text": " The only reason that worked"}, {"start": 7980.96, "end": 7982.96, "text": " Is that"}, {"start": 7982.96, "end": 7984.96, "text": " This is a very very simple problem"}, {"start": 7984.96, "end": 7988.96, "text": " And it's very easy for this neural net to fit this data"}, {"start": 7988.96, "end": 7990.96, "text": " And so the grads ended up accumulating"}, {"start": 7990.96, "end": 7994.96, "text": " And it effectively gave us a massive step size"}, {"start": 7994.96, "end": 7996.96, "text": " And it made us converge extremely fast"}, {"start": 7996.96, "end": 8000.96, "text": " But basically now we have to do more steps"}, {"start": 8000.96, "end": 8002.96, "text": " To get to very low values of loss"}, {"start": 8002.96, "end": 8004.96, "text": " And get why I pray to be really good"}, {"start": 8004.96, "end": 8006.96, "text": " We can try to step a bit greater"}, {"start": 8010.96, "end": 8012.96, "text": " Yeah"}, {"start": 8012.96, "end": 8016.96, "text": " We're going to get closer and closer to one minus one"}, {"start": 8016.96, "end": 8018.96, "text": " And one"}, {"start": 8018.96, "end": 8022.96, "text": " So we're going to do lots of sometimes tricky"}, {"start": 8022.96, "end": 8024.96, "text": " Because"}, {"start": 8024.96, "end": 8026.96, "text": " You may have lots of bugs in the code"}, {"start": 8026.96, "end": 8028.96, "text": " And your network might actually work"}, {"start": 8028.96, "end": 8030.96, "text": " Just like ours worked"}, {"start": 8030.96, "end": 8032.96, "text": " But chances are is that"}, {"start": 8032.96, "end": 8034.96, "text": " We had a more complex problem"}, {"start": 8034.96, "end": 8038.96, "text": " Than actually this bug would have made us not optimize the loss very well"}, {"start": 8038.96, "end": 8040.96, "text": " And we were only able to get away with it because"}, {"start": 8040.96, "end": 8042.96, "text": " The problem is very simple"}, {"start": 8042.96, "end": 8044.96, "text": " So let's now bring everything together"}, {"start": 8044.96, "end": 8046.96, "text": " And summarize what we learned"}, {"start": 8046.96, "end": 8048.96, "text": " What are neural nets?"}, {"start": 8048.96, "end": 8050.96, "text": " Neural nets are these mathematical expressions"}, {"start": 8050.96, "end": 8052.96, "text": " They're really simple mathematical expressions"}, {"start": 8052.96, "end": 8054.96, "text": " In case of multi-layer perceptron"}, {"start": 8054.96, "end": 8056.96, "text": " That take"}, {"start": 8056.96, "end": 8058.96, "text": " Input as the data"}, {"start": 8058.96, "end": 8060.96, "text": " And they take input the weights and the parameters of the neural net"}, {"start": 8060.96, "end": 8062.96, "text": " Mathematical expression for the forward pass"}, {"start": 8062.96, "end": 8064.96, "text": " Followed by a loss function"}, {"start": 8064.96, "end": 8068.96, "text": " And the loss function tries to measure the accuracy of the predictions"}, {"start": 8068.96, "end": 8070.96, "text": " And usually the loss will be low"}, {"start": 8070.96, "end": 8072.96, "text": " When your predictions are matching your targets"}, {"start": 8072.96, "end": 8074.96, "text": " Or where the network is basically behaving well"}, {"start": 8074.96, "end": 8078.96, "text": " So we manipulate the loss function so that when the loss is low"}, {"start": 8078.96, "end": 8082.96, "text": " The network is doing what you wanted to do on your problem"}, {"start": 8082.96, "end": 8084.96, "text": " And then we backward the loss"}, {"start": 8084.96, "end": 8086.96, "text": " Use back propagation to get the gradient"}, {"start": 8086.96, "end": 8088.96, "text": " And then we know how to tune all the parameters to"}, {"start": 8088.96, "end": 8090.96, "text": " Decrease the loss locally"}, {"start": 8090.96, "end": 8092.96, "text": " But then we have to iterate that process many times"}, {"start": 8092.96, "end": 8094.96, "text": " In what's called the gradient descent"}, {"start": 8094.96, "end": 8096.96, "text": " So we simply follow the gradient information"}, {"start": 8096.96, "end": 8098.96, "text": " And that minimizes the loss"}, {"start": 8098.96, "end": 8100.96, "text": " And the loss is arranged so that when the loss is minimized"}, {"start": 8100.96, "end": 8102.96, "text": " The network is doing what you want it to do"}, {"start": 8102.96, "end": 8106.96, "text": " And yeah, so we just have a blob of neural stuff"}, {"start": 8106.96, "end": 8108.96, "text": " And we can make it do arbitrary things"}, {"start": 8108.96, "end": 8110.96, "text": " And that's what gives neural nets their power"}, {"start": 8110.96, "end": 8114.96, "text": " It's, you know, this is a very tiny network with 41 parameters"}, {"start": 8114.96, "end": 8118.96, "text": " But you can build significantly more complicated neural nets"}, {"start": 8118.96, "end": 8122.96, "text": " With billions at this point, almost trillions of parameters"}, {"start": 8122.96, "end": 8124.96, "text": " And it's a massive blob of neural tissue"}, {"start": 8124.96, "end": 8126.96, "text": " Simulated neural tissue"}, {"start": 8126.96, "end": 8128.96, "text": " Roughly speaking"}, {"start": 8128.96, "end": 8132.96, "text": " And you can make it do extremely complex problems"}, {"start": 8132.96, "end": 8134.96, "text": " And these neural nets then have all kinds of very fascinating"}, {"start": 8134.96, "end": 8136.96, "text": " Emergent properties"}, {"start": 8136.96, "end": 8140.96, "text": " In when you try to make them do significantly hard problems"}, {"start": 8140.96, "end": 8142.96, "text": " So you can make it more difficult"}, {"start": 8142.96, "end": 8144.96, "text": " To do significantly hard problems"}, {"start": 8144.96, "end": 8146.96, "text": " As in the case of GPT for example"}, {"start": 8146.96, "end": 8150.96, "text": " We have massive amounts of text from the internet"}, {"start": 8150.96, "end": 8152.96, "text": " And we're trying to get a neural net to predict"}, {"start": 8152.96, "end": 8154.96, "text": " To take like a few words"}, {"start": 8154.96, "end": 8156.96, "text": " And try to predict the next word in a sequence"}, {"start": 8156.96, "end": 8158.96, "text": " That's the learning problem"}, {"start": 8158.96, "end": 8160.96, "text": " And it turns out that when you train this on all of the internet"}, {"start": 8160.96, "end": 8162.96, "text": " The neural net actually has like really remarkable"}, {"start": 8162.96, "end": 8164.96, "text": " Emergent properties"}, {"start": 8164.96, "end": 8166.96, "text": " But that neural net would have hundreds of billions of parameters"}, {"start": 8166.96, "end": 8170.96, "text": " But it works on fundamentally these exact same principles"}, {"start": 8170.96, "end": 8172.96, "text": " And the neural net of course will be a bit more complex"}, {"start": 8172.96, "end": 8174.96, "text": " But otherwise the"}, {"start": 8174.96, "end": 8176.96, "text": " Evaluating the gradient is there"}, {"start": 8176.96, "end": 8178.96, "text": " And it would be identical"}, {"start": 8178.96, "end": 8180.96, "text": " And the gradient descent would be there"}, {"start": 8180.96, "end": 8182.96, "text": " And would be basically identical"}, {"start": 8182.96, "end": 8184.96, "text": " But people usually use slightly different updates"}, {"start": 8184.96, "end": 8188.96, "text": " This is a very simple stochastic gradient sent update"}, {"start": 8188.96, "end": 8190.96, "text": " And the last function would not be an eSquared error"}, {"start": 8190.96, "end": 8192.96, "text": " They would be using something called the cross-entropy loss"}, {"start": 8192.96, "end": 8194.96, "text": " For predicting the next token"}, {"start": 8194.96, "end": 8196.96, "text": " So there's a few more details"}, {"start": 8196.96, "end": 8198.96, "text": " But fundamentally the neural network setup"}, {"start": 8198.96, "end": 8200.96, "text": " Or training is identical and pervasive"}, {"start": 8200.96, "end": 8202.96, "text": " And now you understand intuitively"}, {"start": 8202.96, "end": 8204.96, "text": " How that works under the hood"}, {"start": 8204.96, "end": 8206.96, "text": " In the beginning of this video"}, {"start": 8206.96, "end": 8208.96, "text": " I told you that by the end of it"}, {"start": 8208.96, "end": 8210.96, "text": " You would understand everything in micrograd"}, {"start": 8210.96, "end": 8212.96, "text": " And then we'd slowly build it up"}, {"start": 8212.96, "end": 8214.96, "text": " Let me briefly prove that to you"}, {"start": 8214.96, "end": 8216.96, "text": " So I'm going to start through all the code that is in micrograd"}, {"start": 8216.96, "end": 8218.96, "text": " As of today"}, {"start": 8218.96, "end": 8220.96, "text": " Actually, potentially some of the code will change"}, {"start": 8220.96, "end": 8222.96, "text": " By the time you watch this video"}, {"start": 8222.96, "end": 8224.96, "text": " Because I intend to continue developing micrograd"}, {"start": 8224.96, "end": 8226.96, "text": " But let's look at what we have so far at least"}, {"start": 8226.96, "end": 8228.96, "text": " When you go to engine.py that has the value"}, {"start": 8228.96, "end": 8230.96, "text": " Everything here you should mostly recognize"}, {"start": 8230.96, "end": 8232.96, "text": " So we have the dead.data.grad attributes"}, {"start": 8232.96, "end": 8234.96, "text": " We have the backward function"}, {"start": 8234.96, "end": 8236.96, "text": " We have the previous set of children"}, {"start": 8236.96, "end": 8238.96, "text": " And the operation that produced this value"}, {"start": 8238.96, "end": 8242.96, "text": " We have addition, multiplication, and raising to a scalar power"}, {"start": 8242.96, "end": 8244.96, "text": " We have the Rellou nonlinearity"}, {"start": 8244.96, "end": 8246.96, "text": " Which is slightly different type of nonlinearity than 10H"}, {"start": 8246.96, "end": 8248.96, "text": " That we used in this video"}, {"start": 8248.96, "end": 8250.96, "text": " Both of them are nonlinearties"}, {"start": 8250.96, "end": 8252.96, "text": " And notably 10H is not actually present in micrograd"}, {"start": 8252.96, "end": 8254.96, "text": " As of right now"}, {"start": 8254.96, "end": 8256.96, "text": " But I intend to add it later"}, {"start": 8256.96, "end": 8258.96, "text": " With the backward, which is identical"}, {"start": 8258.96, "end": 8260.96, "text": " And then all of these other operations"}, {"start": 8260.96, "end": 8262.96, "text": " Which are built up on top of operations here"}, {"start": 8262.96, "end": 8264.96, "text": " So value should be very recognizable"}, {"start": 8264.96, "end": 8266.96, "text": " Except for the nonlinearity used in this video"}, {"start": 8266.96, "end": 8268.96, "text": " There's no massive difference between Rellou and 10H"}, {"start": 8268.96, "end": 8270.96, "text": " And sigmoid and these other nonlinearties"}, {"start": 8270.96, "end": 8272.96, "text": " They're all roughly equivalent and can be used in MLPs"}, {"start": 8272.96, "end": 8274.96, "text": " So I use 10H because it's a bit smoother"}, {"start": 8274.96, "end": 8276.96, "text": " And because it's a little bit more complicated than Rellou"}, {"start": 8276.96, "end": 8278.96, "text": " And therefore it's stressed a little bit more"}, {"start": 8278.96, "end": 8280.96, "text": " And therefore it's stressed a little bit more"}, {"start": 8280.96, "end": 8282.96, "text": " And therefore it's stressed a little bit more"}, {"start": 8282.96, "end": 8284.96, "text": " The local gradients and working with those derivatives"}, {"start": 8284.96, "end": 8286.96, "text": " Which I probably would be useful"}, {"start": 8286.96, "end": 8290.96, "text": " And then the pi is the neural networks library as I mentioned"}, {"start": 8290.96, "end": 8292.96, "text": " So you should recognize identical implementation of your own"}, {"start": 8292.96, "end": 8294.96, "text": " Layer and MLP"}, {"start": 8294.96, "end": 8296.96, "text": " Notably, for not so much"}, {"start": 8296.96, "end": 8298.96, "text": " We have a class module here"}, {"start": 8298.96, "end": 8300.96, "text": " There's a parent class of all these modules"}, {"start": 8300.96, "end": 8304.96, "text": " I did that because there's an end-up module class in PyTorch"}, {"start": 8304.96, "end": 8306.96, "text": " And so this exactly matches that API"}, {"start": 8306.96, "end": 8310.96, "text": " And end-up module in PyTorch has also a zero-brad"}, {"start": 8310.96, "end": 8312.96, "text": " Which I refactored out here"}, {"start": 8312.96, "end": 8314.96, "text": " So that's the end of micro-grad really"}, {"start": 8314.96, "end": 8316.96, "text": " Then there's a test"}, {"start": 8316.96, "end": 8318.96, "text": " Which you'll see basically creates two chunks of code"}, {"start": 8318.96, "end": 8320.96, "text": " One in micro-grad and one in PyTorch"}, {"start": 8320.96, "end": 8324.96, "text": " And we'll make sure that the forward and the backward pass agree identically"}, {"start": 8324.96, "end": 8326.96, "text": " For a slightly less complicated expression"}, {"start": 8326.96, "end": 8328.96, "text": " A slightly more complicated expression"}, {"start": 8328.96, "end": 8330.96, "text": " Everything agrees"}, {"start": 8330.96, "end": 8332.96, "text": " So we agree with PyTorch on all of these operations"}, {"start": 8332.96, "end": 8334.96, "text": " And finally there's a demo that IpyY"}, {"start": 8334.96, "end": 8336.96, "text": " And finally there's a demo that IpyY"}, {"start": 8336.96, "end": 8338.96, "text": " And finally there's a demo that IpyY"}, {"start": 8338.96, "end": 8340.96, "text": " And finally there's a demo that IpyY and B here"}, {"start": 8340.96, "end": 8344.96, "text": " And it's a bit more complicated binary classification demo than the one I covered in this lecture"}, {"start": 8344.96, "end": 8348.96, "text": " So we only had a tiny data set of for examples"}, {"start": 8348.96, "end": 8350.96, "text": " Here we have a bit more complicated example"}, {"start": 8350.96, "end": 8352.96, "text": " With lots of blue points and lots of red points"}, {"start": 8352.96, "end": 8356.96, "text": " And we're trying to again build a binary classifier to distinguish"}, {"start": 8356.96, "end": 8358.96, "text": " Two dimensional points as red or blue"}, {"start": 8358.96, "end": 8362.96, "text": " It's a bit more complicated MLP here with it's a bigger MLP"}, {"start": 8362.96, "end": 8364.96, "text": " The loss is a bit more complicated because"}, {"start": 8364.96, "end": 8366.96, "text": " It supports batches"}, {"start": 8366.96, "end": 8368.96, "text": " It supports batches"}, {"start": 8368.96, "end": 8370.96, "text": " So because our data set was so tiny"}, {"start": 8370.96, "end": 8374.96, "text": " We always did a forward pass on the entire data set of for examples"}, {"start": 8374.96, "end": 8376.96, "text": " But when your data set is like a million examples"}, {"start": 8376.96, "end": 8378.96, "text": " What we usually do in practice is we"}, {"start": 8378.96, "end": 8380.96, "text": " We basically pick out some random subset"}, {"start": 8380.96, "end": 8382.96, "text": " We call that a batch"}, {"start": 8382.96, "end": 8384.96, "text": " And then we only process the batch"}, {"start": 8384.96, "end": 8386.96, "text": " Forward, backward, and update"}, {"start": 8386.96, "end": 8388.96, "text": " So we don't have to forward the entire training set"}, {"start": 8388.96, "end": 8390.96, "text": " So this supports batching"}, {"start": 8390.96, "end": 8392.96, "text": " Because there's a lot more examples here"}, {"start": 8392.96, "end": 8394.96, "text": " We do a forward pass"}, {"start": 8394.96, "end": 8396.96, "text": " The loss is slightly more different"}, {"start": 8396.96, "end": 8398.96, "text": " This is a max margin loss that I implement here"}, {"start": 8398.96, "end": 8402.96, "text": " The one that we used was the mean squared error loss"}, {"start": 8402.96, "end": 8404.96, "text": " Because it's the simplest one"}, {"start": 8404.96, "end": 8406.96, "text": " There's also the binary cross-entropy loss"}, {"start": 8406.96, "end": 8408.96, "text": " All of them can be used for binary classification"}, {"start": 8408.96, "end": 8410.96, "text": " And don't make too much of a difference"}, {"start": 8410.96, "end": 8412.96, "text": " In the simple examples that we looked at so far"}, {"start": 8412.96, "end": 8414.96, "text": " There's something called"}, {"start": 8414.96, "end": 8416.96, "text": " Alt-to-regularization used here"}, {"start": 8416.96, "end": 8418.96, "text": " This has to do with generalization of the neural net"}, {"start": 8418.96, "end": 8422.96, "text": " And controls the overfitting in machine learning setting"}, {"start": 8422.96, "end": 8424.96, "text": " So I did not cover these concepts in this video"}, {"start": 8424.96, "end": 8426.96, "text": " Potentially later"}, {"start": 8426.96, "end": 8428.96, "text": " And the training loop you should recognize"}, {"start": 8428.96, "end": 8430.96, "text": " So forward, backward, with, zero grad"}, {"start": 8430.96, "end": 8432.96, "text": " And update, and so on"}, {"start": 8432.96, "end": 8434.96, "text": " You'll notice that in the update here"}, {"start": 8434.96, "end": 8438.96, "text": " The learning rate is scaled as a function of number of iterations"}, {"start": 8438.96, "end": 8440.96, "text": " And it shrinks"}, {"start": 8440.96, "end": 8442.96, "text": " And this is something called learning rate decay"}, {"start": 8442.96, "end": 8444.96, "text": " So in the beginning you have a high learning rate"}, {"start": 8444.96, "end": 8448.96, "text": " And as the network sort of stabilizes near the end"}, {"start": 8448.96, "end": 8450.96, "text": " You bring down the learning rate to get some of the fine details in the end"}, {"start": 8450.96, "end": 8454.96, "text": " And in the end we see the decision surface of the neural net"}, {"start": 8454.96, "end": 8458.96, "text": " And we see that it learns to separate out the red and the blue area"}, {"start": 8458.96, "end": 8460.96, "text": " Based on the data points"}, {"start": 8460.96, "end": 8462.96, "text": " So that's a slightly more complicated example"}, {"start": 8462.96, "end": 8464.96, "text": " And the demo that I buy by YMB"}, {"start": 8464.96, "end": 8466.96, "text": " That you're free to go over"}, {"start": 8466.96, "end": 8468.96, "text": " But yeah, as of today, that is micro-grad"}, {"start": 8468.96, "end": 8470.96, "text": " I also wanted to show you a little bit of real stuff"}, {"start": 8470.96, "end": 8472.96, "text": " So that you get to see how this is actually implemented"}, {"start": 8472.96, "end": 8474.96, "text": " In the production grade library like by torch"}, {"start": 8474.96, "end": 8476.96, "text": " So in particular, I wanted to show"}, {"start": 8476.96, "end": 8478.96, "text": " I wanted to find and show you"}, {"start": 8478.96, "end": 8480.96, "text": " The backward password 10H in PyTorch"}, {"start": 8480.96, "end": 8482.96, "text": " So here in micro-grad"}, {"start": 8482.96, "end": 8484.96, "text": " We see that the backward password 10H is"}, {"start": 8484.96, "end": 8486.96, "text": " 1-t square"}, {"start": 8486.96, "end": 8488.96, "text": " Where t is the output of the 10H of X"}, {"start": 8488.96, "end": 8490.96, "text": " Times of that grad which is the chain rule"}, {"start": 8490.96, "end": 8494.96, "text": " So we're looking for something that looks like this"}, {"start": 8494.96, "end": 8498.96, "text": " Now I went to PyTorch"}, {"start": 8498.96, "end": 8500.96, "text": " Which has open source GitHub code base"}, {"start": 8500.96, "end": 8502.96, "text": " And I looked through a lot of its code"}, {"start": 8502.96, "end": 8504.96, "text": " And honestly, I wanted to show you"}, {"start": 8504.96, "end": 8508.96, "text": " I looked through a lot of its code and honestly"}, {"start": 8508.96, "end": 8510.96, "text": " I spent about 15 minutes and I couldn't find 10H"}, {"start": 8510.96, "end": 8512.96, "text": " And that's because these libraries"}, {"start": 8512.96, "end": 8514.96, "text": " Unfortunately, they grow in size and entropy"}, {"start": 8514.96, "end": 8516.96, "text": " And if you just search for 10H"}, {"start": 8516.96, "end": 8518.96, "text": " You get apparently 2,800 results"}, {"start": 8518.96, "end": 8520.96, "text": " And 400 and 406 files"}, {"start": 8520.96, "end": 8524.96, "text": " So I don't know what these files are doing honestly"}, {"start": 8524.96, "end": 8528.96, "text": " And why there are so many mentions of 10H"}, {"start": 8528.96, "end": 8530.96, "text": " But unfortunately, these libraries are quite complex"}, {"start": 8530.96, "end": 8532.96, "text": " They're meant to be used"}, {"start": 8532.96, "end": 8534.96, "text": " Not really inspected"}, {"start": 8534.96, "end": 8536.96, "text": " Eventually, I did stumble on someone"}, {"start": 8536.96, "end": 8540.96, "text": " Who tries to change the 10H"}, {"start": 8540.96, "end": 8542.96, "text": " Backward code for some reason"}, {"start": 8542.96, "end": 8544.96, "text": " And someone here pointed to the CPU kernel"}, {"start": 8544.96, "end": 8546.96, "text": " And the CUDA kernel for 10H backward"}, {"start": 8546.96, "end": 8550.96, "text": " So this basically depends on if you're using PyTorch"}, {"start": 8550.96, "end": 8552.96, "text": " On the CPU device or on the GPU"}, {"start": 8552.96, "end": 8554.96, "text": " Which these are different devices and I haven't covered this"}, {"start": 8554.96, "end": 8556.96, "text": " But this is the 10H backward kernel"}, {"start": 8556.96, "end": 8558.96, "text": " For CPU"}, {"start": 8558.96, "end": 8560.96, "text": " And the reason it's so large"}, {"start": 8560.96, "end": 8562.96, "text": " Is that a number one"}, {"start": 8562.96, "end": 8564.96, "text": " This is like if you're using a complex type"}, {"start": 8564.96, "end": 8566.96, "text": " Which we haven't even talked about"}, {"start": 8566.96, "end": 8568.96, "text": " If you're using a specific data type of B-float 16"}, {"start": 8568.96, "end": 8570.96, "text": " Which we haven't talked about"}, {"start": 8570.96, "end": 8572.96, "text": " And then if you're not"}, {"start": 8572.96, "end": 8574.96, "text": " Then this is the kernel and deep here"}, {"start": 8574.96, "end": 8576.96, "text": " We see something that resembles our backward pass"}, {"start": 8576.96, "end": 8580.96, "text": " So they have 8 times 1 minus B-square"}, {"start": 8580.96, "end": 8582.96, "text": " So this B here"}, {"start": 8582.96, "end": 8584.96, "text": " Must be the output of the 10H"}, {"start": 8584.96, "end": 8586.96, "text": " And this is the out.grad"}, {"start": 8586.96, "end": 8588.96, "text": " So here we found it"}, {"start": 8588.96, "end": 8590.96, "text": " Deep inside"}, {"start": 8590.96, "end": 8592.96, "text": " PyTorch and this location"}, {"start": 8592.96, "end": 8594.96, "text": " For some reason inside binary ops kernel"}, {"start": 8594.96, "end": 8596.96, "text": " When 10H is not actually binary op"}, {"start": 8596.96, "end": 8600.96, "text": " And then this is the GPU kernel"}, {"start": 8600.96, "end": 8602.96, "text": " We're not complex"}, {"start": 8602.96, "end": 8604.96, "text": " We're here"}, {"start": 8604.96, "end": 8606.96, "text": " And here we go with online opcode"}, {"start": 8606.96, "end": 8608.96, "text": " So we did find it"}, {"start": 8608.96, "end": 8610.96, "text": " But basically unfortunately"}, {"start": 8610.96, "end": 8612.96, "text": " These code bases are very large"}, {"start": 8612.96, "end": 8614.96, "text": " And micrograd is very very simple"}, {"start": 8614.96, "end": 8616.96, "text": " But if you actually"}, {"start": 8616.96, "end": 8618.96, "text": " Want to use real stuff"}, {"start": 8618.96, "end": 8620.96, "text": " Finding the code for it"}, {"start": 8620.96, "end": 8622.96, "text": " You'll actually find that difficult"}, {"start": 8622.96, "end": 8624.96, "text": " I also wanted to show you"}, {"start": 8624.96, "end": 8626.96, "text": " The whole example here"}, {"start": 8626.96, "end": 8628.96, "text": " Where PyTorch is showing you how you can"}, {"start": 8628.96, "end": 8630.96, "text": " Register a new type of function"}, {"start": 8630.96, "end": 8632.96, "text": " That you want to add to PyTorch"}, {"start": 8632.96, "end": 8634.96, "text": " As a LEGO building block"}, {"start": 8634.96, "end": 8636.96, "text": " So here if you want to for example add"}, {"start": 8636.96, "end": 8638.96, "text": " A like jonder polynomial"}, {"start": 8638.96, "end": 8640.96, "text": " 3"}, {"start": 8640.96, "end": 8642.96, "text": " Here's how you can do it"}, {"start": 8642.96, "end": 8644.96, "text": " You will register it"}, {"start": 8644.96, "end": 8646.96, "text": " And then you have to tell PyTorch"}, {"start": 8646.96, "end": 8648.96, "text": " How to forward your new function"}, {"start": 8648.96, "end": 8650.96, "text": " And how to backward through it"}, {"start": 8650.96, "end": 8652.96, "text": " So as long as you can do the forward pass"}, {"start": 8652.96, "end": 8654.96, "text": " Of this little function piece that you want to add"}, {"start": 8654.96, "end": 8656.96, "text": " And as long as you know"}, {"start": 8656.96, "end": 8658.96, "text": " The local derivatives"}, {"start": 8658.96, "end": 8660.96, "text": " Local gradients which are implemented in the"}, {"start": 8660.96, "end": 8662.96, "text": " Backward PyTorch will be able to"}, {"start": 8662.96, "end": 8664.96, "text": " Backpropagate through your function"}, {"start": 8664.96, "end": 8666.96, "text": " And then you can use this as a LEGO block"}, {"start": 8666.96, "end": 8668.96, "text": " In a larger LEGO castle"}, {"start": 8668.96, "end": 8670.96, "text": " Of all the different LEGO blocks that PyTorch already has"}, {"start": 8670.96, "end": 8672.96, "text": " And so that's the only thing you have to tell PyTorch"}, {"start": 8672.96, "end": 8674.96, "text": " And everything would just work"}, {"start": 8674.96, "end": 8676.96, "text": " And you can register new types of functions"}, {"start": 8676.96, "end": 8678.96, "text": " In this way following this example"}, {"start": 8678.96, "end": 8680.96, "text": " And that is everything that I wanted to cover in this lecture"}, {"start": 8680.96, "end": 8682.96, "text": " So I hope you enjoyed building out micro-grad"}, {"start": 8682.96, "end": 8684.96, "text": " With me, I hope you find it interesting"}, {"start": 8684.96, "end": 8686.96, "text": " And insightful"}, {"start": 8686.96, "end": 8688.96, "text": " And yeah, I will post a lot of the links"}, {"start": 8688.96, "end": 8690.96, "text": " That are related to this video"}, {"start": 8690.96, "end": 8692.96, "text": " In the video description below"}, {"start": 8692.96, "end": 8694.96, "text": " I will also probably post a link to a discussion forum"}, {"start": 8694.96, "end": 8696.96, "text": " Or discussion group"}, {"start": 8696.96, "end": 8698.96, "text": " Where you can ask questions related to this video"}, {"start": 8698.96, "end": 8700.96, "text": " And then I can answer"}, {"start": 8700.96, "end": 8702.96, "text": " Or someone else can answer your questions"}, {"start": 8702.96, "end": 8704.96, "text": " And I may also do a follow-up video"}, {"start": 8704.96, "end": 8706.96, "text": " That answers some of the most common questions"}, {"start": 8706.96, "end": 8708.96, "text": " But for now, that's it"}, {"start": 8708.96, "end": 8710.96, "text": " I hope you enjoyed it"}, {"start": 8710.96, "end": 8712.96, "text": " If you did, then please like and subscribe"}, {"start": 8712.96, "end": 8714.96, "text": " So that YouTube knows to feature this video to more people"}, {"start": 8714.96, "end": 8716.96, "text": " And that's it for now"}, {"start": 8716.96, "end": 8718.96, "text": " I'll see you later"}, {"start": 8722.96, "end": 8724.96, "text": " Now here's the problem"}, {"start": 8724.96, "end": 8726.96, "text": " We know DL by"}, {"start": 8726.96, "end": 8728.96, "text": " Wait, what is the problem"}, {"start": 8728.96, "end": 8730.96, "text": " And that's everything I wanted to cover in this lecture"}, {"start": 8730.96, "end": 8732.96, "text": " So I hope you enjoyed"}, {"start": 8732.96, "end": 8734.96, "text": " Us building out micro-grad"}, {"start": 8734.96, "end": 8736.96, "text": " Micro-grad"}, {"start": 8736.96, "end": 8738.96, "text": " Okay now let's do the exact same thing for multiple"}, {"start": 8738.96, "end": 8740.96, "text": " Because we can't do something like"}, {"start": 8740.96, "end": 8742.96, "text": " Eight times two"}, {"start": 8742.96, "end": 8744.96, "text": " Oops"}, {"start": 8744.96, "end": 8746.96, "text": " I know what happened there"}, {"start": 8746.96, "end": 8748.96, "text": " I know what happened there"}, {"start": 8748.96, "end": 8750.96, "text": " I know what happened there"}, {"start": 8750.96, "end": 8760.96, "text": " I know what happened there"}]