We dive into some of the internals of MLPs with multiple layers and scrutinize the statistics of the forward pass activations, backward pass gradients, and some of the pitfalls when they are improperly scaled. We also look at the typical diagnostic tools and visualizations you'd want to use to understand the health of your deep network. We learn why training deep neural nets can be fragile and introduce the first modern innovation that made doing so much easier: Batch Normalization. Residual connections and the Adam optimizer remain notable todos for later video.

Links:
- makemore on github: https://github.com/karpathy/makemore
- jupyter notebook I built in this video: https://github.com/karpathy/nn-zero-to-hero/blob/master/lectures/makemore/makemore_part3_bn.ipynb
- collab notebook: https://colab.research.google.com/drive/1H5CSy-OnisagUgDUXhHwo1ng2pjKHYSN?usp=sharing
- my website: https://karpathy.ai
- my twitter: https://twitter.com/karpathy
- Discord channel: https://discord.gg/Hp2m3kheJn

Useful links:
- "Kaiming init" paper: https://arxiv.org/abs/1502.01852
- BatchNorm paper: https://arxiv.org/abs/1502.03167
- Bengio et al. 2003 MLP language model paper (pdf): https://www.jmlr.org/papers/volume3/bengio03a/bengio03a.pdf

Exercises:
- E01: I did not get around to seeing what happens when you initialize all weights and biases to zero. Try this and train the neural net. You might think either that 1) the network trains just fine or 2) the network doesn't train at all, but actually it is 3) the network trains but only partially, and achieves a pretty bad final performance. Inspect the gradients and activations to figure out what is happening and why the network is only partially training, and what part is being trained exactly.
- E02: BatchNorm, unlike other normalization layers like LayerNorm/GroupNorm etc. has the big advantage that after training, the batchnorm gamma/beta can be "folded into" the weights of the preceeding Linear layers, effectively erasing the need to forward it at test time. Set up a small 3-layer MLP with batchnorms, train the network, then "fold" the batchnorm gamma/beta into the preceeding Linear layer's W,b by creating a new W2, b2 and erasing the batch norm. Verify that this gives the same forward pass during inference. i.e. we see that the batchnorm is there just for stabilizing the training, and can be thrown out after training is done! pretty cool.

Chapters:
00:00:00 intro
00:01:22 starter code
00:04:19 fixing the initial loss 
00:12:59 fixing the saturated tanh
00:27:53 calculating the init scale: “Kaiming init”
00:40:40 batch normalization
01:03:07 batch normalization: summary
01:04:50 real example: resnet50 walkthrough
01:14:10 summary of the lecture
01:18:35 just kidding: part2: PyTorch-ifying the code
01:26:51 viz #1: forward pass activations statistics
01:30:54 viz #2: backward pass gradient statistics
01:32:07 the fully linear case of no non-linearities
01:36:15 viz #3: parameter activation and gradient statistics
01:39:55 viz #4: update:data ratio over time
01:46:04 bringing back batchnorm, looking at the visualizations
01:51:34 summary of the lecture for real this time
 Hi everyone. Today we are continuing our implementation of Makemore. Now in the last lecture we implemented the Multalia Perceptron along the lines of Benjueil to Hal 2003 for Character Level Language modeling. So we followed this paper, took in a few characters in the past, and used an MLP to predict the next character in a sequence. So what we'd like to do now is we'd like to move on to more complex and larger neural networks, like recurrent neural networks, and their variations like the GRU, LSTM, and so on. Now before we do that though, we have to stick around the level of Multalia Perceptron for a bit longer. And I'd like to do this because I would like us to have a very good intuitive understanding of the activations in the neural net during training, and especially the gradients that are flowing backwards, and how they behave, and what they look like. This is going to be very important to understand the history of the development of these architectures, because we'll see that recurrent neural networks, while they are very expressive in that they are a universal approximator and can imprince the implement all the algorithms. We'll see that they are not very easily optimizable with the first order of gradient-based techniques that we have available to us, and that we use all the time. And the key to understanding why they are not optimizable easily is to understand the the activations and the gradients and how they behave during training. And we'll see that a lot of the variants since recurrent neural networks have tried to improve that situation. And so that's the path that we have to take, and let's go start it. So the starting code for this lecture is largely the code from before, but I've cleaned it up a little bit. So you'll see that we are importing all the torch and math plotlet utilities. We're reading in the words just like before. These are eight example words. There's a total of 32,000 of them. Here's a vocabulary of all the lowercase letters and the special dot token. Here we are reading the dataset and processing it, and creating three splits, the train, dev, and the test split. Now in MLP, this is the identical same MLP, except you see that I removed a bunch of magic numbers that we had here. And instead we have the dimensionality of the embedding space of the characters and the number of hidden units in the hidden layer. And so I've pulled them outside here so that we don't have to go and change all these magic numbers all the time. With the same neural net with 11,000 parameters that we optimize now over 200,000 steps with batch size of 32. And you'll see that I refactored the code here a little bit, but there are no functional changes. I just created a few extra variables, a few more comments, and I removed all the magic numbers. And otherwise is the exact same thing. Then when we optimize, we saw that our loss looked something like this. We saw that the train and valve loss were about 2.16 and so on. Here I refactored the code a little bit for the evaluation of arbitrary splits. So you pass in a string of which split you'd like to evaluate. And then here depending on train, valve, or test, I index in and I get the correct split. And then this is the forward pass of the network and evaluation of the loss and printing it. So just making it nicer, I want to think that you'll notice here is I'm using a decorator torched.no grad, which you can also look up and read documentation of. Basically what this decorator does on top of a function is that whatever happens in this function is synced by torched to never require an ingredients. So it will not do any of the bookkeeping that it does to keep track of all the gradients in anticipation of an eventual backward pass. It's almost as if all the tensors that get created here have a requires grad of false. And so it just makes everything much more efficient because you're telling torched that I will not call dot backward on any of this computation and you don't need to maintain the graph under the hood. So that's what this does. And you can also use a context manager with torched.no grad and you can let those out. Then here we have the sampling from a model just as before. Just a four pass of a neural nut getting the distribution sampling from it adjusting the context window and repeating until we get the special and token. And we see that we are starting to get much nicer looking words simple from the model. It's still not amazing and they're still not fully name like. But it's much better than what we had with the bi-gram model. So that's our starting point. Now the first thing I would like to scrutinize is the initialization. I can tell that our network is very improperly configured at initialization. And there's multiple things wrong with it. But let's just start with the first one. Look here on the zero iteration, the very first iteration, we are recording a loss of 27 and this rapidly comes down to roughly one or two or so. So I can tell that the initialization is all messed up because this is way too high. In training of neural nuts, it is almost always the case that you will have a rough idea for what loss to expect at initialization. And that just depends on the loss function and the problem setup. In this case, I do not expect 27. I expect a much lower number and we can calculate it together. Basically, at initialization, or we'd like is that there's 27 characters that could come next for anyone training example. At initialization, we have no reason to believe any characters to be much more likely than others. And so we'd expect that the probability distribution that comes out initially is a uniform distribution, assigning about equal probability to all the 27 characters. So basically what we'd like is the probability for any character would be roughly one over 27. That is the probability we should record. And then the loss is the negative log probability. So let's wrap this in a tensor. And then then we can take the log of it. And then the negative log probability is the loss we would expect, which is 3.29, much, much lower than 27. And so what's happening right now is that at initialization, the neural net is creating probability distributions that are all messed up. Some characters are very confident and some characters are very not confident. And then basically what's happening is that the network is very confidently wrong. And that makes that's what makes it record very high loss. So here's a smaller four dimensional example of the issue. Let's say we only have four characters and then we have logits that come out of the neural net and they are very very close to zero. Then when we take the softmax of all zeroes, we get probabilities there are a diffuse distribution. So sums to one and is exactly uniform. And then in this case, if the label is say two, it doesn't actually matter if this if the label is two or three or one or zero because it's a uniform distribution, we're recording the exact same loss in this case 1.38. So this is the loss we would expect for a four dimensional example. And I can see of course that as we start to manipulate these logits, we're going to be changing the loss here. So it could be that we lock out and by chance this could be a very high number like five or something like that. Then in that case, we'll record a very low loss because we're signing the correct probability at initialization by chance to the correct label. Much more likely, it is that some other dimension will have a high logit and then what will happen is we start to record much higher loss. And what can come what can happen is basically the logits come out like something like this, you know, and they take on extreme values and we record really high loss. For example, if we have torched out random of four, so these are uniform, sorry, these are normally distributed numbers, forum. And here we can also print the logits, probabilities that come out of it and the loss. And so because these logits are near zero, for the most part, the loss that comes out is okay. But suppose this is like time 10 now. You see how because these are more extreme values, it's very unlikely that you're going to be guessing the correct bucket and then you're confidently wrong and recording very high loss. If your logits are coming up even more extreme, you might get extremely same losses like infinity even at initialization. So basically this is not good and we want the logits to be roughly zero when the work is initialized. In fact, the logits can don't have to be just zero. They just have to be equal. So for example, if all the logits are one, then because of the normalization inside the softmax, this will actually come out okay. But by symmetry, we don't want it to be any arbitrary positive or negative number. We just want it to be all zeros and record the loss that we expect at initialization. So let's not completely see where things go wrong in our example. Here we have the initialization. Let me initialize the neural ad. And here let me break after the very first iteration. So we only see the initial loss, which is 27. So that's way too high. And intuitively now we can expect the variables involved. And we see that the logits here, if we just print some of these, if we just print the first row, we see that the logits take on quite extreme values. And that's what's creating the fake confidence and incorrect answers. And makes the loss get very, very high. So these logits should be much, much closer to zero. So now let's think through how we can achieve logits coming out of this neural net to be more closer to zero. You see here that the logits are calculated as they hit in states multiplied by W2 plus B2. So first of all, currently we're initializing B2 as random values of the right size. But because we want roughly zero, we don't actually want to be adding a bias of random numbers. So in fact, I'm going to add a times a zero here to make sure that B2 is just basically zero at initialization. And second, this is H multiplied by W2. So if we want logits to be very, very small, then we would be multiplying W2 and making that small. So for example, if we scale down W2 by 0.1 all the elements, then if I do again just a very first iteration, you see that we are getting much closer to what we expect. So roughly what we want is about 3.29, this is 4.2. I can make this maybe even smaller, 3.32. Okay, so we're getting closer and closer. Now, you're probably wondering, can we just set this to zero? Then we get, of course, exactly what we're looking for at initialization. And the reason I don't usually do this is because I'm very nervous and I'll show you in a second why you don't want to be setting W's or weights of a neural net exactly to zero. You usually want to be small numbers instead of exactly zero. For this output layer in this specific case, I think it would be fine, but I'll show you in a second where things go wrong very quickly if you do that. So let's just go with 0.01. In that case, our loss is close enough, but has some entropy. It's not exactly zero. It's got some low entropy and that's used for symmetry braking as we'll see in a second. Logits are now coming out much closer to zero and everything is well and good. So if I just erase these and I now take away the brake statement, we can run the optimization with this new initialization. And let's just see what losses we record. Okay, so I'll let it run and you see that we started off good and then we came down a bit. The plot of the loss now doesn't have this hockey shape appearance because basically what's happening in the hockey stick, the very first few iterations of the loss, what's happening during the optimization is the optimization is just squashing down the logits and then it's rearranging the logits. So basically we took away this easy part of the loss function where just the weights were just being shrunk down. And so therefore we don't get these easy gains in the beginning and we're just getting some of the hard gains of training the actual neural nut. And so there's no hockey stick appearance. So good things are happening in that both number one loss and initialization is what we expect. And the loss doesn't look like a hockey stick. And this is true for any neural nut you might train and something to look at for. And second, the loss that came out is actually quite a bit improved. Unfortunately, I erased what we had here before. I believe this was 2.12 and this was 2.16. So we get a slightly improved result. And the reason for that is because we're spending more cycles, more time optimizing the neural nut actually instead of just spending the first several thousand iterations probably just squashing down the weights because they are so way too high in the beginning and the initialization. So something to look out for and that's number one. Now let's look at the second problem. Let me re-initialize our neural nut and let me reintroduce the break statement. So we have a reasonable initial loss. So even though everything is looking good on the level of the loss and we get something that we expect, there's still a deeper problem lurking inside this neural nut and its initialization. So the logits are now okay. The problem now is with the values of H, the activations of the hidden states. Now if we just visualize this vector, sorry, this tensor H, it's kind of hard to see but the problem here roughly speaking is you see how many of the elements are one or negative one. Now recall that tortsh.10h, the 10h function is a squashing function. It takes arbitrary numbers and it squashes them into a range of negative one and one and it does so smoothly. So let's look at the histogram of H to get a better idea of the distribution of the values inside this tensor. We can do this first. Well we can see that H is 32 examples and 200 activations in each example. We can view it as negative one to stretch it out into one large vector and we can then call two lists to convert this into one large Python list of floats and then we can pass this into plt.hist for histogram and we say we want 50 bins and a semicolon to suppress a bunch of output we don't want. So we see this histogram and we see that most of the values by far take on value of negative one and one. So this 10h is very very active and we can also look at basically why that is we can look at the preactivations that feed into the 10h and we can see that the distribution of the preactivations are is very very broad. These take numbers between negative 15 and 15 and that's why in a tortsh.10h everything is being squashed and capped to be in the range of negative one and one and lots of numbers here take on very extreme values. Now if you are new to it in your networks you might not actually see this as an issue but if you're well burst in the dark arts of back propagation and then have an intuitive sense of how these radians float through in your own net you are looking at your distribution of 10h activations here and you are sweating. So let me show you why. We have to keep in mind that during back propagation just like we saw in micrograd we are doing backward pass starting at the loss and flowing through the network backwards. In particular we're going to back propagate through this tortsh.10h and this layer here is made up of 200 neurons for each one of these examples and it implements an element twice 10h. So let's look at what happens in 10h in the backward pass. We can actually go back to our previous micrograd code in the very first lecture and see how we implement the 10h. We saw that the input here was x and then we calculate t which is the 10h of x. So that's t and t is between negative one and one. It's the output of the 10h and then in the backward pass how do we back propagate through a 10h? We take out that grad and then we multiply it this is the chain rule with the local gradient which took the form of 1 minus t squared. So what happens if the outputs of your 10h are very close to negative one or one. If you plug in t equals one here you're going to get a zero multiplying out that grad. No matter what out that grad is we are killing the gradient and we're stopping effectively the back propagation through this 10h unit. Similarly when t is negative one this will again become zero and out that grad just stops and intuitively this makes sense because this is a 10h neuron and what's happening is if its output is very close to one then we are in the tail of this 10h. So changing basically the input is not going to impact the output of the 10h too much because it's in the flat region of the 10h and so therefore there's no impact on the loss. And so indeed the weights and the biases along with this 10h neuron do not impact the loss because the output of this 10h unit is in the flat region of the 10h and there's no influence. We can we can be changing them whatever we want however we want and the loss is not impacted. That's another way to justify that indeed the gradient would be basically zero advantages. Indeed when t equals zero we get one times out that grad so when the 10h takes on exactly value of zero then out that grad is just passed through. So basically what this is doing right is if t is equal to zero then this the 10h unit is sort of inactive and gradient just passes through but the more you are in the flat tails the more degrading is squashed. So in fact you'll see that the gradient flowing through 10h can only ever decrease in the amount that it decreases is proportional through a square here depending on how far you are in the flat tails of this 10h. And so that's kind of what's happening here and through this the concern here is that if all of these outputs h are in the flat regions of negative one to one then the gradients that are flowing through the network will just get destroyed at this layer. Now there is some redeeming quality here and that we can actually get a sense of the problem here as follows. I brought some code here and basically what we want to do here is we want to take a look at h, take the absolute value and see how often it is in the in a flat region. So say grading point 99 and what you get is the following and this is a Boolean tensor. So in the Boolean tensor you get a white if this is true and a black if this is false. And so basically what we have here is the 32 examples and a 200 hidden neurons and we see that a lot of this is white and what that's telling us is that all these 10h neurons were very very active and they're in the flat tail and so in all these cases the backward gradient would get destroyed. Now we would be in a lot of trouble if for for any one of these 200 neurons if it was the case that the entire column is white because in that case we have what's called a dead neuron and this could be a tenet neuron where the initialization of the weights in the biases could be such that no single example ever activates this 10h in the sort of active part of the 10h. If all the examples land in the tail then this neuron will never learn it is a dead neuron. And so just scrutinizing this and looking for columns of completely white we see that this is not the case. So I don't see a single neuron that is all of white and so therefore it is the case that for every one of these 10h neurons we do have some examples that activate them in the active part of the 10h and so some gradients will flow through this neuron will learn and neuron will change and it will move and it will do something. But you can sometimes get yourself in cases where you have dead neurons and weight this manifests is that for 10h neuron this would be when no matter what inputs you plug in from your data set this 10h neuron always fires completely one or completely negative one and then it will just not learn because all the gradients will be just zero that. This is true not just for 10h but for a lot of other non-linearities that people use in neural networks. So we certainly use 10h a lot but sigmoid will have the exact same issue because it is a squashing neuron and so the same will be true for sigmoid but you know basically the same will actually apply to sigmoid. The same will also apply to a relu so relu has a completely flat region here below zero. So if you have a relu neuron then it is a pass-through if it is positive and if the pre-activation is negative it will just shut it off. Since the region here is completely flat then during back propagation this would be exactly zeroing out the gradient. Like all of the gradient would be set exactly to zero instead of just like a very very small number depending on how positive or negative t is. And so you can get for example a dead relu neuron and a dead relu neuron would basically look like. Basically what it is is if a neuron with a relu non-linearity never activates it. So for any examples that you plug in in the data set it never turns on it is always in this flat region then this relu neuron is a dead neuron. Its weights and bias will never learn they will never get agradient because the neuron never activated. And this can sometimes happen at initialization because the weights in the bias is just make it so that by chance some neurons are just forever dead but it can also happen during optimization. If you have like a too high learning weight for example sometimes you have these neurons that get too much of a gradient and they get knocked out off the data manifold. And what happens is that from then on no example ever activates this neuron so this neuron remains dead forever. So it's kind of like a permanent brain damage in a in a mind of a network. And so sometimes what can happen is if your learning rate is very high for example and you have a neural net with relu neurons you train the neural net and you get some last loss. But then actually what you do is you go through the entire training set and you forward on your examples and you can find neurons that never activate they are dead neurons in your network and so those neurons will will never turn on. And usually what happens is that during training these relu neurons are changing moving etc and then because of a high gradient somewhere by chance they get knocked off and then nothing ever activates them and from then on they are just dead. So that's kind of like a permanent brain damage that can happen to some of these neurons. These other nonlinearities like leaky relu will not suffer from this issue as much because you can see that it doesn't have flat tails. You almost always get gradients and ilu is also fairly frequently used. It also might suffer from this issue because it has flat parts. So that's just something to be aware of and something to be concerned about. And in this case we have way too many activations age that take on extreme values and because there's no column of white I think we will be okay and indeed the network optimizes and gives us a pretty decent loss but it's just not optimal and this is not something you want especially during initialization. And so basically what's happening is that this age preactivation that's flowing to 10h it's too extreme it's too large it's creating very it's creating distribution that is too saturated in both sides of the 10h and it's not something you want because it needs that there's less training for these neurons because they update less frequently. So how do we fix this? Well, age preactivation is MCAT which comes from C. So these are uniform Gaussian but then multiply by W1 plus B1 and age preact is too far off from 0 and that's causing the issue. So we want this preactivation to be closer to 0 very similar to what we have with logists. So here we want actually something very very similar. Now it's okay to set the biases to very small number. We can either multiply by 0, 0, 1 to get like a little bit of entropy. I sometimes like to do that just so that there's like a little bit of variation and diversity in the original initialization of these 10h neurons and I find in practice that that can help optimization a little bit and then the weights we can also just like squash. So let's multiply everything by 0.1. Let's rerun the first batch and now let's look at this and well first let's look at here. You see now because we multiply W by 0.1 we have a much better histogram and that's because the preactivations are now between negative 1.5 and 1.5 and this we expect much much less white. Okay, there's no white. So basically that's because there are no neurons that saturated above 0.99 in either direction. So it's actually a pretty decent place to be. Maybe we can go up a little bit. Sorry am I changing W1 here? Maybe we can go to 0.2. Okay, so maybe something like this is a nice distribution. So maybe this is what our initialization should be. So let me now erase these and let me starting with initialization. Let me run the full optimization without the break and let's see what we get. Okay, so the optimization finished and I rerun the loss and this is the result that we get and then just as a reminder I put down all the losses that we saw previously in this lecture. So we see that we actually do get an improvement here and just as a reminder we started off with the validation loss of 2.17 when we started. By fixing the softmax being confidently wrong we came down to 2.13 and by fixing the 10H layer being way too saturated we came down to 2.10 and the reason this is happening of course is because our initialization is better and so we're spending more time being productive training instead of not very productive training because our gradients are set to zero and we have to learn very simple things like the overconfidence of the softmax in the beginning and we're spending cycles just like squashing down the white matrix. So this is illustrating basically initialization and its impacts on performance just by being aware of the intervals of these neural nets and their actuations and their gradients. Now we're working with a very small network. This is just one layer, multi-layer perception. So because the network is so shallow the optimization problem is actually quite easy and very forgiving. So even though our initialization was terrible the network still learned eventually it just got a bit worse result. This is not the case in general though. Once we actually start working with much deeper networks that have saved 50 layers things can get much more complicated and these problems stack up and so you can actually get into a place where the network is basically not training at all if your initialization is bad enough and the deeper your network is and the more complex it is the less forgiving it is to some of these errors. And so something to be definitely aware of something to scrutinize, something to plot and something to be careful with and yeah. Okay so that's great that that worked for us but what we have here now is all these magic numbers like point two like where do I come up with this and how am I supposed to set these if I have a large neural net with lots and lots of layers and so obviously no one does this by hand there's actually some relatively principled ways of setting these scales that I would like to introduce to you now. So let me paste some code here that I prepared just to motivate the discussion of this. So what I'm doing here is we have some random input here x that is drawn from a Gaussian and there's 1000 examples that are 10 dimensional and then we have a weight layer here that is also initialized using Gaussian just like we did here and we these neurons in the hidden layer look at 10 inputs and there are 200 neurons in this hidden layer and then we have here just like here in this case the multiplication x multiply by w to get the preactivations of these neurons. And basically the analysis here looks at okay suppose these are uniform Gaussian and these weights are uniform Gaussian. If I do x times w and we forget for now the bias and the nonlinearity then what is the mean and the standard deviation of these Gaussians? So in the beginning here the input is just a normal Gaussian distribution mean zero and the standard deviation is one and the standard deviation again is just the measure of a spread of the Gaussian. But then once we multiply here and we look at the histogram of y we see that the mean of course stays the same it's about zero because this is a symmetric operation but we see here that the standard deviation has expanded to three. So the inputs and deviation was one but now we've grown to three and so what you're seeing in the histogram is that this Gaussian is expanding and so we're expanding this Gaussian from the input and we don't want that we want most of the neural nets to have relatively similar activations so unit Gaussian roughly throughout the neural net. And so the question is how do we scale these w's to preserve the to preserve this distribution to remain a Gaussian. And so intuitively if I multiply here these elements of w by a large number like say by 5 then this Gaussian grows and grows in standard deviation so now we're at 15 so basically these numbers here in the output y take on more and more extreme values but if we scale it down like say 0.2 then conversely this Gaussian is getting smaller and smaller and it's shrinking and you can see that the standard deviation is 0.6 and so the question is what do I multiply by here to exactly preserve the standard deviation to be one and it turns out that the correct answer mathematically when you work out through the variance of this multiplication here is that you are supposed to divide by the square root of the fan in. The fan in is basically the number of input components here 10 so we are supposed to divide by 10 square root and this is one way to do the square root you raise it to a power of 0.5 and that's the same as doing a square root. So when you divide by the square root of 10 then we see that the output Gaussian it has exactly standard deviation of 1. Now unsurprisingly a number of papers have looked into how but to best initialize neural networks and in the case in multiple perceptions we can have fairly deep networks that had these non-linearities in between and we want to make sure that the activations are well behaved and they don't expand to infinity or shrink all the way to 0 and the question is how do we initialize the weights so that these activations take on reasonable values throughout the network. Now one paper that has studied this in quite a bit detail that is often referenced is this paper by Kaminghe Tull called delving deep interactive fires. Now in this case they actually study convolutional neural networks and they study especially the relu nullinarity and the p-relu nullinarity instead of a 10-h nullinarity but the analysis is very similar and basically what happens here is for them the the relu nullinarity that they care about quite a bit here is a squashing function where all the negative numbers are simply clamped to 0 so the positive numbers are a path through but everything negative is just set to 0 and because you are basically throwing away half of the distribution they find in their analysis of the forward activations in the neural net that you have to complicate for that with a gain and so here they find that basically when they initialize their weights they have to do it with a zero mean Gaussian whose standard deviation is square root of 2 over the fanning what we have here is we are initializing the Gaussian with the square root of fanning this NL here is the fanning so what we have is square root of 1 over the fanning because we have the division here now they have to add this factor of 2 because of the relu which basically discards half of the distribution and clamps it at 0 and so that's where you get an initial factor now in addition to that this paper also studies not just the sort of behavior of the activations in the forward path of the neural net but it also studies the back propagation and we have to make sure that the gradients also are well behaved and so because ultimately they end up updating our parameters and what they find here through a lot of the analysis that I am much into read through but it's not exactly approachable what they find is basically if you properly initialize the forward path the backward path is also approximately initialized up to a constant factor that has to do with the size of the number of hidden neurons and early and late layer and but basically they find empirically that this is not a choice that matters too much now this chiming initialization is also implemented in PyTorch so if you go to torshtat and end up in it documentation you'll find chiming normal and in my opinion this is probably the most common way of initializing neural networks now and it takes a few keyword arguments here so another one it wants to know the mode would you like to normalize the activations or would you like to normalize the gradients to to be always caution with zero mean and unit or one standard deviation and because they find a paper that this doesn't matter too much most of the people just leave it as the default which is pen in and then second pass in the nullinearity that you are using because depending on the nullinearity we need to calculate a slightly different gain and so if your nullinearity is just a linear so there's no nullinearity then the gain here will be one and we have these same kind of formula that we've caught here but if the nullinearity is something else we're going to get a slightly different gain and so if we come up here to the top we see that for example in the case of relu this gain is a square root of two and the reason is the square root because in this paper you see how the two is inside of the square root so the gain is a square root of two in a case of linear or identity we just get a gain of one in a case of 10h which is what we're using here the advised gain is a 5 over 3 and intuitively why do we need a gain on top of the initialization is because 10h just like relu is a contractive transformation so what that means is you're taking the output distribution from this matrix multiplication and then you are squashing it in some way now relu squashes it by taking everything below zero and clamping it to zero 10h also squashes it because it's a contractive operation it will take the tails and it will squeeze them in and so in order to fight the squeezing in we need to boost the weights a little bit so that we re-normalize everything back to unit standard deviation so that's why there's a little bit of a gain that comes out now i'm skipping through this section a little bit quickly and i'm doing that actually intentionally and the reason for that is because about seven years ago when this paper was written you had to actually be extremely careful with the activations and ingredients and their ranges and their histograms and you had to be very careful with the precise setting of gains and the scrutinizing of the null and your t's used and so on and everything was very finicky and very fragile and to be very properly arranged for the neural network to train especially if your neural network was very deep but there are a number of modern innovations that have made everything significantly more stable and more well behaved and it's become less important to initialize these networks exactly right and some of those modern innovations for example are residual connections which we will cover in the future the use of a number of normalization layers like for example bash normalization layer normalization work normalization we're going to go into a lot of these as well and number three much better optimizers not just to cast a gradient scent the simple optimizer we're basically using here but slightly more complex optimizers like armistrop and especially atom and so all of these modern innovations make it less important for you to precisely calibrate the initialization of the neural net all that being said in practice what should we do in practice when I initialize these neural nets I basically just normalize my weights by the square root of the pen and so basically roughly what we did here is what I do now if we want to be exactly accurate here we and go by in it of kind of normal this is how we would implement it we want to set the standard deviation to be gained over the square root of fan in right so to set the standard deviation of our weights we will proceed as follows basically when we have torshtot ran and let's say I just create a thousand numbers we can look at the standard deviation of this and of course that's one that's the amount of spread let's make this a bit bigger so it's closer to one so this is the spread of the Gaussian of zero mean and unit standard deviation now basically when you take these and you multiply by say 0.2 that basically scales down the Gaussian and that makes its standard deviation 0.2 so basically the number that you multiply by here ends up being the standard deviation of this Gaussian so here this is a standard deviation 0.2 Gaussian here when we sample our w1 but we want to set the standard deviation to gain over square root of fan mode which is fan in so in other words we want to multiply by gain which for 10 h is 5 over 3 5 over 3 is the gain and then times um I guess I divide square root of the fan in and in this example here the fan in was 10 and I just noticed that actually here the fan in for w1 is actually an embed times block size which as you will recall is actually 30 and that's because each character is 10 dimensional but then we have three of them and we can cutinate them so actually the fan in here was 30 and I should have used 30 here probably but basically we want 30 square root so this is the number this is what our standard deviation we want to be and this number turns out to be 0.3 whereas here just by fiddling with it and looking at the distribution and making sure it looks okay we came up with 0.2 and so instead what we want to do here is we want to make the standard deviation be um 5 over 3 which is our gain divide this amount times 0.2 square root and these brackets here are not that necessary but I'll just put them here for clarity this is basically what we want this is the kiming in it in our case for a 10 h nonlinearity and this is how we would initialize the neural net and so we're multiplying by 0.3 instead of multiplying by 0.2 and so we can we can initialize this way and then we can train the neural net and see what we got okay so I trained the neural net and we end up in roughly the same spot so looking at the values should loss we now get 2.10 and previously we also had 2.10 there's a little bit of a difference but that's just the randomness the process I suspect but the big deal of course is we get to the same spot but we did not have to introduce any um magic numbers that we got from just looking at histograms and guessing checking we have something that is semi-principled and will scale us to much bigger networks and uh something that we can sort of use as a guide so I mentioned that the precise setting of these initializations is not as important today due to some modern innovations and I think now is a pretty good time to introduce one of those modern innovations and that is best normalization so best normalization came out in uh 2015 from a team at Google and it was an extremely impactful paper because it made it possible to train very deep neural nets quite reliably and uh it basically just worked so here's what best normalization does and what's implemented um basically we have these uh hidden states h pre-act right and we were talking about how we don't want these uh these um pre-activation states to be way too small because that then the 10h is not um doing anything but we don't want them to be too large because then the 10h is saturated in fact we want them to be roughly roughly Gaussian so zero mean and a units or one standard deviation at least at initialization so the insight from the best normalization paper is okay you have these hidden states and you'd like them to be roughly Gaussian then why not take the hidden states and uh just normalize them to be Gaussian and it sounds kind of crazy but you can just do that because uh standardizing hidden states so that their unit Gaussian is a perfectly differentiable operation as we'll soon see and so that was kind of like the big insight in this paper and when I first read it my mind was blown because you can just normalize these hidden states and if you'd like unit Gaussian states in your network uh at least initialization you can just normalize them to be in Gaussian so uh let's see how that works so we're going to scroll to our pre-activations here just before they enter into the 10h now the idea again is remember we're trying to make these roughly Gaussian and that's because if these are way too small numbers then the 10h here is kind of an active but if these are very large numbers then the 10h is way to saturated and graded in the flow so we'd like this to be roughly Gaussian so the insight in best romanization again is that we can just standardize these activations so they are exactly Gaussian so here h preact has a shape of 32 by 200 32 examples by 200 neurons in the hidden layer so basically what we can do is we can take h preact and we can just calculate the mean um and the mean we want to calculate across the zero dimension and we want to also keep them as true so that we can easily broadcast this so the shape of this is 1 by 200 in other words we are doing the mean over all the uh elements in the batch and similarly we can calculate the standard deviation of these activations and then we'll also be 1 by 200 now in this paper they have the uh sort of prescription here and see here we are calculating the mean which is just taking the average value of any neurons activation and then the standard deviation is basically kind of like um this the measure of the spread that we've been using which is the distance of every one of these values away from the mean and that squared and averaged that's the that's the variance and then if you want to take the standard deviation you would the square root the variance to get the standard deviation so these are the two that we're calculating and now we're going to normalize or standardize these x's by subtracting the mean and um dividing by the standard deviation so basically we're taking edge preact and we subtract the mean and then we divide by the standard deviation this is exactly what these two STD and mean are calculating oops sorry this is the mean and this is the variance you see how the sigma is the standard deviation usually so this is sigma square which is the variance is the square of the standard deviation so this is how you standardize these values and what this will do is that every single neuron now and its firing rate will be exactly unit Gaussian on these 32 examples at least of this batch that's why it's called batch normalization we are normalizing these batches and then we could in principle train this notice that calculating the mean and standard deviation these are just mathematical formulas they're perfectly differentiable all this is perfectly differentiable and we can just train this the problem is you actually won't achieve a very good result with this and the reason for that is we want these to be roughly Gaussian but only at initialization but we don't want these to be forced to be Gaussian always we would like to allow the neural nets to move this around to potentially make it more diffuse to make it more sharp to make some 10 H neurons maybe mean more trigger more trigger happy or less trigger happy so we'd like this distribution to move around and we'd like the back propagation to tell us how the distribution should move around and so in addition to this idea of standardizing the activations at any point in the network we have to also introduce this additional component in the paper here describe the scale and shift and so basically what we're doing is we're taking these normalized inputs and we are additionally scaling them by some gain and offsetting them by some bias to get our final output from this layer and so what that amounts to is the following we are going to allow a bash normalization gain to be initialized at just a once and the once will be in the shape of 1 by n hidden and then we also will have a bn bias which will be torched at 0's and it will also be of the shape n by 1 by n hidden and then here the bn gain will multiply this and the bn bias will offset it here so because this is initialized to 1 and this to 0 at initialization each neurons firing values in this batch will be exactly unit Gaussian and will have nice numbers no matter what the distribution of the h preact is coming in coming out it will be unit Gaussian for each neuron and that's roughly what we want at least at initialization and then during optimization we'll be able to back propagate to bn gain and bn bias and change them so the network is given the full ability to do with this whatever it wants internally here we just have to make sure that we include these in the parameters of the neural nut because they will be trained with back propagation so let's initialize this and then we should be able to train and then we're going to also copy this line which is the bash normalization layer here on a single line of code and we're going to swing down here and we're also going to do the exact same thing at test time here so similar to train time we're going to normalize and then scale and that's going to give us our train and validation loss and we'll see in a second that we're actually going to change this a little bit but for now I'm going to keep it this way so I'm just going to wait for this to converge okay so I allowed the neural nut to converge here and when we scroll down we see that our validation loss here is 2.10 roughly which I wrote down here and we see that this is actually kind of comparable to some of the results that we've achieved previously now I'm not actually expecting an improvement in this case and that's because we are dealing with a very simple neural nut that has just a single hidden layer so in fact in this very simple case of just one hidden layer we were able to actually calculate what the scale of W should be to make these pre-activations already have a roughly Gaussian shape so the best normalization is not doing much here but you might imagine that once you have a much deeper neural nut that has lots of different types of operations and there's also for example residual connections which we'll cover and so on it will become basically very very difficult to tune the scales of your wake matrices such that all the activations throughout the neural nut are roughly Gaussian and so that's going to become very quickly intractable but compared to that it's going to be much much easier to sprinkle batch normalization layers throughout the neural nut so in particular it's common to to look at every single linear layer like this one this is a linear layer multiplied by weight matrix and adding the bias or for example convolutions which we'll cover later and also perform basically a multiplication with weight matrix but in a more spatially structured format it's customer it's customer to take these linear layer or convolutional layer and append a best normalization layer right after it to control the scale of these activations at every point in the neural nut so we'd be adding these best normal layers throughout the neural nut and then this controls the scale of these activations throughout the neural nut it doesn't require us to do a perfect mathematics and care about the activation distributions for all these different types of neural nut or Lego building blocks that you might want to introduce into your neural nut and it significantly stabilizes the train and that's why these layers are quite popular now the stability offered by batch normalization actually comes at a terrible cost and that cost is that if you think about what's happening here something's something terribly strange and unnatural is happening it used to be that we have a single example feeding into a neural nut and then we calculate this activations and its logits and this is a deterministic sort of process so you arrive at some logits for this example and then because of efficiency of training we suddenly started to use batches of examples but those batches of examples were processed independently and it was just an efficiency thing but now suddenly in batch normalization because of the normalization through the batch we are coupling these examples mathematically and in the forward pass and backward pass of the neural nut so now the hidden state activations H pre-oct and your logits for any one input example are not just a function of that example and its input but they're also a function of all the other examples that happen to come for a ride in that batch and these examples are sampled randomly and so what's happening is for example when you look at H pre-oct that's going to feed into H the hidden state activations for for example for for any one of these input examples is going to actually change slightly depending on what other examples there are in the batch and depending on what other examples happen to come for a ride H is going to change suddenly and it's going to look jitter if you imagine sampling different examples because the statistics of the mean and the standard deviation are going to be impacted and so you'll get a jitter for H and you'll get a jitter for logits and you think that this would be a bug or something undesirable but in a very strange way this actually turns out to be good in neural network training and as a side effect and the reason for that is that you can think of this as kind of like a regularizer because what's happening is you have your input and you get your H and then depending on the other examples this is jittering a bit and so what that does is that it's effectively padding out any one of these input examples and it's introducing a little bit of entropy and because of the padding out it's actually kind of like a form of a data augmentation which will cover in the future and it's not kind of like augmenting the input a little bit and jittering it and that makes it harder for the neural nets to overfit these concrete specific examples so by introducing all this noise it actually like pads out the examples and it regularizes the neural net and that's one of the reasons why deceiving me as a second order effect this is actually a regularizer and that has made it harder for us to remove the use of bachelor normalization because basically no one likes this property that the examples in the batch are coupled mathematically and in the forward pass and at least all kinds of like strange results will go into some of that in a second as well and at least do a lot of bugs and so on and so no one likes this property and so people have tried to deprecate the use of bachelor normalization and move to other normalization techniques that do not couple the examples of a batch examples are linear normalization instance normalization group normalization and so on and we'll come rest we'll come rest on these later but basically a long story short bachelor normalization was the first kind of normalization later to be introduced it worked extremely well it happens to have this regularizing effect it's stabilized training and people have been trying to remove it and move to some of the other normalization techniques but it's been hard because it it just works quite well and some of the reason that it works quite well is again because of this regularizing effect and because of the because it is quite effective at controlling the activations and their distributions so that's kind of like the brief story of bachelor normalization and I like to show you one of the other weird sort of outcomes of this coupling so here's one of the strange outcomes that I only lost over previously when I was evaluating the loss on the validation set basically once we've trained a neural net we'd like to deploy it in some kind of a setting and we'd like to be able to feed in a single individual example and get a prediction out from our neural net but how do we do that when our neural net now in a forward pass estimates the statistics of the mean understand deviation of a batch the neural net expects batches as an input now so how do we feed in a single example and get sensible results out and so the proposal in the bachelor normalization paper is the following what we would like to do here is we would like to basically have a step after training that calculates and sets the bachelor mean and standard deviation a single time over the training set and so I wrote this code here in interest of time and we're going to call what's called calibrate the bachelor statistics and basically what we do is torshot no grad telling by torshot that none of this we will call the dot backward on and it's going to be a bit more efficient we're going to take the training set get the preactivations for every single training example and then one single time estimate the mean and standard deviation over the entire training set and then we're going to get b and mean and b and standard deviation and now these are fixed numbers estimated over the entire training set and here instead of estimating it dynamically we are going to instead here use b and mean and here we're just going to use b and standard deviation and so at test time we are going to fix these clamp them and use them during inference and now you see that we get basically identical result but the benefit that we've gained is that we can now also forward a single example because the mean and standard deviation are now fixed all sorts of tensors that said nobody actually wants to estimate this mean and standard deviation as a second stage after neural network training because everyone is lazy and so this specialization paper actually introduced one more idea which is that we can we can estimate the mean and standard deviation in a running manner running manner during training of the neural network and then we can simply just have a single stage of training and on the side of that training we are estimating the running mean as a deviation so let's see what that would look like let me basically take the mean here that we are estimating on the batch and let me call this b and mean on the i-th iteration and then here this is b and sd d b and sd d at i okay and the mean comes here and the sd d comes here so so far I've done nothing I've just moved around and I created these extra variables for the mean and standard deviation and I've put them here so so far nothing has changed but what we're going to do now is we're going to keep a running mean of both of these values during training so let me swing up here and let me create a b and mean underscore running and I'm going to initialize it at zeros and then b and sd d running which I'll initialize at once because in the beginning because of the way we initialized w1 and b1 hpx will be roughly unit Gaussian so the mean will be roughly zero and extend deviation roughly one so I'm going to initialize these that way but then here I'm going to update these and in PyTorch these mean and standard deviation that are running they're not actually part of the gradient based optimization we're never going to derive gradients with respect to them they're they're updated on the side of training and so what we're going to do here is we're going to say with torsched up no grad telling PyTorch that the update here is not supposed to be building out a graph because there will be no dot backward but this running mean is basically going to be 0.99 nine times the current value plus 0.001 times the best value this new mean and in the same way bnstd running will be mostly what it used to be but it will receive a small update in the direction of what the current standard deviation is and as you're seeing here this update is outside and on the side of the gradient based optimization and it's simply being updated not using gradient sent is just being updated using a jenky like smooth sort of running mean manner and so while the network is training and these preactivations are sort of changing and shifting around during during back propagation we are keeping track of the typical mean and standard deviation and rest of the mean them once and when I run this now I'm keeping track of this in a running manner and what we're hoping for of course is that the mean bn mean underscore running and bn mean underscore std are going to be very similar to the ones that we calculated here before and that way we don't need a second stage because we've sort of combined the two stages and we've put them on the side of each other if you want to look at it that way and this is how this is also implemented in the bastion realization layer in PyTorch so during training the exact same thing will happen and then later when you're using inference it will use the estimated running mean of both the mean and standard deviation of those hidden states so let's wait for the optimization to converge and hopefully the running mean and standard deviation are roughly equal to these two and then we can simply use it here and we don't need this stage of explicit calibration at the end okay so the optimization finished I'll rerun the explicit estimation and then the bn mean from the explicit estimation is here and bn mean from the running estimation during the during the optimization you can see it's very very similar it's not identical but it's pretty close and the same way bnstd is this and bnstd running is this as you can see that once again they are fairly similar values not identical but pretty close and so then here instead of being mean we can use the bn mean running instead of bnstd we can use bnstd running and hopefully the validation loss will not be impacted too much okay so it's basically identical and this way we've eliminated the need for this explicit stage of calibration because we are doing it in line over here okay so we're almost done with batch normalization there are only two more notes that I'd like to make number one I've skipped a discussion over what is this plus epsilon doing here this epsilon is usually like some small fixed number for example one in negative 5 by default and what it's doing is that it's basically preventing a division by zero in the case that the variance over your batch is exactly zero in that case here we normally have a division by zero but because of the plus epsilon this is going to become a small number in the denominator instead and things will be more well behaved so feel free to also add a plus epsilon here of a very small number it doesn't actually substantially change the result I'm going to skip it in our case just because this is unlikely to happen in our very simple example here and the second thing I want you to notice is that we're being wasteful here and it's very subtle but right here where we are adding the bias into each preact these biases now are actually useless because we're adding them to the each preact but then we are calculating the mean for every one of these neurons and subtracting it so whatever bias you add here is going to get subtracted right here and so these biases are not doing anything in fact but they're being subtracted out and they don't impact the rest of the calculation so if you look at b1.grad it's actually going to be zero because it being subtracted out and doesn't actually have any effect and so whenever you're using bachelor normalization layers then if you have any weight layers before like a linear or a conv or something like that you're better off coming here and just like not using bias so you don't want to use bias and then here you don't want to add it because that's that's spurious instead we have this bachelor normalization bias here and that bachelor normalization bias is now in charge of the biasing of this distribution instead of this b1 that we had here originally and so basically the bachelor normalization layer has its own bias and there's no need to have a bias in the layer before it because that bias is going to be subtracted out anyway so that's the other small detail to be careful with sometimes it's not going to do anything catastrophic this b1 will just be useless it will never get any gradient it will not learn it will stay constant and it's just wasteful but it doesn't actually really impact anything otherwise okay so I rearranged the code a little bit with comments and I just wanted to give a very quick summary of the bachelor normalization layer we are using bachelor normalization to control the statistics of activations in the neural net it is common to sprinkle bachelor normalization layer across the neural net and usually we will place it after layers that have multiplications like for example a linear layer or a convolutional layer which we may cover in the future now the bachelor normalization internally has parameters for the gain and the bias and these are trained using back propagation it also has two buffers the buffers are the mean and the standard deviation the running mean and the running mean of the standard deviation and these are not trained using back propagation these are trained using this janky update of kind of like a running mean update so these are sort of the parameters and the buffers of bachelor layer and then really what is doing is it's calculating the mean and standard deviation of the activations that are feeding into the bachelor layer over that batch then it's centering that batch to be unit Gaussian and then it's offsetting and scaling it by the learned bias and gain and then on top of that it's keeping track of the mean and standard deviation of the inputs and it's maintaining this running mean and standard deviation and this will later be used at inference so that we don't have to re-estimate the mean and standard deviation all the time and in addition that allows us to basically forward individual examples at test time so that's the bachelor normalization layer it's a fairly complicated layer but this is what it's doing internally now I wanted to show you a little bit of a real example so you can search ResNet which is a residual neural network and these are contact of neural arcs used for image classification and of course we haven't common dress nets in detail so I'm not going to explain all the pieces of it but for now just note that the image feeds into a ResNet on the top here and there's many many layers with repeating structure all the way to predictions of what's inside that image this repeating structure is made up of these blocks and these blocks are just sequentially stacked up in this deep neural network now the code for this the block basically that's used and repeated sequentially in series is called this bottleneck block and there's a lot here this is all PyTorch and of course we haven't covered all of it but I want to point out some small pieces of it here in the init is where we initialize the neural net so this code of block here is basically the kind of stuff we're doing here we're initializing all the layers and in the forward we are specifying how the neural net acts once you actually have the input so this code here is a long lines of what we're doing here and now these blocks are replicated and stacked up serially and that's what a residual network would be and so notice what's happening here come one these are convolutional layers and these convolutional layers basically they're the same thing as a linear layer except convolutional layers don't apply convolutional layers are used for images and so they have spatial structure and basically this linear multiplication and bias offset are done on patches instead of a map instead of the full input so because these images have structure spatial structure convolutional is just basically do wx plus b but they do it on overlapping patches of the input but otherwise is wx plus b then we have the normal layer which by default here is initialize to be a bashed norm in 2d so 2-dimensional bashed normalization layer and then we have a nonlinearity like relu so instead of here they use relu we are using 10h in this case but both both are just nonlinearities and you can just use them relatively interchangeably for very deep networks relu typically empirically work a bit better so see the motif that's being repeated here we have convolution, bashed normalization relu convolution, bashed normalization relu etc and then here this is residual connection that we haven't covered yet but basically that's the exact same pattern we have here we have a weight layer like a convolution or like a linear layer, bashed normalization and then 10h which is nonlinearity but basically a weight layer, a normalization layer and nonlinearity and that's the motif that you would be stacking up when you create these deep neural networks exactly as it's done here and one more thing I'd like you to notice is that here when they are initializing the comp layers like comp one by one the depth for that is right here and so it's initializing an nn.comp2d which is a convolutional layer in PyTorch and there's a bunch of keyword arguments here that I am not going to explain yet but you see how there's bias equals false the bias equals false is exactly for the same reason as bias is not used in our case you see how I raised the use of bias and the use of bias is spurious because after this weight layer there's the bashed normalization and the bashed normalization subtracts that bias and that has its own bias so there's no need to introduce these spurious parameters it wouldn't hurt performance it's just useless and so because they have this motif of calm bashed normalization they don't need a bias here because there's a bias inside here so by the way this example here is very easy to find just do a resonant PyTorch and it's this example here so this is kind of like the stock implementation of a residual neural network in PyTorch and you can find that here but of course I haven't covered many of these parts yet and I would also like to briefly descend into the definitions of these PyTorch layers and the parameters that they take now instead of a convolutional layer we're going to look at a linear layer because that's the one that we're using here this is a linear layer and I haven't covered cover the convolutions yet but as I mentioned convolutions are basically linear layers except on patches so a linear layer performs a wx plus b except here they're calling the w a transpose so the call is wx plus b very much like we did here to initialize this layer you need to know the fan in the fan out and that's so that they can initialize this w this is the fan in and the fan out so they know how big the weight matrix should be you need to also pass in whether you whether or not you want a bias and if you set it to false the no bias will be inside this layer and you may want to do that exactly like in our case if your layer is followed by a normalization layer such as bach norm so this allows you to basically disable bias in terms of the initialization if we swing down here this is reporting the variables used inside this linear layer and our linear layer here has two parameters the weight and the bias in the same way they have a weight and a bias and they're talking about how they initialize it by default so by default PyTorch will initialize your weights by taking the fan in and then doing one over fan in square root and then instead of a normal distribution they are using a uniform distribution so it's very much the same thing but they are using a one instead of five over three so there's no gain being calculated here the gain is just one but otherwise is exactly one over the square root of fan in exactly as we have here so one over the square root of k is the is this scale of the weights but when they are drawing the numbers they're not using a Gaussian by default they're using a uniform distribution by default and so they draw uniformly from negative square root of k to square root of k but it's the exact same thing and the same motivation from for with respect to what we've seen in this lecture and the reason they're doing this is if you have a roughly Gaussian input this will ensure that out of this layer you will have a roughly Gaussian output and you you basically achieve that by scaling the weights by one over the square root of fan in so that's what this is doing and then the second thing is the bachelor normalization layer so let's look at what that looks like in PyTorch so here we have a one-dimensional bachelor normalization layer exactly as we are using here and there are a number of keyword arguments going into it as well so we need to know the number of features for us that is 200 and that is needed so that we can initialize these parameters here the gain the bias and the buffers for the running mean and serredivation then they need to know the value of epsilon here and by default this is one negative five you don't typically change this too much then they need to know the momentum and the momentum here as they explain is basically used for these running mean and running standard deviation so by default the momentum here is point one the momentum we are using here in this example is 0.001 and basically you may want to change this sometimes and roughly speaking if you have a very large batch size then typically what you'll see is that when you estimate the mean and the same deviation for every single batch size if it's large enough you're going to get roughly the same result and so therefore you can use slightly higher momentum like point one but for a batch size s small is 32 the mean and understanding deviation here might take on slightly different numbers because there's only 32 examples we are using to estimate the mean and standard deviation so the value is changing around a lot and if your momentum is point one that that might not be good enough for this value to settle and converge to the actual mean and standard deviation over the entire training set and so basically if your batch size is very small momentum of point one is potentially dangerous and it might make it so that the running mean and standard deviation is thrashing too much during training and it's not actually converging properly affine equals true determines whether this batch normalization layer has these learnable affine parameters the gain and the bias and this is almost always kept the true I'm not actually sure why you would want to change this to false then track running stats is determining whether or not batch normalization layer of bite or chip will be doing this and one reason you may you may want to skip the running stats is because you may want to for example estimate them at the end as a stage two like this and in that case you don't want the batch normalization layer to be doing all this extra compute that you're not going to use and finally we need to know which device we're going to run this batch normalization on a CPU or a GPU and what the data type should be a half precision single precision double precision and so on so that's the batch normalization layer otherwise they linked to the paper is the same formula we've implemented and everything is the same exactly as we've done here okay so that's everything that I wanted to cover for this lecture really what I wanted to talk about is the importance of understanding the activations and the gradients and their statistics in neural networks and this becomes increasingly important especially as you make your neural networks bigger larger and deeper we looked at the distributions basically at the output layer and we saw that if you have two confident misperdictions because the activations are too messed up at the last layer you can end up with these hockey stick losses and if you fix this you get a better loss at the end of training because your training is not doing wasteful work then we also saw that we need to control the activations we don't want them to you know squash to zero or explore to infinity and because that you can run into a lot of trouble with all of these nonlinearities in these neural nets and basically you want everything to be fairly homogeneous throughout the neural net you want roughly Gaussian activations throughout the neural net let me talk about okay if we want roughly Gaussian activations how do we scale these weight matrices and biases during initialization of the neural net so that we don't get you know so everything is as controlled as possible so that give us a large boost in improvement and then I talked about how that strategy is not actually possible for much much deeper neural nets because when you have much deeper neural nets with lots of different types of layers it becomes really really hard to precisely set the weights and the biases in such a way that the activations are roughly uniform throughout the neural net so then I introduced the notion of the normalization layer now there are many normalization layers that that people use in practice, Beshaw normalization layer normalization this is normalization group normalization we haven't covered most of them but I've introduced the first one and also the one that I believe came out first and that's called Beshaw normalization and we saw how Beshaw normalization works this is a layer that you can sprinkle throughout your deep neural net and the basic idea is if you want roughly Gaussian activations well then take your activations and take the mean and the standard deviation and center your data and you can do that because the centering operation is differentiable but on top of that we actually had to add a lot of bells and whistles and that gave you a sense of the complexities of the Beshaw normalization layer because now we're centering the data that's great but suddenly we need the gain and the bias and now those are trainable and then because we are coupling all the training examples now suddenly the question is how do you do the inference or to do the inference we need to now estimate these mean and standard deviation once or the entire training set and then use those at inference but then no one likes to do stage two so instead we fold everything into the Beshaw normalization layer during training and try to estimate these in the running manner so that everything is a bit simpler and that gives us the Beshaw normalization layer and as I mentioned no one likes this layer it causes a huge amount of bugs and intuitively it's because it is coupling examples in the form of passive and neural net and I've shocked myself in the foot with this layer over and over again in my life and I don't want you to suffer the same so basically try to avoid it as much as possible some of the other alternatives to these layers are for example group normalization or layer normalization and those have become more common in more recent deep learning but we haven't covered those yet but definitely Beshaw normalization was very influential at the time when it came out in roughly 2015 because it was kind of the first time that you could train reliably much deeper neural nets and fundamentally the reason for that is because this layer was very effective at controlling the statistics of the activations in the neural net so that's the story so far and that's all I wanted to cover and in the future lecture so hopefully we can start going into recurring neural nets and recurring neural nets as we'll see are just very very deep networks because you you unrolled the loop and when you actually optimize these neural nets and that's where a lot of this analysis around the activations statistics and all these normalization layers will become very very important for good performance so we'll see that next time bye okay so I lied I would like us to do one more summary here as a bonus and I think it's useful as to have one more summary of everything I've presented in this lecture but also I would like us to start by torturing our code a little bit so it looks much more like what you would encounter in PyTorch so you'll see that I will structure our code into these modules like a linear module and a bachelor module and I'm putting the code inside these modules so that we can construct neural networks very much like we would construct the in PyTorch and I will go through this in detail so we'll create our neural net then we will do the optimization loop as we did before and then the one more thing that I want to do here is I want to look at the activations statistics both in the forward pass and in the backward pass and then here we have the evaluation and sampling just like before so let me rewind all the way up here and go a little bit slower so here I'm creating a linear layer you'll notice that Torch.nn has lots of different types of layers and one of those layers is the linear layer Torch.nn.nl it takes a number of input features output features whether or not we should have bias and then the device that we want to place this layer on and the data type so I will omit these two but otherwise we have the exact same thing we have the fan in which is the number of inputs, fan out the number of outputs and whether or not we want to use a bias and internally inside this layer there's a weight and a bias if you like it it is typical to initialize the weight using say random numbers drawn from a Gaussian and then here's the coming initialization that we discussed already in this lecture and that's a good default and also the default that I believe PyTorch chooses and by default the bias is usually initialized to zeros. Now when you call this module this will basically calculate W times x plus B if you have NB and then when you also call that parameters on this module it will return the tensors that are the parameters of this layer. Now next we have the Bachelormalization layer so I've written that here and this is very similar to PyTorch's and then that Bachelorm 1D layer as shown here. So I'm kind of taking these three parameters here the dimensionality the epsilon that we'll use in the division and the momentum that we will use in keeping track of these running stats the running mean and the running variance. Now PyTorch actually takes quite a few more things but I'm assuming some of their settings so for us I'll find will be true that means that we will be using a gamma beta after denormalization. The track running stats will be true so we will be keeping track of the running mean and the running variance in the in the classroom. Our device by default is the CPU and the data type by default is float float 32. So those are the defaults otherwise we are taking all the same parameters in this Bachelorm layer so first I'm just saving them. Now here's something new there's that training which by default is true and PyTorch and then modules also have this attribute that training and that's because many modules and Bachelorm is included in that have a different behavior whether you are training your own or not or whether you are running it in an evaluation mode and calculating your evaluation laws or using it for inference on some test examples. And Bachelorm is an example of this because when we are training we are going to be using the mean and the variance estimated from the current batch but during inference we are using the running mean and running variance and so also if we are training we are updating mean and variance but if we are testing then these are not being updated they're kept fixed and so this flag is necessary and by default true just like in PyTorch. Now the parameters of Bachelorm 1D are the gamma and the beta here and then the running mean and running variance are called buffers in PyTorch nomenclature and these buffers are trained using exponential moving average here explicitly and they are not part of the back propagation and stochastic gradient descent so they are not sort of like parameters of this layer and that's why when we have a parameters here we only return gamma and beta we do not return the mean and the variance this is trained sort of like internally here every forward pass using exponential moving average. So that's the initialization. Now in a forward pass if we are training then we use the mean and the variance estimated by the batch only a block of paper here. We calculate the mean and the variance. Now up above I was estimating the standard deviation and keeping track of the standard deviation here in the running standard deviation instead of running variance but let's follow the paper exactly here they calculate the variance which is the standard deviation squared and that's what's kept track of in the running variance instead of a running standard deviation but those two would be very very similar I believe. If we are not training then we use running mean and variance we normalize and then here I am calculating the output of this layer and I'm also assigning it to an attribute called dot out. Now dot out is something that I'm using in our modules here. This is not what you would find in PyTorch we are slightly deviating from it. I'm creating a dot out because I would like to very easily maintain all those variables so that we can create statistics of them and plot them but PyTorch and modules will not have a dot out attribute. And finally here we are updating the buffers using again as I mentioned exponential moving average given the provided momentum and importantly you'll notice that I'm using the torshtap no-grat context manager and I'm doing this because if we don't use this then PyTorch will start building out an entire computational graph out of these tensors because it is expecting that we will eventually call a dot backward but we are never going to be calling that backward on anything that includes running mean and running variance. So that's why we need to use this context manager so that we are not sort of maintaining them using all this additional memory. So this will make it more efficient and it's just telling PyTorch that they will need no backward. We just have a bunch of tensors we want to update them that's it and then we return. Okay now scrolling down we have the 10H layer. This is very very similar to torshtap 10H and it doesn't do too much it just calculates 10H as you might expect. So that's torshtap 10H and there's no parameters in this layer but because these are layers it now becomes very easy to sort of like stack them up into basically just a list and we can do all the initializations that we're used to. So we have the initial sort of embedding matrix we have our layers and we can call them sequentially and then again with torshtap no grad there's some initializations here. So we want to make the output softmax a bit less confident like we saw and in addition to that because we are using a six layer multi layer perceptron here so you see how I'm stacking linear 10H linear 10H etc. I'm going to be using the gain here and I'm going to play with this in a second so you'll see how when we change this what happens to the statistics. Finally the primers are basically the embedding matrix and all the parameters in all the layers and notice here I'm using a double list comprehension if you want to call it that but for every layer in layers and for every parameter in each of those layers we are just stacking up all those piece all those parameters. Now in total we have 46,000 parameters and I'm telling PyTorch that all of them require gradient. Then here we have everything here we are actually mostly used to. We are sampling batch we are doing a forward pass the forward pass now is just the linear application of all the layers in order followed by the cross entropy and then in the backward pass you'll notice that for every single layer I now iterate over all the outputs and I'm telling PyTorch to retain the gradient of them and then here we are already used to all the all the gradients set to none do the backward to fill in the gradients do an update using the cast gradient send and then track some statistics and then I am going to break after a single iteration. Now here in this cell in this diagram I am visualizing the histogram the histograms of the forward pass activations and I'm specifically doing it at the 10 each layers. So iterating over all the layers except for the very last one which is basically just the softmax layer. If it is a 10 each layer and I'm using a 10 each layer just because they have a finite output negative 1 to 1 and so it's very easy to visualize here so you see negative 1 to 1 and it's a finite range and easy to work with. I take the out tensor from that layer into t and then I'm calculating the mean this 10 deviation and the percent saturation of t and the way I define the percent saturation is that t dot absolute value is greater than 0.97 so that means we are here at the tails of the 10 each and remember that when we are in the tails of the 10 each that will actually stop gradients so we don't want this to be too high. Now here I'm calling torshot histogram and then I am plotting this histogram. So basically what this is doing is that every different type of layer and they all have a different color we are looking at how many values in these tensors take on any of the values below on this axis here. So the first layer is fairly saturated here at 20 percent so you can see that it's got tails here but then everything sort of stabilizes and if we had more layers here it would actually just stabilize at around the 10 deviation of about 0.65 and the saturation will be roughly 5 percent and the reason that this stabilizes and gives us a nice distribution here is because gain is set to 5 over 3. Now here this gain you see that by default we initialize with 1 over square root of fennel but then here during initialization I come in and I iterate our all the layers and if it's a linear layer I boost that by the gain. Now we saw that one so basically if we just do not use a gain then what happens? If I retraw this you will see that the standard deviation is shrinking and the saturation is coming to 0 and basically what's happening is the first layer is you know pretty decent but then further layers are just kind of like shrinking down to 0 and it's happening slowly but it's shrinking to 0 and the reason for that is when you just have a sandwich of linear layers alone then a then initializing our weights in this manner we saw previously would have conserved the standard deviation of 1 but because we have this interspersed 10H layers in there these 10H layers are squashing functions and so they take your distribution and they slightly squash it and so some gain is necessary to keep expanding it to fight the squashing so it just turns out that 5 over 3 is a good value so if we have something too small like 1 we saw that things will come towards 0 but if it's something too high let's do 2 then here we see that well let me do something a bit more extreme because so it's a bit more visible let's try 3 okay so we see here that the saturation is not going to be too large okay so 3 would create weight-o-saturated activations so 5 over 3 is a good setting for a sandwich of linear layers with 10H activations and it roughly stabilizes the standard deviation at a reasonable point now honestly I have no idea where 5 over 3 came from in PyTorch when we were looking at the counting initialization I see empirically that it stabilizes the sandwich of linear and 10H and that the saturation is in a good range but I don't actually know if this came out of some math formula I tried searching briefly for where this comes from but I wasn't able to find anything but certainly we see that empirically these are very nice ranges our saturation is roughly 5% which is a pretty good number and this is a good setting of the gain in this context similarly we can do the exact same thing with the gradients so here is a very same loop if it's a 10H but instead of taking the layer that out I'm taking the grad and then I'm also showing the mean and the standard deviation and I'm plotting the histogram of these values and so you'll see that the gradient distribution is fairly reasonable and in particular what we're looking for is that all the different layers in this sandwich has roughly the same gradient things are not shrinking or exploding so we can for example come here and we can take a look at what happens if this gain was way too small so this was 0.5 then you see the first of all the activations are shrinking to zero but also the gradients are doing something weird the gradients started out here and then now they're like expanding out and similarly if we for example have a two-hymogain select three then we see that also the gradients have there's some asymmetry going on where as you go into deeper and deeper layers the activations are also changing and so that's not what we want and in this case we saw that without use of besterm as we are going through right now we have to very carefully set those gains to get nice activations in both the forward pass and the backward pass now before we move on to bestermalization I would also like to take a look at what happens when we have no 10H units here so erasing all the 10H nonlinearities but keeping the gain at 5 over 3 we now have just a giant linear sandwich so let's see what happens to the activations as we saw before the correct gain here is one that is the standard deviation preserving gain so 1.667 is too high and so what's going to happen now is the following I have to change this to be linear so we are because there's no more 10H layers and let me change this to linear as well so what we're seeing is the activations started out on the blue and have by layer 4 become very diffuse so what's happening to the activations is this and with the gradients on the top layer the activation the gradient statistics are the purple and then they diminish as you go down deeper in the layers and so basically you have an asymmetry like in the neural net and you might imagine that if you have very deep neural networks say like 50 layers or something like that this just this is not a good place to be also that's why before best normalization this was an incredibly tricky to to set in particular if this is too large of a gain this happens and if it's too little of a gain then this happens also the opposite of that basically happens here we have a shrinking and a diffusion depending on which direction you look at it from and so certainly this is not what you want and in this case the correct setting of the gain is exactly one just like we're doing at initialization and then we see that the statistics for the forward and the backward pass are well behaved and so the reason I want to show you this is the basically like getting neuralness to train before these normalization layers and before the use of advanced optimizers like Adam which we still have to cover and residual connections and so on training neuralness basically look like this it's like a total balancing act you have to make sure that everything is precisely orchestrated and you have to care about the activations and the gradients and their statistics and then maybe you can train something but it was basically impossible to train very deep networks and this is fundamentally the reason for that you'd have to be very very careful with your initialization the other point here is you might be asking yourself by the way I'm not sure if I covered this why do we need these 10H layers at all why do we include them and then have to worry about the gain and the reason for that of course is that if you just have a stack of linear layers then certainly we're getting very easily nice activations and so on but this is just a massive linear sandwich and it turns out that it collapses to a single linear layer in terms of its representation power so if you were to plot the output as a function of the input you're just getting a linear function no matter how many linear layers you stack up you still just end up with a linear transformation all the WX plus B's just collapse into a large WX plus B with slightly different W's as slightly different B but interestingly even though the forward pass collapses to just a linear layer because of backpapigation and the dynamics of the backward pass the optimization is really is not identical you actually end up with all kinds of interesting dynamics in the backward pass because of the the way the chain rule is calculating it and so optimizing a linear layer by itself and optimizing a sandwich of 10 linear layers in both cases those are just a linear transformation in the forward pass but the training dynamics would be different and there's entire papers that analyze in fact like infinitely layered linear layers and so on and so there's a lot of things to that you can play with there but basically the tenational linearities allow us to turn this sandwich from just a linear function into a neural network that can in principle approximate any arbitrary function okay so now I've reset the code to use the linear ten each sandwich like before and I've reset everything so the gains five over three we can run a single step of optimization and we can look at the activations statistics of the forward pass and the backward pass but I've added one more plot here that I think is really important to look at when you're training your neural nets and to consider and ultimately what we're doing is we're updating the parameters of the neural net so we care about the parameters and their values and their gradients so here what I'm doing is I'm actually iterating over all the parameters available and that I'm only restricting it to the two-dimensional parameters which are basically the weights of these linear layers and I'm skipping the biases and I'm skipping the gammas and the betas and the best room just for simplicity but you can also take a look at those as well but what's happening with the weights is instructive by itself so here we have all the different weights their shapes so this is the embedding layer the first linear layer all the way to the very last linear layer and then we have the mean the standard deviation of all these primers the histogram and you can see that actually it doesn't look that amazing so there's some trouble in paradise even though these gradients looked okay there's something weird going on here I'll get to that in a second and the last thing here is the gradient to data ratio so sometimes I'll have to visualize this as well because what this could see a sense of is what is the scale of the gradient compared to the scale of the actual values and this is important because we're going to end up taking a step update that is the learning rate times the gradient onto the data and so the gradient has two larger magnitudes if the numbers and there are two large compared to the numbers in data then you'd be in trouble but in this case the gradient to data is our low numbers so the values inside grad are 1000 times smaller than the values inside data in these weights most of them now notably that is not true about the last layer and so the last layer actually here the output layer is a bit of a trouble maker in the way that this is currently arranged because you can see that the last layer here in pink takes on values there are much larger than some of the values inside inside the neural net so the standard deviations are roughly 1-3 throughout except for the last but last new layer which actually has roughly 1-2 standard deviation of gradients and so the gradients on the last layer are currently about 100 times greater sorry 10 times greater than all the other weights inside the neural net and so this problematic because in the simple stochastic gradient in the sense setup you would be training this last layer about 10 times faster than you would be training the other layers at initialization now this actually like kind of fixes itself a little bit if you train for a bit longer so for example if I agree then 1000 only then do a break let me initialize and then let me do it 1000 steps and after 1000 steps we can look at the for a pass okay so you see how the neurons are a bit are saturating a bit and we can also look at the backward pass but otherwise they look good they're about equal and there's no shrinking to zero or exploding to infinities and you can see that here in the weights things are also stabilizing a little bit so the tails of the last pink layer are actually coming in during the optimization but certainly this is like a little bit of troubling especially if you are using a very simple update rule like stochastic gradient descent instead of a modern optimizer like atom now I'd like to show you one more plot that I usually look at when I train your own works and basically the gradient to data ratio is not actually that informative because what matters at the end is not the gradient to data ratio but the update to the data ratio because that is the amount by which we will actually change the data in these tensors so coming up here what I'd like to do is I'd like to introduce a new update to data ratio it's going to be less than we're going to build it out every single iteration and here I'd like to keep track of basically the ratio every single iteration so without any ingredients I'm comparing the update which is learning rate times the time is the gradient that is the update that we're going to apply to every parameter associated with random world of parameters and then I'm taking the basically standard deviation of the update we're going to apply and divided by the actual content the data of that parameter and its standard deviation so this is the ratio of basically how great are the updates to the values in these tensors then we're going to take a log of it and actually I'd like to take a log 10 just so it's a nice serviceualization so we're going to be basically looking at the exponents of this division here and then that item to pop out the float and we're going to be keeping track of this for all the parameters and adding it to this UD tensor so now let me re-initialize and run a thousand iterations we can look at the activations the gradients and the parameter gradients as we did before but now I have one more plot here to introduce now what's happening here is we're every interval of parameters and I'm constraining it again like I did here to just to weights so the number of dimensions in these sensors is two and then I'm basically plotting all of these update ratios over time so when I plot this I plot those ratios and you can see that they evolve over time during initialization that they concern values and then these updates are like start stabilizing usually during training then the other thing that I'm plotting here is I'm plotting here like an approximate value that is a rough guide for what it roughly should be and it should be like roughly one in negative three and so that means that basically there's some values in this tensor and they take on certain values and the updates to them at every single iteration are no more than roughly one thousand of the actual magnitude in those tensors if this was much larger like for example if this was if the log of this was like same negative one this is actually updating those values quite a lot they're undergoing a lot of change but the reason that the final rate the final layer here is an outlier is because this layer was artificially shrugged down to keep the softmax income unconfident so here you see how we multiply the weight by point one in the initialization to make the last layer prediction less confident that made that artificially made the values inside that tensor way too low and that's why we're getting temporarily a very high ratio but you see that that stabilizes over time once that weight starts to learn starts to learn but basically I like to look at the evolution of this update ratio for all my parameters usually and I like to make sure that it's not too much above one negative three roughly so around negative three on this log plot if it's below negative three usually that means that parameters are not training fast enough so if our learning rate was very low let's do that experiment let's initialize and then let's actually do a learning rate of say one in negative three here so 0.001 if your learning rate is way too low this plot will typically reveal it so you see how all of these updates are way too small so the size of the update is basically 10,000 times in magnitude to the size of the numbers in that tensor in the first place so this is a symptom of training way too slow so this is another way to sometimes start to learning rate and to get a sense of what that learning rate should be and ultimately this is something that you would keep track of if anything the learning rate here is a little bit on the higher side because you see that we're above the black line of negative three we're somewhere around negative 2.5 it's like okay and but everything is like somewhat stabilizing and so this looks like a pretty decent setting of of learning rates and so on but this is something to look at and when things are miscalibrated you will you will see very quickly so for example everything looks pretty well behaved right but just as a comparison when things are not properly calibrated what does that look like let me come up here and let's say that for example what do we do let's say that we forgot to apply this a fan in normalization so the weights inside the linear layers are just sample from a Gaussian in all the stages what happens to our how do we notice that something's off well the activation plot will tell you whoa your neurons are way too saturated the gradients are going to be all messed up the histogram for these weights are going to be all messed up as well and there's a lot of asymmetry and then if we look here I suspect it's all going to be also pretty messed up so you see there's a lot of discrepancy in how fast these layers are learning and some of them are learning way too fast so negative 1 negative 1.5 those are very large numbers in terms of this ratio again you should be somewhere around negative three and not much more about that so this is how miscalibration so if your neurons are going to manifest and these kinds of plots here are a good way of sort of bringing those miscalibration sort of to your attention and so you can address them okay so so far we've seen that when we have this linear 10-H sandwich we can actually precisely calibrate the gains and make the activations the gradients and the parameters and the updates all look pretty decent but it definitely feels a little bit like balancing of a pencil on your finger and that's because this gain has to be very precisely calibrated so now let's introduce bashfulization layers into the fix into the mix and let's let's see how that helps fix the problem so here I'm going to take the bashful monday class and I'm going to start placing it inside and as I mentioned before the standard typical placey would place it is between the linear layer so right after it before the non-linearity but people have definitely played with that and in fact you can get very similar results even if you place it after the non-linearity and the other thing that I wanted to mention is it's totally fine to also place it at the end after the last linear layer and before the last function so this is potentially fine as well and in this case this would be output would be woke up size now because the last layer is bashful we would not be changing to wait to make the softmax less confident we'd be changing the gamma because gamma remember in the bash room is the variable that multiplicatively interacts with the output of that normalization so we can initialize this sandwich now we can train and we can see that the activations are going to of course look very good and they are going to necessarily look good because now before every single 10-H layer there is a normalization in the bash room so this is unsurprisingly all looks pretty good it's going to be standard deviation of roughly 0.65 2% and roughly equals standard deviation throughout the entire layers so everything looks very homogeneous the gradients look good the weights look good and their distributions and then the updates also look pretty reasonable we're going above negative 3 a little bit but not by too much so all the parameters are training in roughly the same rate here but now what we gained is we are going to be slightly less brittle with respect to the gain of these so for example I can make the gain B say 0.2 here which is much more personal over than what we had with the 10-H but as we'll see the activations will actually be exactly unaffected and that's because again this explicit normalization the gradients are going to look okay the weight gradients are going to look okay but actually the updates will change and so even though the forward and backward paths to a very large extent look okay because of the backward paths of the bash room and how the scale of the incoming activations interact in the bash room and its backward paths this is actually changing the scale of the updates on these parameters so the gradients of these weights are affected so we still don't get a completely free pass to pass in arbitrary weights here but everything else is significantly more robust in terms of the forward backward and the weight gradients it's just that you may have to retune your learning rate if you are changing sufficiently the scale of the activations that are coming into the bash rooms so here for example this we changed the gains of these linear layers to be greater and we're seeing that the updates are coming out lower as a result and then finally we can also if we are using bash rooms we don't actually need to necessarily let me reset this to one so there's no gain we don't necessarily even have to normalize back then in sometimes so if I take out the fan in so these are just now random Gaussian we'll see that because of bash rooms this will actually be relatively well behaved so this this is look of course in the forward pass look good the gradients look good the backward the weight updates look okay a little bit of fat tails and some delayers and this looks okay as well but as you as you can see we're significantly below negative three so we'd have to bump up the learning rate of this bachelor so that we are training more properly and in particular looking at this roughly looks like we have to 10x the learning rate to get to about 1 e negative 3 so we'd come here and we would change this to be update of 1.0 and if I initialize then we'll see that everything still of course looks good and now we are roughly here and we expect this to be an okay training run so long story short we are significantly more robust to the gain of these linear layers whether or not we have to apply the fan in and then we can change the gain but we actually do have to worry a little bit about the update scales and making sure that the learning rate is properly calibrated here but the activations of the forward backward pass and the updates are all are looking significantly more well behaved except for the global scale that is potentially being adjusted here okay so now let me summarize there are three things I was hoping to achieve with this section number one I wanted to introduce you to bachelor normalization which is one of the first modern innovations that we're looking into that helped stabilize very deep neural networks and their training and I hope you understand how the bachelor normalization works and how it would be used in neural network number two I was hoping to pie torchify some wire code and wrap it up into these modules so like linear bachelor mondi 10h etc these are layers or modules and they can be stacked up into neural nets like Lego building blocks and these layers actually exist in pie torch and if you import torch and then then you can actually the way I've constructed it you can simply just use pie torch by pre-pending nn dot to all these different layers and actually everything will just work because the API that I have developed here is identical to the API that pie torch uses and the implementation also is basically as far as I'm aware identical to the one in pie torch and number three I try to introduce you to the diagnostic tools that you would use to understand whether your neural network is in a good state dynamically so we are looking at the statistics and histograms and activation of the forward pass activation activations the backward pass gradients and then also we're looking at the weights that are going to be updated as part of stochasticity in the send and we're looking at their means standard deviations and also the ratio of gradients to data or even better the updates to data and we saw that typically we don't actually look at it as a single snapshot frozen in time at some particular iteration typically people look at this as a over time just like I've done here and they look at these update to data ratios and they make sure everything looks okay and in particular I said that one in negative three or basically negative three on the log scale is a good rough heuristic for what you want this ratio to be and if it's way too high then probably the learning rate or the updates are too big and if it's way too small that the learning rate is probably too small so that's just some of the things that you may want to play with when you try to get your neural network to work well very well now there's a number of things I did not try to achieve I did not try to beat our previous performance as an example by introducing the bachelor layer actually I did try and I found that you I used the learning rate finding mechanism that I've described before I tried to train the bachelor layer a bachelor neural nut and I actually ended up with results that are very very similar to what we obtained before and that's because our performance now is not bottlenecked by the optimization which is what bachelor is helping with the performance at the stage is bottlenecked by what I suspect is the context length of our context so currently we are taking three characters to predict the fourth one and I think we need to go beyond that and we need to look at more powerful architectures like recurrent neural networks and transformers in order to further push the block probabilities that we're achieving on this day as it and I also did not try to have a full explanation of all of these activations, the gradients and the backward pass and the statistics of all these gradients and so you may have found some of the parts here on intuitive and maybe you're slightly confused about okay if I change the gain here how come that we need a different learning rate and I didn't go into the full detail because you'd have to actually look at the backward pass of all these different layers and get an intuitive understanding of how that works and I did not go into that in this lecture the purpose really was just to introduce you to the diagnostic tools and what they look like but there's still a lot of work remaining on the intuitive level to understand the initialization the backward pass and how all that interacts but you shouldn't feel too bad because honestly we are getting to the cutting edge of where the field is we certainly haven't I would say solved initialization and we haven't solved back propagation and these are still very much an active area of research people are still trying to figure out where's the best way to initialize these networks what is the best update rule to use and so on so none of this is really solved and we don't really have all the answers to all the to you know all these cases but at least you know we're making progress and at least we have some tools to tell us whether or not things are on the right track for now so I think we've made positive progress in this lecture and I hope you enjoyed that and I will see you next time
[{"start": 0.0, "end": 5.6000000000000005, "text": " Hi everyone. Today we are continuing our implementation of Makemore. Now in the last lecture we implemented"}, {"start": 5.6000000000000005, "end": 10.72, "text": " the Multalia Perceptron along the lines of Benjueil to Hal 2003 for Character Level Language modeling."}, {"start": 10.72, "end": 15.52, "text": " So we followed this paper, took in a few characters in the past, and used an MLP to predict the next"}, {"start": 15.52, "end": 20.16, "text": " character in a sequence. So what we'd like to do now is we'd like to move on to more complex and"}, {"start": 20.16, "end": 24.96, "text": " larger neural networks, like recurrent neural networks, and their variations like the GRU, LSTM,"}, {"start": 24.96, "end": 30.32, "text": " and so on. Now before we do that though, we have to stick around the level of Multalia Perceptron"}, {"start": 30.32, "end": 34.56, "text": " for a bit longer. And I'd like to do this because I would like us to have a very good intuitive"}, {"start": 34.56, "end": 39.6, "text": " understanding of the activations in the neural net during training, and especially the gradients"}, {"start": 39.6, "end": 44.400000000000006, "text": " that are flowing backwards, and how they behave, and what they look like. This is going to be very"}, {"start": 44.400000000000006, "end": 48.8, "text": " important to understand the history of the development of these architectures, because we'll see that"}, {"start": 48.8, "end": 53.760000000000005, "text": " recurrent neural networks, while they are very expressive in that they are a universal approximator"}, {"start": 53.76, "end": 59.44, "text": " and can imprince the implement all the algorithms. We'll see that they are not very easily"}, {"start": 59.44, "end": 63.28, "text": " optimizable with the first order of gradient-based techniques that we have available to us, and that"}, {"start": 63.28, "end": 69.44, "text": " we use all the time. And the key to understanding why they are not optimizable easily is to understand"}, {"start": 69.44, "end": 73.44, "text": " the the activations and the gradients and how they behave during training. And we'll see that a lot"}, {"start": 73.44, "end": 79.52, "text": " of the variants since recurrent neural networks have tried to improve that situation. And so"}, {"start": 79.52, "end": 84.32, "text": " that's the path that we have to take, and let's go start it. So the starting code for this lecture"}, {"start": 84.32, "end": 89.19999999999999, "text": " is largely the code from before, but I've cleaned it up a little bit. So you'll see that we are"}, {"start": 89.19999999999999, "end": 95.52, "text": " importing all the torch and math plotlet utilities. We're reading in the words just like before."}, {"start": 95.52, "end": 100.64, "text": " These are eight example words. There's a total of 32,000 of them. Here's a vocabulary of all the"}, {"start": 100.64, "end": 107.28, "text": " lowercase letters and the special dot token. Here we are reading the dataset and processing it,"}, {"start": 107.28, "end": 115.36, "text": " and creating three splits, the train, dev, and the test split. Now in MLP, this is the identical"}, {"start": 115.36, "end": 120.32000000000001, "text": " same MLP, except you see that I removed a bunch of magic numbers that we had here. And instead we"}, {"start": 120.32000000000001, "end": 124.96000000000001, "text": " have the dimensionality of the embedding space of the characters and the number of hidden units"}, {"start": 124.96000000000001, "end": 129.68, "text": " in the hidden layer. And so I've pulled them outside here so that we don't have to go and change all"}, {"start": 129.68, "end": 134.88, "text": " these magic numbers all the time. With the same neural net with 11,000 parameters that we optimize"}, {"start": 134.88, "end": 141.68, "text": " now over 200,000 steps with batch size of 32. And you'll see that I refactored the code here a little"}, {"start": 141.68, "end": 147.2, "text": " bit, but there are no functional changes. I just created a few extra variables, a few more comments,"}, {"start": 147.2, "end": 152.8, "text": " and I removed all the magic numbers. And otherwise is the exact same thing. Then when we optimize,"}, {"start": 152.8, "end": 158.4, "text": " we saw that our loss looked something like this. We saw that the train and valve loss were about"}, {"start": 158.4, "end": 167.04000000000002, "text": " 2.16 and so on. Here I refactored the code a little bit for the evaluation of arbitrary splits."}, {"start": 167.04000000000002, "end": 171.68, "text": " So you pass in a string of which split you'd like to evaluate. And then here depending on"}, {"start": 171.68, "end": 176.64000000000001, "text": " train, valve, or test, I index in and I get the correct split. And then this is the forward pass"}, {"start": 176.64000000000001, "end": 181.92000000000002, "text": " of the network and evaluation of the loss and printing it. So just making it nicer,"}, {"start": 181.92, "end": 188.07999999999998, "text": " I want to think that you'll notice here is I'm using a decorator torched.no grad, which you can"}, {"start": 188.07999999999998, "end": 193.76, "text": " also look up and read documentation of. Basically what this decorator does on top of a function"}, {"start": 194.39999999999998, "end": 200.95999999999998, "text": " is that whatever happens in this function is synced by torched to never require an"}, {"start": 200.95999999999998, "end": 206.64, "text": " ingredients. So it will not do any of the bookkeeping that it does to keep track of all the gradients"}, {"start": 206.64, "end": 211.51999999999998, "text": " in anticipation of an eventual backward pass. It's almost as if all the tensors that get created"}, {"start": 211.52, "end": 216.64000000000001, "text": " here have a requires grad of false. And so it just makes everything much more efficient because"}, {"start": 216.64000000000001, "end": 220.96, "text": " you're telling torched that I will not call dot backward on any of this computation and you don't"}, {"start": 220.96, "end": 227.12, "text": " need to maintain the graph under the hood. So that's what this does. And you can also use a"}, {"start": 227.12, "end": 234.24, "text": " context manager with torched.no grad and you can let those out. Then here we have the sampling from"}, {"start": 234.24, "end": 240.64000000000001, "text": " a model just as before. Just a four pass of a neural nut getting the distribution sampling from it"}, {"start": 240.64, "end": 245.67999999999998, "text": " adjusting the context window and repeating until we get the special and token. And we see that we"}, {"start": 245.67999999999998, "end": 251.11999999999998, "text": " are starting to get much nicer looking words simple from the model. It's still not amazing and"}, {"start": 251.11999999999998, "end": 255.76, "text": " they're still not fully name like. But it's much better than what we had with the bi-gram model."}, {"start": 257.68, "end": 261.59999999999997, "text": " So that's our starting point. Now the first thing I would like to scrutinize is the initialization."}, {"start": 262.56, "end": 268.0, "text": " I can tell that our network is very improperly configured at initialization. And there's"}, {"start": 268.0, "end": 272.08, "text": " multiple things wrong with it. But let's just start with the first one. Look here on the zero"}, {"start": 272.08, "end": 278.4, "text": " iteration, the very first iteration, we are recording a loss of 27 and this rapidly comes down to"}, {"start": 278.4, "end": 282.96, "text": " roughly one or two or so. So I can tell that the initialization is all messed up because this is way"}, {"start": 282.96, "end": 288.0, "text": " too high. In training of neural nuts, it is almost always the case that you will have a rough idea"}, {"start": 288.0, "end": 293.2, "text": " for what loss to expect at initialization. And that just depends on the loss function and the"}, {"start": 293.2, "end": 299.03999999999996, "text": " problem setup. In this case, I do not expect 27. I expect a much lower number and we can calculate"}, {"start": 299.03999999999996, "end": 306.56, "text": " it together. Basically, at initialization, or we'd like is that there's 27 characters that could come"}, {"start": 306.56, "end": 311.91999999999996, "text": " next for anyone training example. At initialization, we have no reason to believe any characters to be"}, {"start": 311.91999999999996, "end": 316.71999999999997, "text": " much more likely than others. And so we'd expect that the probability distribution that comes out"}, {"start": 316.72, "end": 323.44000000000005, "text": " initially is a uniform distribution, assigning about equal probability to all the 27 characters."}, {"start": 323.44000000000005, "end": 330.0, "text": " So basically what we'd like is the probability for any character would be roughly one over 27."}, {"start": 331.92, "end": 336.56, "text": " That is the probability we should record. And then the loss is the negative log probability."}, {"start": 336.56, "end": 343.04, "text": " So let's wrap this in a tensor. And then then we can take the log of it. And then the negative"}, {"start": 343.04, "end": 350.32, "text": " log probability is the loss we would expect, which is 3.29, much, much lower than 27. And so what's"}, {"start": 350.32, "end": 355.04, "text": " happening right now is that at initialization, the neural net is creating probability distributions"}, {"start": 355.04, "end": 359.92, "text": " that are all messed up. Some characters are very confident and some characters are very not confident."}, {"start": 360.64000000000004, "end": 366.96000000000004, "text": " And then basically what's happening is that the network is very confidently wrong. And that makes"}, {"start": 366.96, "end": 373.35999999999996, "text": " that's what makes it record very high loss. So here's a smaller four dimensional example of the issue."}, {"start": 373.35999999999996, "end": 378.56, "text": " Let's say we only have four characters and then we have logits that come out of the neural net"}, {"start": 378.56, "end": 384.08, "text": " and they are very very close to zero. Then when we take the softmax of all zeroes, we get"}, {"start": 384.08, "end": 390.15999999999997, "text": " probabilities there are a diffuse distribution. So sums to one and is exactly uniform."}, {"start": 391.03999999999996, "end": 396.64, "text": " And then in this case, if the label is say two, it doesn't actually matter if this if the label is two"}, {"start": 396.64, "end": 401.28, "text": " or three or one or zero because it's a uniform distribution, we're recording the exact same loss"}, {"start": 401.28, "end": 405.59999999999997, "text": " in this case 1.38. So this is the loss we would expect for a four dimensional example."}, {"start": 406.15999999999997, "end": 411.28, "text": " And I can see of course that as we start to manipulate these logits, we're going to be changing"}, {"start": 411.28, "end": 417.12, "text": " the loss here. So it could be that we lock out and by chance this could be a very high number"}, {"start": 417.12, "end": 421.44, "text": " like five or something like that. Then in that case, we'll record a very low loss because we're"}, {"start": 421.44, "end": 427.6, "text": " signing the correct probability at initialization by chance to the correct label. Much more likely,"}, {"start": 427.6, "end": 435.12, "text": " it is that some other dimension will have a high logit and then what will happen is we start to"}, {"start": 435.12, "end": 439.84, "text": " record much higher loss. And what can come what can happen is basically the logits come out"}, {"start": 440.4, "end": 446.08, "text": " like something like this, you know, and they take on extreme values and we record really high loss."}, {"start": 446.08, "end": 455.2, "text": " For example, if we have torched out random of four, so these are uniform, sorry, these are normally"}, {"start": 455.2, "end": 464.88, "text": " distributed numbers, forum. And here we can also print the logits, probabilities that come out of"}, {"start": 464.88, "end": 471.52, "text": " it and the loss. And so because these logits are near zero, for the most part, the loss that comes"}, {"start": 471.52, "end": 481.68, "text": " out is okay. But suppose this is like time 10 now. You see how because these are more extreme values,"}, {"start": 481.68, "end": 487.12, "text": " it's very unlikely that you're going to be guessing the correct bucket and then you're confidently"}, {"start": 487.12, "end": 493.28, "text": " wrong and recording very high loss. If your logits are coming up even more extreme, you might get"}, {"start": 493.28, "end": 502.0, "text": " extremely same losses like infinity even at initialization. So basically this is not good and we want"}, {"start": 502.0, "end": 509.44, "text": " the logits to be roughly zero when the work is initialized. In fact, the logits can don't have to"}, {"start": 509.44, "end": 514.56, "text": " be just zero. They just have to be equal. So for example, if all the logits are one, then because of"}, {"start": 514.56, "end": 519.6, "text": " the normalization inside the softmax, this will actually come out okay. But by symmetry, we don't"}, {"start": 519.6, "end": 524.32, "text": " want it to be any arbitrary positive or negative number. We just want it to be all zeros and record"}, {"start": 524.32, "end": 528.32, "text": " the loss that we expect at initialization. So let's not completely see where things go wrong"}, {"start": 528.32, "end": 534.32, "text": " in our example. Here we have the initialization. Let me initialize the neural ad. And here let me"}, {"start": 534.32, "end": 540.4, "text": " break after the very first iteration. So we only see the initial loss, which is 27. So that's"}, {"start": 540.4, "end": 545.44, "text": " way too high. And intuitively now we can expect the variables involved. And we see that the logits"}, {"start": 545.44, "end": 551.9200000000001, "text": " here, if we just print some of these, if we just print the first row, we see that the logits take"}, {"start": 551.9200000000001, "end": 558.24, "text": " on quite extreme values. And that's what's creating the fake confidence and incorrect answers. And"}, {"start": 558.24, "end": 565.84, "text": " makes the loss get very, very high. So these logits should be much, much closer to zero. So now let's"}, {"start": 565.84, "end": 571.6800000000001, "text": " think through how we can achieve logits coming out of this neural net to be more closer to zero."}, {"start": 571.68, "end": 576.9599999999999, "text": " You see here that the logits are calculated as they hit in states multiplied by W2 plus B2."}, {"start": 577.5999999999999, "end": 583.28, "text": " So first of all, currently we're initializing B2 as random values of the right size."}, {"start": 584.2399999999999, "end": 589.28, "text": " But because we want roughly zero, we don't actually want to be adding a bias of random numbers."}, {"start": 589.28, "end": 595.68, "text": " So in fact, I'm going to add a times a zero here to make sure that B2 is just basically zero"}, {"start": 595.68, "end": 602.2399999999999, "text": " at initialization. And second, this is H multiplied by W2. So if we want logits to be very, very small,"}, {"start": 603.04, "end": 609.28, "text": " then we would be multiplying W2 and making that small. So for example, if we scale down W2 by"}, {"start": 609.28, "end": 615.4399999999999, "text": " 0.1 all the elements, then if I do again just a very first iteration, you see that we are getting"}, {"start": 615.4399999999999, "end": 622.8, "text": " much closer to what we expect. So roughly what we want is about 3.29, this is 4.2. I can make this"}, {"start": 622.8, "end": 630.88, "text": " maybe even smaller, 3.32. Okay, so we're getting closer and closer. Now, you're probably wondering,"}, {"start": 630.88, "end": 637.3599999999999, "text": " can we just set this to zero? Then we get, of course, exactly what we're looking for at initialization."}, {"start": 638.16, "end": 644.16, "text": " And the reason I don't usually do this is because I'm very nervous and I'll show you in a second why"}, {"start": 644.16, "end": 649.92, "text": " you don't want to be setting W's or weights of a neural net exactly to zero. You usually want"}, {"start": 649.92, "end": 655.5999999999999, "text": " to be small numbers instead of exactly zero. For this output layer in this specific case,"}, {"start": 655.5999999999999, "end": 660.0, "text": " I think it would be fine, but I'll show you in a second where things go wrong very quickly if you"}, {"start": 660.0, "end": 666.56, "text": " do that. So let's just go with 0.01. In that case, our loss is close enough, but has some entropy."}, {"start": 666.56, "end": 672.0799999999999, "text": " It's not exactly zero. It's got some low entropy and that's used for symmetry braking as we'll"}, {"start": 672.0799999999999, "end": 677.52, "text": " see in a second. Logits are now coming out much closer to zero and everything is well and good."}, {"start": 677.52, "end": 686.48, "text": " So if I just erase these and I now take away the brake statement, we can run the optimization"}, {"start": 686.48, "end": 693.84, "text": " with this new initialization. And let's just see what losses we record. Okay, so I'll let it run"}, {"start": 693.84, "end": 700.24, "text": " and you see that we started off good and then we came down a bit. The plot of the loss now doesn't"}, {"start": 700.24, "end": 706.64, "text": " have this hockey shape appearance because basically what's happening in the hockey stick, the very first"}, {"start": 706.64, "end": 711.6, "text": " few iterations of the loss, what's happening during the optimization is the optimization is just"}, {"start": 711.6, "end": 717.4399999999999, "text": " squashing down the logits and then it's rearranging the logits. So basically we took away this easy part"}, {"start": 717.4399999999999, "end": 723.68, "text": " of the loss function where just the weights were just being shrunk down. And so therefore we don't"}, {"start": 723.68, "end": 727.68, "text": " get these easy gains in the beginning and we're just getting some of the hard gains of training"}, {"start": 727.68, "end": 733.4399999999999, "text": " the actual neural nut. And so there's no hockey stick appearance. So good things are happening in that"}, {"start": 733.44, "end": 739.84, "text": " both number one loss and initialization is what we expect. And the loss doesn't look like a hockey"}, {"start": 739.84, "end": 746.24, "text": " stick. And this is true for any neural nut you might train and something to look at for. And second,"}, {"start": 746.24, "end": 751.2800000000001, "text": " the loss that came out is actually quite a bit improved. Unfortunately, I erased what we had here"}, {"start": 751.2800000000001, "end": 760.32, "text": " before. I believe this was 2.12 and this was 2.16. So we get a slightly improved result. And the"}, {"start": 760.32, "end": 766.48, "text": " reason for that is because we're spending more cycles, more time optimizing the neural nut actually"}, {"start": 766.48, "end": 772.0, "text": " instead of just spending the first several thousand iterations probably just squashing down the weights"}, {"start": 773.12, "end": 778.1600000000001, "text": " because they are so way too high in the beginning and the initialization. So something to look out for"}, {"start": 778.1600000000001, "end": 782.96, "text": " and that's number one. Now let's look at the second problem. Let me re-initialize our neural"}, {"start": 782.96, "end": 789.12, "text": " nut and let me reintroduce the break statement. So we have a reasonable initial loss. So even though"}, {"start": 789.12, "end": 792.64, "text": " everything is looking good on the level of the loss and we get something that we expect,"}, {"start": 792.64, "end": 796.4, "text": " there's still a deeper problem lurking inside this neural nut and its initialization."}, {"start": 797.44, "end": 804.32, "text": " So the logits are now okay. The problem now is with the values of H, the activations of the hidden"}, {"start": 804.32, "end": 810.4, "text": " states. Now if we just visualize this vector, sorry, this tensor H, it's kind of hard to see but"}, {"start": 810.4, "end": 815.04, "text": " the problem here roughly speaking is you see how many of the elements are one or negative one."}, {"start": 815.04, "end": 821.76, "text": " Now recall that tortsh.10h, the 10h function is a squashing function. It takes arbitrary numbers"}, {"start": 821.76, "end": 826.64, "text": " and it squashes them into a range of negative one and one and it does so smoothly. So let's look at"}, {"start": 826.64, "end": 831.52, "text": " the histogram of H to get a better idea of the distribution of the values inside this tensor."}, {"start": 832.3199999999999, "end": 840.8, "text": " We can do this first. Well we can see that H is 32 examples and 200 activations in each example."}, {"start": 840.8, "end": 847.52, "text": " We can view it as negative one to stretch it out into one large vector and we can then call"}, {"start": 847.52, "end": 855.04, "text": " two lists to convert this into one large Python list of floats and then we can pass this into"}, {"start": 855.04, "end": 862.3199999999999, "text": " plt.hist for histogram and we say we want 50 bins and a semicolon to suppress a bunch of output we"}, {"start": 862.3199999999999, "end": 868.8, "text": " don't want. So we see this histogram and we see that most of the values by far take on value of"}, {"start": 868.8, "end": 876.88, "text": " negative one and one. So this 10h is very very active and we can also look at basically why that is"}, {"start": 877.8399999999999, "end": 884.24, "text": " we can look at the preactivations that feed into the 10h and we can see that the distribution of"}, {"start": 884.24, "end": 890.4, "text": " the preactivations are is very very broad. These take numbers between negative 15 and 15 and"}, {"start": 890.4, "end": 894.8, "text": " that's why in a tortsh.10h everything is being squashed and capped to be in the range of negative"}, {"start": 894.8, "end": 900.4799999999999, "text": " one and one and lots of numbers here take on very extreme values. Now if you are new to it in your"}, {"start": 900.4799999999999, "end": 905.3599999999999, "text": " networks you might not actually see this as an issue but if you're well burst in the dark arts"}, {"start": 905.3599999999999, "end": 909.4399999999999, "text": " of back propagation and then have an intuitive sense of how these radians float through in your"}, {"start": 909.4399999999999, "end": 914.3199999999999, "text": " own net you are looking at your distribution of 10h activations here and you are sweating."}, {"start": 914.88, "end": 919.04, "text": " So let me show you why. We have to keep in mind that during back propagation just like we saw in"}, {"start": 919.04, "end": 924.7199999999999, "text": " micrograd we are doing backward pass starting at the loss and flowing through the network backwards."}, {"start": 924.7199999999999, "end": 930.4, "text": " In particular we're going to back propagate through this tortsh.10h and this layer here is made"}, {"start": 930.4, "end": 936.9599999999999, "text": " up of 200 neurons for each one of these examples and it implements an element twice 10h. So let's"}, {"start": 936.9599999999999, "end": 942.4, "text": " look at what happens in 10h in the backward pass. We can actually go back to our previous micrograd"}, {"start": 942.4, "end": 949.1999999999999, "text": " code in the very first lecture and see how we implement the 10h. We saw that the input here was x"}, {"start": 949.1999999999999, "end": 954.88, "text": " and then we calculate t which is the 10h of x. So that's t and t is between negative one and one."}, {"start": 954.88, "end": 959.1999999999999, "text": " It's the output of the 10h and then in the backward pass how do we back propagate through a 10h?"}, {"start": 960.0799999999999, "end": 966.16, "text": " We take out that grad and then we multiply it this is the chain rule with the local gradient"}, {"start": 966.16, "end": 972.0799999999999, "text": " which took the form of 1 minus t squared. So what happens if the outputs of your 10h are very close"}, {"start": 972.08, "end": 978.4000000000001, "text": " to negative one or one. If you plug in t equals one here you're going to get a zero multiplying"}, {"start": 978.4000000000001, "end": 984.32, "text": " out that grad. No matter what out that grad is we are killing the gradient and we're stopping"}, {"start": 984.32, "end": 989.44, "text": " effectively the back propagation through this 10h unit. Similarly when t is negative one this will"}, {"start": 989.44, "end": 995.6800000000001, "text": " again become zero and out that grad just stops and intuitively this makes sense because this is a"}, {"start": 995.6800000000001, "end": 1001.9200000000001, "text": " 10h neuron and what's happening is if its output is very close to one then we are in the"}, {"start": 1001.92, "end": 1011.28, "text": " tail of this 10h. So changing basically the input is not going to impact the output of the 10h"}, {"start": 1011.28, "end": 1017.8399999999999, "text": " too much because it's in the flat region of the 10h and so therefore there's no impact on the loss."}, {"start": 1018.4799999999999, "end": 1025.44, "text": " And so indeed the weights and the biases along with this 10h neuron do not impact the loss"}, {"start": 1025.44, "end": 1029.6, "text": " because the output of this 10h unit is in the flat region of the 10h and there's no influence."}, {"start": 1029.6, "end": 1034.48, "text": " We can we can be changing them whatever we want however we want and the loss is not impacted."}, {"start": 1034.48, "end": 1039.6799999999998, "text": " That's another way to justify that indeed the gradient would be basically zero advantages."}, {"start": 1040.8799999999999, "end": 1049.84, "text": " Indeed when t equals zero we get one times out that grad so when the 10h takes on exactly"}, {"start": 1049.84, "end": 1056.8, "text": " value of zero then out that grad is just passed through. So basically what this is doing right is"}, {"start": 1056.8, "end": 1064.72, "text": " if t is equal to zero then this the 10h unit is sort of inactive and gradient just passes through"}, {"start": 1064.72, "end": 1070.8799999999999, "text": " but the more you are in the flat tails the more degrading is squashed. So in fact you'll see that"}, {"start": 1070.8799999999999, "end": 1076.8, "text": " the gradient flowing through 10h can only ever decrease in the amount that it decreases is"}, {"start": 1077.6, "end": 1083.84, "text": " proportional through a square here depending on how far you are in the flat tails of this 10h."}, {"start": 1083.84, "end": 1090.6399999999999, "text": " And so that's kind of what's happening here and through this the concern here is that if all"}, {"start": 1090.6399999999999, "end": 1096.3999999999999, "text": " of these outputs h are in the flat regions of negative one to one then the gradients that are"}, {"start": 1096.3999999999999, "end": 1102.3999999999999, "text": " flowing through the network will just get destroyed at this layer. Now there is some redeeming"}, {"start": 1102.9599999999998, "end": 1106.56, "text": " quality here and that we can actually get a sense of the problem here as follows."}, {"start": 1107.6, "end": 1112.24, "text": " I brought some code here and basically what we want to do here is we want to take a look at"}, {"start": 1112.24, "end": 1119.52, "text": " h, take the absolute value and see how often it is in the in a flat region. So say"}, {"start": 1120.24, "end": 1127.52, "text": " grading point 99 and what you get is the following and this is a Boolean tensor. So in the Boolean"}, {"start": 1127.52, "end": 1133.28, "text": " tensor you get a white if this is true and a black if this is false. And so basically what we"}, {"start": 1133.28, "end": 1139.68, "text": " have here is the 32 examples and a 200 hidden neurons and we see that a lot of this is white"}, {"start": 1139.68, "end": 1147.28, "text": " and what that's telling us is that all these 10h neurons were very very active and they're in"}, {"start": 1147.28, "end": 1154.8, "text": " the flat tail and so in all these cases the backward gradient would get destroyed."}, {"start": 1156.48, "end": 1163.3600000000001, "text": " Now we would be in a lot of trouble if for for any one of these 200 neurons if it was the case"}, {"start": 1163.3600000000001, "end": 1168.8, "text": " that the entire column is white because in that case we have what's called a dead neuron and"}, {"start": 1168.8, "end": 1173.12, "text": " this could be a tenet neuron where the initialization of the weights in the biases could be such that"}, {"start": 1173.12, "end": 1180.8799999999999, "text": " no single example ever activates this 10h in the sort of active part of the 10h. If all the examples"}, {"start": 1180.8799999999999, "end": 1188.3999999999999, "text": " land in the tail then this neuron will never learn it is a dead neuron. And so just scrutinizing"}, {"start": 1188.3999999999999, "end": 1195.84, "text": " this and looking for columns of completely white we see that this is not the case. So I don't see"}, {"start": 1195.84, "end": 1202.24, "text": " a single neuron that is all of white and so therefore it is the case that for every one of these"}, {"start": 1202.24, "end": 1209.4399999999998, "text": " 10h neurons we do have some examples that activate them in the active part of the 10h and so some"}, {"start": 1209.4399999999998, "end": 1214.56, "text": " gradients will flow through this neuron will learn and neuron will change and it will move and it"}, {"start": 1214.56, "end": 1220.48, "text": " will do something. But you can sometimes get yourself in cases where you have dead neurons and"}, {"start": 1220.48, "end": 1226.32, "text": " weight this manifests is that for 10h neuron this would be when no matter what inputs you plug in"}, {"start": 1226.32, "end": 1231.44, "text": " from your data set this 10h neuron always fires completely one or completely negative one and"}, {"start": 1231.44, "end": 1237.3600000000001, "text": " then it will just not learn because all the gradients will be just zero that. This is true not just"}, {"start": 1237.3600000000001, "end": 1242.0, "text": " for 10h but for a lot of other non-linearities that people use in neural networks. So we certainly"}, {"start": 1242.0, "end": 1247.84, "text": " use 10h a lot but sigmoid will have the exact same issue because it is a squashing neuron and so"}, {"start": 1247.84, "end": 1257.04, "text": " the same will be true for sigmoid but you know basically the same will actually apply to sigmoid."}, {"start": 1257.04, "end": 1262.6399999999999, "text": " The same will also apply to a relu so relu has a completely flat region here below zero."}, {"start": 1263.4399999999998, "end": 1270.56, "text": " So if you have a relu neuron then it is a pass-through if it is positive and if the pre-activation is"}, {"start": 1270.56, "end": 1275.6799999999998, "text": " negative it will just shut it off. Since the region here is completely flat then during back"}, {"start": 1275.68, "end": 1281.92, "text": " propagation this would be exactly zeroing out the gradient. Like all of the gradient would be"}, {"start": 1281.92, "end": 1286.72, "text": " set exactly to zero instead of just like a very very small number depending on how positive or"}, {"start": 1286.72, "end": 1293.3600000000001, "text": " negative t is. And so you can get for example a dead relu neuron and a dead relu neuron would"}, {"start": 1293.3600000000001, "end": 1300.4, "text": " basically look like. Basically what it is is if a neuron with a relu non-linearity never activates"}, {"start": 1300.4, "end": 1306.0800000000002, "text": " it. So for any examples that you plug in in the data set it never turns on it is always in this"}, {"start": 1306.0800000000002, "end": 1312.3200000000002, "text": " flat region then this relu neuron is a dead neuron. Its weights and bias will never learn they will"}, {"start": 1312.3200000000002, "end": 1317.76, "text": " never get agradient because the neuron never activated. And this can sometimes happen at initialization"}, {"start": 1317.76, "end": 1322.0800000000002, "text": " because the weights in the bias is just make it so that by chance some neurons are just forever dead"}, {"start": 1322.72, "end": 1327.52, "text": " but it can also happen during optimization. If you have like a too high learning weight for example"}, {"start": 1327.52, "end": 1331.6, "text": " sometimes you have these neurons that get too much of a gradient and they get knocked out"}, {"start": 1331.6, "end": 1337.84, "text": " off the data manifold. And what happens is that from then on no example ever activates this neuron"}, {"start": 1337.84, "end": 1341.44, "text": " so this neuron remains dead forever. So it's kind of like a permanent brain damage in a"}, {"start": 1341.44, "end": 1346.6399999999999, "text": " in a mind of a network. And so sometimes what can happen is if your learning rate is very high"}, {"start": 1346.6399999999999, "end": 1351.52, "text": " for example and you have a neural net with relu neurons you train the neural net and you get some"}, {"start": 1351.52, "end": 1357.2, "text": " last loss. But then actually what you do is you go through the entire training set and you forward"}, {"start": 1357.2, "end": 1364.0, "text": " on your examples and you can find neurons that never activate they are dead neurons in your network"}, {"start": 1364.0, "end": 1368.32, "text": " and so those neurons will will never turn on. And usually what happens is that during training"}, {"start": 1368.32, "end": 1373.04, "text": " these relu neurons are changing moving etc and then because of a high gradient somewhere by chance"}, {"start": 1373.04, "end": 1377.84, "text": " they get knocked off and then nothing ever activates them and from then on they are just dead."}, {"start": 1377.84, "end": 1382.24, "text": " So that's kind of like a permanent brain damage that can happen to some of these neurons."}, {"start": 1382.24, "end": 1387.76, "text": " These other nonlinearities like leaky relu will not suffer from this issue as much because you can"}, {"start": 1387.76, "end": 1395.36, "text": " see that it doesn't have flat tails. You almost always get gradients and ilu is also fairly frequently"}, {"start": 1395.36, "end": 1401.36, "text": " used. It also might suffer from this issue because it has flat parts. So that's just something to"}, {"start": 1401.36, "end": 1408.24, "text": " be aware of and something to be concerned about. And in this case we have way too many activations"}, {"start": 1408.24, "end": 1414.32, "text": " age that take on extreme values and because there's no column of white I think we will be okay"}, {"start": 1414.32, "end": 1419.12, "text": " and indeed the network optimizes and gives us a pretty decent loss but it's just not optimal and"}, {"start": 1419.12, "end": 1423.9199999999998, "text": " this is not something you want especially during initialization. And so basically what's happening"}, {"start": 1423.92, "end": 1432.0, "text": " is that this age preactivation that's flowing to 10h it's too extreme it's too large it's creating"}, {"start": 1432.0, "end": 1437.68, "text": " very it's creating distribution that is too saturated in both sides of the 10h and it's not"}, {"start": 1437.68, "end": 1443.68, "text": " something you want because it needs that there's less training for these neurons because they update"}, {"start": 1444.72, "end": 1453.1200000000001, "text": " less frequently. So how do we fix this? Well, age preactivation is MCAT which comes from C. So these"}, {"start": 1453.12, "end": 1460.2399999999998, "text": " are uniform Gaussian but then multiply by W1 plus B1 and age preact is too far off from 0 and"}, {"start": 1460.2399999999998, "end": 1465.6799999999998, "text": " that's causing the issue. So we want this preactivation to be closer to 0 very similar to what we have"}, {"start": 1465.6799999999998, "end": 1473.84, "text": " with logists. So here we want actually something very very similar. Now it's okay to set the biases"}, {"start": 1473.84, "end": 1478.08, "text": " to very small number. We can either multiply by 0, 0, 1 to get like a little bit of entropy."}, {"start": 1478.08, "end": 1485.9199999999998, "text": " I sometimes like to do that just so that there's like a little bit of variation and diversity in"}, {"start": 1485.9199999999998, "end": 1492.1599999999999, "text": " the original initialization of these 10h neurons and I find in practice that that can help optimization"}, {"start": 1492.1599999999999, "end": 1497.76, "text": " a little bit and then the weights we can also just like squash. So let's multiply everything by"}, {"start": 1497.76, "end": 1505.12, "text": " 0.1. Let's rerun the first batch and now let's look at this and well first let's look at here."}, {"start": 1505.12, "end": 1511.76, "text": " You see now because we multiply W by 0.1 we have a much better histogram and that's because the"}, {"start": 1511.76, "end": 1517.52, "text": " preactivations are now between negative 1.5 and 1.5 and this we expect much much less white."}, {"start": 1518.56, "end": 1526.32, "text": " Okay, there's no white. So basically that's because there are no neurons that saturated above 0.99"}, {"start": 1526.32, "end": 1535.28, "text": " in either direction. So it's actually a pretty decent place to be. Maybe we can go up a little bit."}, {"start": 1536.6399999999999, "end": 1544.8799999999999, "text": " Sorry am I changing W1 here? Maybe we can go to 0.2. Okay, so maybe something like this is a"}, {"start": 1544.8799999999999, "end": 1550.24, "text": " nice distribution. So maybe this is what our initialization should be. So let me now erase"}, {"start": 1550.24, "end": 1559.76, "text": " these and let me starting with initialization. Let me run the full optimization without the break"}, {"start": 1560.48, "end": 1566.64, "text": " and let's see what we get. Okay, so the optimization finished and I rerun the loss and this is the"}, {"start": 1566.64, "end": 1571.04, "text": " result that we get and then just as a reminder I put down all the losses that we saw previously in"}, {"start": 1571.04, "end": 1576.56, "text": " this lecture. So we see that we actually do get an improvement here and just as a reminder we"}, {"start": 1576.56, "end": 1581.2, "text": " started off with the validation loss of 2.17 when we started. By fixing the softmax being"}, {"start": 1581.2, "end": 1586.8, "text": " confidently wrong we came down to 2.13 and by fixing the 10H layer being way too saturated we"}, {"start": 1586.8, "end": 1591.9199999999998, "text": " came down to 2.10 and the reason this is happening of course is because our initialization is better"}, {"start": 1591.9199999999998, "end": 1598.0, "text": " and so we're spending more time being productive training instead of not very productive training"}, {"start": 1598.0, "end": 1604.0, "text": " because our gradients are set to zero and we have to learn very simple things like the overconfidence"}, {"start": 1604.0, "end": 1608.0, "text": " of the softmax in the beginning and we're spending cycles just like squashing down the white matrix."}, {"start": 1608.96, "end": 1616.96, "text": " So this is illustrating basically initialization and its impacts on performance just by being aware"}, {"start": 1616.96, "end": 1621.76, "text": " of the intervals of these neural nets and their actuations and their gradients. Now we're working"}, {"start": 1621.76, "end": 1626.96, "text": " with a very small network. This is just one layer, multi-layer perception. So because the network is"}, {"start": 1626.96, "end": 1632.24, "text": " so shallow the optimization problem is actually quite easy and very forgiving. So even though our"}, {"start": 1632.24, "end": 1637.04, "text": " initialization was terrible the network still learned eventually it just got a bit worse result."}, {"start": 1637.04, "end": 1642.72, "text": " This is not the case in general though. Once we actually start working with much deeper networks"}, {"start": 1642.72, "end": 1649.52, "text": " that have saved 50 layers things can get much more complicated and these problems stack up"}, {"start": 1650.4, "end": 1655.2, "text": " and so you can actually get into a place where the network is basically not training at all if"}, {"start": 1655.2, "end": 1660.0, "text": " your initialization is bad enough and the deeper your network is and the more complex it is the"}, {"start": 1660.0, "end": 1666.16, "text": " less forgiving it is to some of these errors. And so something to be definitely aware of"}, {"start": 1667.44, "end": 1673.04, "text": " something to scrutinize, something to plot and something to be careful with and yeah."}, {"start": 1673.52, "end": 1677.6, "text": " Okay so that's great that that worked for us but what we have here now is all these"}, {"start": 1677.6, "end": 1682.0, "text": " magic numbers like point two like where do I come up with this and how am I supposed to set these"}, {"start": 1682.0, "end": 1687.04, "text": " if I have a large neural net with lots and lots of layers and so obviously no one does this by"}, {"start": 1687.04, "end": 1692.3999999999999, "text": " hand there's actually some relatively principled ways of setting these scales that I would like to"}, {"start": 1692.3999999999999, "end": 1697.52, "text": " introduce to you now. So let me paste some code here that I prepared just to motivate the discussion"}, {"start": 1697.52, "end": 1704.6399999999999, "text": " of this. So what I'm doing here is we have some random input here x that is drawn from a Gaussian"}, {"start": 1705.28, "end": 1711.84, "text": " and there's 1000 examples that are 10 dimensional and then we have a weight layer here that is also"}, {"start": 1711.84, "end": 1717.9199999999998, "text": " initialized using Gaussian just like we did here and we these neurons in the hidden layer look at 10"}, {"start": 1717.9199999999998, "end": 1723.4399999999998, "text": " inputs and there are 200 neurons in this hidden layer and then we have here just like here"}, {"start": 1724.24, "end": 1729.6799999999998, "text": " in this case the multiplication x multiply by w to get the preactivations of these neurons."}, {"start": 1730.8, "end": 1735.9199999999998, "text": " And basically the analysis here looks at okay suppose these are uniform Gaussian and these weights"}, {"start": 1735.92, "end": 1742.24, "text": " are uniform Gaussian. If I do x times w and we forget for now the bias and the nonlinearity"}, {"start": 1743.28, "end": 1748.0800000000002, "text": " then what is the mean and the standard deviation of these Gaussians? So in the beginning here the"}, {"start": 1748.0800000000002, "end": 1753.76, "text": " input is just a normal Gaussian distribution mean zero and the standard deviation is one and the"}, {"start": 1753.76, "end": 1759.6000000000001, "text": " standard deviation again is just the measure of a spread of the Gaussian. But then once we multiply"}, {"start": 1759.6, "end": 1766.3999999999999, "text": " here and we look at the histogram of y we see that the mean of course stays the same it's about zero"}, {"start": 1766.9599999999998, "end": 1771.6, "text": " because this is a symmetric operation but we see here that the standard deviation has expanded to"}, {"start": 1771.6, "end": 1777.04, "text": " three. So the inputs and deviation was one but now we've grown to three and so what you're seeing"}, {"start": 1777.04, "end": 1785.04, "text": " in the histogram is that this Gaussian is expanding and so we're expanding this Gaussian from the"}, {"start": 1785.04, "end": 1790.56, "text": " input and we don't want that we want most of the neural nets to have relatively similar activations"}, {"start": 1790.56, "end": 1795.52, "text": " so unit Gaussian roughly throughout the neural net. And so the question is how do we scale these"}, {"start": 1795.52, "end": 1805.2, "text": " w's to preserve the to preserve this distribution to remain a Gaussian. And so intuitively if I"}, {"start": 1805.2, "end": 1814.32, "text": " multiply here these elements of w by a large number like say by 5 then this Gaussian grows and"}, {"start": 1814.32, "end": 1820.1599999999999, "text": " grows in standard deviation so now we're at 15 so basically these numbers here in the output y"}, {"start": 1820.1599999999999, "end": 1826.8, "text": " take on more and more extreme values but if we scale it down like say 0.2 then conversely"}, {"start": 1826.8, "end": 1832.1599999999999, "text": " this Gaussian is getting smaller and smaller and it's shrinking and you can see that the standard"}, {"start": 1832.1599999999999, "end": 1838.6399999999999, "text": " deviation is 0.6 and so the question is what do I multiply by here to exactly preserve the standard"}, {"start": 1838.6399999999999, "end": 1843.4399999999998, "text": " deviation to be one and it turns out that the correct answer mathematically when you work out"}, {"start": 1843.44, "end": 1850.0800000000002, "text": " through the variance of this multiplication here is that you are supposed to divide by the"}, {"start": 1850.0800000000002, "end": 1858.3200000000002, "text": " square root of the fan in. The fan in is basically the number of input components here 10 so we"}, {"start": 1858.3200000000002, "end": 1862.96, "text": " are supposed to divide by 10 square root and this is one way to do the square root you raise it"}, {"start": 1862.96, "end": 1869.76, "text": " to a power of 0.5 and that's the same as doing a square root. So when you divide by the square root"}, {"start": 1869.76, "end": 1878.64, "text": " of 10 then we see that the output Gaussian it has exactly standard deviation of 1. Now unsurprisingly"}, {"start": 1878.64, "end": 1884.24, "text": " a number of papers have looked into how but to best initialize neural networks and in the case"}, {"start": 1884.24, "end": 1888.64, "text": " in multiple perceptions we can have fairly deep networks that had these non-linearities in between"}, {"start": 1889.12, "end": 1893.44, "text": " and we want to make sure that the activations are well behaved and they don't expand to infinity"}, {"start": 1893.44, "end": 1897.6, "text": " or shrink all the way to 0 and the question is how do we initialize the weights so that these"}, {"start": 1897.6, "end": 1902.7199999999998, "text": " activations take on reasonable values throughout the network. Now one paper that has studied this"}, {"start": 1902.7199999999998, "end": 1907.84, "text": " in quite a bit detail that is often referenced is this paper by Kaminghe Tull called delving deep"}, {"start": 1907.84, "end": 1913.28, "text": " interactive fires. Now in this case they actually study convolutional neural networks and they study"}, {"start": 1913.28, "end": 1919.84, "text": " especially the relu nullinarity and the p-relu nullinarity instead of a 10-h nullinarity"}, {"start": 1919.84, "end": 1925.6, "text": " but the analysis is very similar and basically what happens here is for them the"}, {"start": 1925.6, "end": 1932.32, "text": " the relu nullinarity that they care about quite a bit here is a squashing function where all the"}, {"start": 1932.32, "end": 1938.32, "text": " negative numbers are simply clamped to 0 so the positive numbers are a path through but everything"}, {"start": 1938.32, "end": 1944.6399999999999, "text": " negative is just set to 0 and because you are basically throwing away half of the distribution"}, {"start": 1944.6399999999999, "end": 1948.9599999999998, "text": " they find in their analysis of the forward activations in the neural net that you have to"}, {"start": 1948.96, "end": 1956.88, "text": " complicate for that with a gain and so here they find that basically when they initialize their"}, {"start": 1956.88, "end": 1961.28, "text": " weights they have to do it with a zero mean Gaussian whose standard deviation is square root of"}, {"start": 1961.28, "end": 1967.76, "text": " 2 over the fanning what we have here is we are initializing the Gaussian with the square root of"}, {"start": 1967.76, "end": 1974.48, "text": " fanning this NL here is the fanning so what we have is square root of 1 over the fanning"}, {"start": 1974.48, "end": 1980.96, "text": " because we have the division here now they have to add this factor of 2 because of the relu"}, {"start": 1980.96, "end": 1986.24, "text": " which basically discards half of the distribution and clamps it at 0 and so that's where you get"}, {"start": 1986.24, "end": 1992.64, "text": " an initial factor now in addition to that this paper also studies not just the sort of behavior"}, {"start": 1992.64, "end": 1997.68, "text": " of the activations in the forward path of the neural net but it also studies the back propagation"}, {"start": 1997.68, "end": 2001.44, "text": " and we have to make sure that the gradients also are well behaved and so"}, {"start": 2001.44, "end": 2007.76, "text": " because ultimately they end up updating our parameters and what they find here through a lot of"}, {"start": 2007.76, "end": 2012.3200000000002, "text": " the analysis that I am much into read through but it's not exactly approachable what they find"}, {"start": 2012.3200000000002, "end": 2017.6000000000001, "text": " is basically if you properly initialize the forward path the backward path is also"}, {"start": 2017.6000000000001, "end": 2023.52, "text": " approximately initialized up to a constant factor that has to do with the size of the number of"}, {"start": 2024.3200000000002, "end": 2031.2, "text": " hidden neurons and early and late layer and but basically they find empirically that"}, {"start": 2031.2, "end": 2037.2, "text": " this is not a choice that matters too much now this chiming initialization is also implemented in"}, {"start": 2037.2, "end": 2041.8400000000001, "text": " PyTorch so if you go to torshtat and end up in it documentation you'll find chiming normal"}, {"start": 2042.48, "end": 2046.4, "text": " and in my opinion this is probably the most common way of initializing neural networks now"}, {"start": 2047.3600000000001, "end": 2052.8, "text": " and it takes a few keyword arguments here so another one it wants to know the mode"}, {"start": 2052.8, "end": 2057.12, "text": " would you like to normalize the activations or would you like to normalize the gradients to"}, {"start": 2057.12, "end": 2063.44, "text": " to be always caution with zero mean and unit or one standard deviation and because they find"}, {"start": 2063.44, "end": 2067.12, "text": " a paper that this doesn't matter too much most of the people just leave it as the default which is"}, {"start": 2067.12, "end": 2073.52, "text": " pen in and then second pass in the nullinearity that you are using because depending on the nullinearity"}, {"start": 2073.52, "end": 2079.2799999999997, "text": " we need to calculate a slightly different gain and so if your nullinearity is just a linear"}, {"start": 2079.2799999999997, "end": 2084.72, "text": " so there's no nullinearity then the gain here will be one and we have these same kind of formula"}, {"start": 2084.72, "end": 2088.72, "text": " that we've caught here but if the nullinearity is something else we're going to get a slightly"}, {"start": 2088.72, "end": 2093.68, "text": " different gain and so if we come up here to the top we see that for example in the case of"}, {"start": 2093.68, "end": 2098.8799999999997, "text": " relu this gain is a square root of two and the reason is the square root because in this paper"}, {"start": 2103.04, "end": 2108.08, "text": " you see how the two is inside of the square root so the gain is a square root of two"}, {"start": 2108.08, "end": 2115.2799999999997, "text": " in a case of linear or identity we just get a gain of one in a case of 10h which is what we're"}, {"start": 2115.2799999999997, "end": 2121.44, "text": " using here the advised gain is a 5 over 3 and intuitively why do we need a gain on top of the"}, {"start": 2121.44, "end": 2128.08, "text": " initialization is because 10h just like relu is a contractive transformation so what that means"}, {"start": 2128.08, "end": 2132.56, "text": " is you're taking the output distribution from this matrix multiplication and then you are squashing"}, {"start": 2132.56, "end": 2137.2, "text": " it in some way now relu squashes it by taking everything below zero and clamping it to zero"}, {"start": 2137.2, "end": 2142.0, "text": " 10h also squashes it because it's a contractive operation it will take the tails and it will"}, {"start": 2142.8799999999997, "end": 2148.96, "text": " squeeze them in and so in order to fight the squeezing in we need to boost the weights a little bit"}, {"start": 2148.96, "end": 2154.72, "text": " so that we re-normalize everything back to unit standard deviation so that's why there's a"}, {"start": 2154.72, "end": 2158.72, "text": " little bit of a gain that comes out now i'm skipping through this section a little bit quickly"}, {"start": 2158.72, "end": 2164.16, "text": " and i'm doing that actually intentionally and the reason for that is because about seven years ago"}, {"start": 2164.16, "end": 2168.7999999999997, "text": " when this paper was written you had to actually be extremely careful with the activations and"}, {"start": 2168.7999999999997, "end": 2173.12, "text": " ingredients and their ranges and their histograms and you had to be very careful with the"}, {"start": 2173.12, "end": 2177.6, "text": " precise setting of gains and the scrutinizing of the null and your t's used and so on and everything"}, {"start": 2177.6, "end": 2182.48, "text": " was very finicky and very fragile and to be very properly arranged for the neural network to train"}, {"start": 2182.48, "end": 2186.3199999999997, "text": " especially if your neural network was very deep but there are a number of modern innovations that"}, {"start": 2186.3199999999997, "end": 2190.8799999999997, "text": " have made everything significantly more stable and more well behaved and it's become less important"}, {"start": 2190.88, "end": 2196.32, "text": " to initialize these networks exactly right and some of those modern innovations for example are"}, {"start": 2196.32, "end": 2202.1600000000003, "text": " residual connections which we will cover in the future the use of a number of normalization"}, {"start": 2202.1600000000003, "end": 2207.44, "text": " layers like for example bash normalization layer normalization work normalization we're going to"}, {"start": 2207.44, "end": 2212.0, "text": " go into a lot of these as well and number three much better optimizers not just to cast a gradient"}, {"start": 2212.0, "end": 2217.2000000000003, "text": " scent the simple optimizer we're basically using here but slightly more complex optimizers"}, {"start": 2217.2, "end": 2222.0, "text": " like armistrop and especially atom and so all of these modern innovations make it less"}, {"start": 2222.0, "end": 2226.96, "text": " important for you to precisely calibrate the initialization of the neural net all that being said"}, {"start": 2226.96, "end": 2232.96, "text": " in practice what should we do in practice when I initialize these neural nets I basically just"}, {"start": 2232.96, "end": 2239.4399999999996, "text": " normalize my weights by the square root of the pen and so basically roughly what we did here"}, {"start": 2239.4399999999996, "end": 2245.7599999999998, "text": " is what I do now if we want to be exactly accurate here we and go by in it of"}, {"start": 2245.76, "end": 2251.44, "text": " kind of normal this is how we would implement it we want to set the standard deviation to be"}, {"start": 2251.44, "end": 2259.28, "text": " gained over the square root of fan in right so to set the standard deviation of our weights we will"}, {"start": 2259.28, "end": 2265.84, "text": " proceed as follows basically when we have torshtot ran and let's say I just create a thousand numbers"}, {"start": 2265.84, "end": 2269.92, "text": " we can look at the standard deviation of this and of course that's one that's the amount of spread"}, {"start": 2269.92, "end": 2274.88, "text": " let's make this a bit bigger so it's closer to one so this is the spread of the Gaussian"}, {"start": 2274.88, "end": 2280.88, "text": " of zero mean and unit standard deviation now basically when you take these and you multiply by say"}, {"start": 2280.88, "end": 2287.44, "text": " 0.2 that basically scales down the Gaussian and that makes its standard deviation 0.2 so basically"}, {"start": 2287.44, "end": 2292.8, "text": " the number that you multiply by here ends up being the standard deviation of this Gaussian so here"}, {"start": 2292.8, "end": 2300.4, "text": " this is a standard deviation 0.2 Gaussian here when we sample our w1 but we want to set the standard"}, {"start": 2300.4, "end": 2307.76, "text": " deviation to gain over square root of fan mode which is fan in so in other words we want to multiply"}, {"start": 2307.76, "end": 2317.44, "text": " by gain which for 10 h is 5 over 3 5 over 3 is the gain and then times"}, {"start": 2317.44, "end": 2324.4, "text": " um"}, {"start": 2324.4, "end": 2334.08, "text": " I guess I divide square root of the fan in and in this example here the fan in was 10 and I"}, {"start": 2334.08, "end": 2339.92, "text": " just noticed that actually here the fan in for w1 is actually an embed times block size which as"}, {"start": 2339.92, "end": 2344.32, "text": " you will recall is actually 30 and that's because each character is 10 dimensional but then we have"}, {"start": 2344.32, "end": 2348.6400000000003, "text": " three of them and we can cutinate them so actually the fan in here was 30 and I should have used"}, {"start": 2348.6400000000003, "end": 2355.52, "text": " 30 here probably but basically we want 30 square root so this is the number this is what our standard"}, {"start": 2355.52, "end": 2361.36, "text": " deviation we want to be and this number turns out to be 0.3 whereas here just by fiddling with it"}, {"start": 2361.36, "end": 2366.8, "text": " and looking at the distribution and making sure it looks okay we came up with 0.2 and so instead what"}, {"start": 2366.8, "end": 2375.36, "text": " we want to do here is we want to make the standard deviation be um 5 over 3 which is our gain divide"}, {"start": 2377.28, "end": 2384.8, "text": " this amount times 0.2 square root and these brackets here are not that necessary but I'll just put"}, {"start": 2384.8, "end": 2390.6400000000003, "text": " them here for clarity this is basically what we want this is the kiming in it in our case for a 10"}, {"start": 2390.64, "end": 2398.24, "text": " h nonlinearity and this is how we would initialize the neural net and so we're multiplying by 0.3 instead"}, {"start": 2398.24, "end": 2406.64, "text": " of multiplying by 0.2 and so we can we can initialize this way and then we can train the neural net"}, {"start": 2406.64, "end": 2412.16, "text": " and see what we got okay so I trained the neural net and we end up in roughly the same spot"}, {"start": 2412.16, "end": 2417.7599999999998, "text": " so looking at the values should loss we now get 2.10 and previously we also had 2.10 there's"}, {"start": 2417.76, "end": 2422.0800000000004, "text": " a little bit of a difference but that's just the randomness the process I suspect but the big deal"}, {"start": 2422.0800000000004, "end": 2429.28, "text": " of course is we get to the same spot but we did not have to introduce any um magic numbers that we"}, {"start": 2429.28, "end": 2434.0, "text": " got from just looking at histograms and guessing checking we have something that is semi-principled"}, {"start": 2434.0, "end": 2440.0800000000004, "text": " and will scale us to much bigger networks and uh something that we can sort of use as a guide"}, {"start": 2440.0800000000004, "end": 2444.48, "text": " so I mentioned that the precise setting of these initializations is not as important today"}, {"start": 2444.48, "end": 2448.4, "text": " due to some modern innovations and I think now is a pretty good time to introduce one of those modern"}, {"start": 2448.4, "end": 2454.8, "text": " innovations and that is best normalization so best normalization came out in uh 2015 from a team"}, {"start": 2454.8, "end": 2460.56, "text": " at Google and it was an extremely impactful paper because it made it possible to train very deep"}, {"start": 2460.56, "end": 2466.48, "text": " neural nets quite reliably and uh it basically just worked so here's what best normalization does"}, {"start": 2466.48, "end": 2474.08, "text": " and what's implemented um basically we have these uh hidden states h pre-act right and we were"}, {"start": 2474.08, "end": 2480.64, "text": " talking about how we don't want these uh these um pre-activation states to be way too small because"}, {"start": 2480.64, "end": 2485.6, "text": " that then the 10h is not um doing anything but we don't want them to be too large because then the"}, {"start": 2485.6, "end": 2492.3199999999997, "text": " 10h is saturated in fact we want them to be roughly roughly Gaussian so zero mean and a units or"}, {"start": 2492.3199999999997, "end": 2499.44, "text": " one standard deviation at least at initialization so the insight from the best normalization paper is"}, {"start": 2499.44, "end": 2504.7200000000003, "text": " okay you have these hidden states and you'd like them to be roughly Gaussian then why not take the"}, {"start": 2504.7200000000003, "end": 2510.56, "text": " hidden states and uh just normalize them to be Gaussian and it sounds kind of crazy but you can"}, {"start": 2510.56, "end": 2517.44, "text": " just do that because uh standardizing hidden states so that their unit Gaussian is a perfectly"}, {"start": 2517.44, "end": 2522.16, "text": " differentiable operation as we'll soon see and so that was kind of like the big insight in this paper"}, {"start": 2522.16, "end": 2526.48, "text": " and when I first read it my mind was blown because you can just normalize these hidden states and"}, {"start": 2526.48, "end": 2532.2400000000002, "text": " if you'd like unit Gaussian states in your network uh at least initialization you can just normalize"}, {"start": 2532.2400000000002, "end": 2538.2400000000002, "text": " them to be in Gaussian so uh let's see how that works so we're going to scroll to our pre-activations"}, {"start": 2538.2400000000002, "end": 2543.6, "text": " here just before they enter into the 10h now the idea again is remember we're trying to make these"}, {"start": 2543.6, "end": 2549.2, "text": " roughly Gaussian and that's because if these are way too small numbers then the 10h here is kind"}, {"start": 2549.2, "end": 2555.6, "text": " of an active but if these are very large numbers then the 10h is way to saturated and graded in the"}, {"start": 2555.6, "end": 2561.8399999999997, "text": " flow so we'd like this to be roughly Gaussian so the insight in best romanization again is that we"}, {"start": 2561.8399999999997, "end": 2568.3199999999997, "text": " can just standardize these activations so they are exactly Gaussian so here h preact"}, {"start": 2569.8399999999997, "end": 2576.72, "text": " has a shape of 32 by 200 32 examples by 200 neurons in the hidden layer so basically what we can"}, {"start": 2576.72, "end": 2583.04, "text": " do is we can take h preact and we can just calculate the mean um and the mean we want to calculate"}, {"start": 2583.04, "end": 2590.64, "text": " across the zero dimension and we want to also keep them as true so that we can easily broadcast this"}, {"start": 2591.6, "end": 2599.2799999999997, "text": " so the shape of this is 1 by 200 in other words we are doing the mean over all the uh elements in the"}, {"start": 2599.2799999999997, "end": 2604.96, "text": " batch and similarly we can calculate the standard deviation of these activations"}, {"start": 2604.96, "end": 2614.0, "text": " and then we'll also be 1 by 200 now in this paper they have the uh sort of prescription here"}, {"start": 2614.64, "end": 2622.7200000000003, "text": " and see here we are calculating the mean which is just taking the average value of any neurons"}, {"start": 2622.7200000000003, "end": 2628.32, "text": " activation and then the standard deviation is basically kind of like um this the measure of"}, {"start": 2628.32, "end": 2634.4, "text": " the spread that we've been using which is the distance of every one of these values away from the"}, {"start": 2634.4, "end": 2642.4, "text": " mean and that squared and averaged that's the that's the variance and then if you want to take"}, {"start": 2642.4, "end": 2646.56, "text": " the standard deviation you would the square root the variance to get the standard deviation"}, {"start": 2647.76, "end": 2652.56, "text": " so these are the two that we're calculating and now we're going to normalize or standardize"}, {"start": 2652.56, "end": 2659.2000000000003, "text": " these x's by subtracting the mean and um dividing by the standard deviation so basically we're taking"}, {"start": 2659.2, "end": 2672.3999999999996, "text": " edge preact and we subtract the mean and then we divide by the standard deviation"}, {"start": 2674.3199999999997, "end": 2682.08, "text": " this is exactly what these two STD and mean are calculating oops sorry this is the mean and this"}, {"start": 2682.08, "end": 2686.72, "text": " is the variance you see how the sigma is the standard deviation usually so this is sigma square"}, {"start": 2686.72, "end": 2692.16, "text": " which is the variance is the square of the standard deviation so this is how you standardize"}, {"start": 2692.16, "end": 2697.3599999999997, "text": " these values and what this will do is that every single neuron now and its firing rate will be"}, {"start": 2697.3599999999997, "end": 2702.3999999999996, "text": " exactly unit Gaussian on these 32 examples at least of this batch that's why it's called batch"}, {"start": 2702.3999999999996, "end": 2709.7599999999998, "text": " normalization we are normalizing these batches and then we could in principle train this notice"}, {"start": 2709.7599999999998, "end": 2714.24, "text": " that calculating the mean and standard deviation these are just mathematical formulas they're perfectly"}, {"start": 2714.24, "end": 2719.52, "text": " differentiable all this is perfectly differentiable and we can just train this the problem is you"}, {"start": 2719.52, "end": 2726.72, "text": " actually won't achieve a very good result with this and the reason for that is we want these to be"}, {"start": 2726.72, "end": 2733.04, "text": " roughly Gaussian but only at initialization but we don't want these to be forced to be Gaussian"}, {"start": 2733.04, "end": 2738.64, "text": " always we would like to allow the neural nets to move this around to potentially make it more"}, {"start": 2738.64, "end": 2744.0, "text": " diffuse to make it more sharp to make some 10 H neurons maybe mean more trigger more trigger happy"}, {"start": 2744.0, "end": 2748.24, "text": " or less trigger happy so we'd like this distribution to move around and we'd like the back"}, {"start": 2748.24, "end": 2754.4, "text": " propagation to tell us how the distribution should move around and so in addition to this idea"}, {"start": 2754.4, "end": 2760.8, "text": " of standardizing the activations at any point in the network we have to also introduce this"}, {"start": 2760.8, "end": 2766.64, "text": " additional component in the paper here describe the scale and shift and so basically what we're"}, {"start": 2766.64, "end": 2772.16, "text": " doing is we're taking these normalized inputs and we are additionally scaling them by some gain"}, {"start": 2772.16, "end": 2778.8799999999997, "text": " and offsetting them by some bias to get our final output from this layer and so what that amounts"}, {"start": 2778.8799999999997, "end": 2786.0, "text": " to is the following we are going to allow a bash normalization gain to be initialized at just"}, {"start": 2786.0, "end": 2794.48, "text": " a once and the once will be in the shape of 1 by n hidden and then we also will have a bn bias"}, {"start": 2794.48, "end": 2802.88, "text": " which will be torched at 0's and it will also be of the shape n by 1 by n hidden and then here"}, {"start": 2803.6, "end": 2812.48, "text": " the bn gain will multiply this and the bn bias will offset it here so because this is initialized"}, {"start": 2812.48, "end": 2820.88, "text": " to 1 and this to 0 at initialization each neurons firing values in this batch will be exactly"}, {"start": 2820.88, "end": 2827.12, "text": " unit Gaussian and will have nice numbers no matter what the distribution of the h preact is coming in"}, {"start": 2827.12, "end": 2831.36, "text": " coming out it will be unit Gaussian for each neuron and that's roughly what we want at least at"}, {"start": 2831.36, "end": 2838.48, "text": " initialization and then during optimization we'll be able to back propagate to bn gain and bn bias"}, {"start": 2838.48, "end": 2844.7200000000003, "text": " and change them so the network is given the full ability to do with this whatever it wants internally"}, {"start": 2844.72, "end": 2852.3199999999997, "text": " here we just have to make sure that we include these in the parameters of the neural nut because"}, {"start": 2852.3199999999997, "end": 2859.2799999999997, "text": " they will be trained with back propagation so let's initialize this and then we should be able to train"}, {"start": 2865.6, "end": 2871.2799999999997, "text": " and then we're going to also copy this line which is the bash normalization layer"}, {"start": 2871.28, "end": 2875.76, "text": " here on a single line of code and we're going to swing down here and we're also going to"}, {"start": 2876.4, "end": 2884.2400000000002, "text": " do the exact same thing at test time here so similar to train time we're going to normalize"}, {"start": 2884.96, "end": 2889.0400000000004, "text": " and then scale and that's going to give us our train and validation loss"}, {"start": 2890.6400000000003, "end": 2893.6000000000004, "text": " and we'll see in a second that we're actually going to change this a little bit but for now I'm"}, {"start": 2893.6000000000004, "end": 2898.5600000000004, "text": " going to keep it this way so I'm just going to wait for this to converge okay so I allowed the"}, {"start": 2898.56, "end": 2903.92, "text": " neural nut to converge here and when we scroll down we see that our validation loss here is 2.10"}, {"start": 2903.92, "end": 2908.56, "text": " roughly which I wrote down here and we see that this is actually kind of comparable to some of the"}, {"start": 2908.56, "end": 2914.24, "text": " results that we've achieved previously now I'm not actually expecting an improvement in this case"}, {"start": 2914.7999999999997, "end": 2918.72, "text": " and that's because we are dealing with a very simple neural nut that has just a single hidden layer"}, {"start": 2919.44, "end": 2923.92, "text": " so in fact in this very simple case of just one hidden layer we were able to actually"}, {"start": 2923.92, "end": 2929.28, "text": " calculate what the scale of W should be to make these pre-activations already have a roughly"}, {"start": 2929.28, "end": 2934.4, "text": " Gaussian shape so the best normalization is not doing much here but you might imagine that once"}, {"start": 2934.4, "end": 2939.76, "text": " you have a much deeper neural nut that has lots of different types of operations and there's also"}, {"start": 2939.76, "end": 2944.7200000000003, "text": " for example residual connections which we'll cover and so on it will become basically very very"}, {"start": 2944.7200000000003, "end": 2950.4, "text": " difficult to tune the scales of your wake matrices such that all the activations throughout the"}, {"start": 2950.4, "end": 2956.48, "text": " neural nut are roughly Gaussian and so that's going to become very quickly intractable but"}, {"start": 2956.48, "end": 2960.88, "text": " compared to that it's going to be much much easier to sprinkle batch normalization layers throughout"}, {"start": 2960.88, "end": 2966.96, "text": " the neural nut so in particular it's common to to look at every single linear layer like this one"}, {"start": 2966.96, "end": 2972.2400000000002, "text": " this is a linear layer multiplied by weight matrix and adding the bias or for example convolutions"}, {"start": 2972.2400000000002, "end": 2977.84, "text": " which we'll cover later and also perform basically a multiplication with weight matrix but in a"}, {"start": 2977.84, "end": 2982.56, "text": " more spatially structured format it's customer it's customer to take these linear layer or"}, {"start": 2982.56, "end": 2989.2000000000003, "text": " convolutional layer and append a best normalization layer right after it to control the scale of"}, {"start": 2989.2000000000003, "end": 2993.76, "text": " these activations at every point in the neural nut so we'd be adding these best normal layers throughout"}, {"start": 2993.76, "end": 2998.0, "text": " the neural nut and then this controls the scale of these activations throughout the neural nut"}, {"start": 2998.56, "end": 3003.84, "text": " it doesn't require us to do a perfect mathematics and care about the activation distributions"}, {"start": 3003.84, "end": 3007.76, "text": " for all these different types of neural nut or Lego building blocks that you might want to"}, {"start": 3007.76, "end": 3013.2000000000003, "text": " introduce into your neural nut and it significantly stabilizes the train and that's why these"}, {"start": 3013.2000000000003, "end": 3017.6800000000003, "text": " layers are quite popular now the stability offered by batch normalization actually comes at a"}, {"start": 3017.6800000000003, "end": 3023.2000000000003, "text": " terrible cost and that cost is that if you think about what's happening here something's something"}, {"start": 3023.2000000000003, "end": 3029.1200000000003, "text": " terribly strange and unnatural is happening it used to be that we have a single example feeding"}, {"start": 3029.12, "end": 3035.12, "text": " into a neural nut and then we calculate this activations and its logits and this is a"}, {"start": 3035.12, "end": 3041.12, "text": " deterministic sort of process so you arrive at some logits for this example and then because of"}, {"start": 3041.12, "end": 3046.0, "text": " efficiency of training we suddenly started to use batches of examples but those batches of examples"}, {"start": 3046.0, "end": 3051.52, "text": " were processed independently and it was just an efficiency thing but now suddenly in batch normalization"}, {"start": 3051.52, "end": 3056.56, "text": " because of the normalization through the batch we are coupling these examples mathematically"}, {"start": 3056.56, "end": 3062.16, "text": " and in the forward pass and backward pass of the neural nut so now the hidden state activations H"}, {"start": 3062.16, "end": 3067.6, "text": " pre-oct and your logits for any one input example are not just a function of that example and its"}, {"start": 3067.6, "end": 3073.04, "text": " input but they're also a function of all the other examples that happen to come for a ride in that"}, {"start": 3073.04, "end": 3078.16, "text": " batch and these examples are sampled randomly and so what's happening is for example when you look"}, {"start": 3078.16, "end": 3083.84, "text": " at H pre-oct that's going to feed into H the hidden state activations for for example for for any one"}, {"start": 3083.84, "end": 3088.96, "text": " of these input examples is going to actually change slightly depending on what other examples"}, {"start": 3088.96, "end": 3095.04, "text": " there are in the batch and depending on what other examples happen to come for a ride H is going"}, {"start": 3095.04, "end": 3099.6000000000004, "text": " to change suddenly and it's going to look jitter if you imagine sampling different examples"}, {"start": 3099.6000000000004, "end": 3103.1200000000003, "text": " because the statistics of the mean and the standard deviation are going to be impacted"}, {"start": 3104.0, "end": 3109.52, "text": " and so you'll get a jitter for H and you'll get a jitter for logits and you think that this would"}, {"start": 3109.52, "end": 3116.24, "text": " be a bug or something undesirable but in a very strange way this actually turns out to be good"}, {"start": 3116.24, "end": 3121.7599999999998, "text": " in neural network training and as a side effect and the reason for that is that you can think of this"}, {"start": 3121.7599999999998, "end": 3126.48, "text": " as kind of like a regularizer because what's happening is you have your input and you get your H"}, {"start": 3126.48, "end": 3131.04, "text": " and then depending on the other examples this is jittering a bit and so what that does is that"}, {"start": 3131.04, "end": 3135.52, "text": " it's effectively padding out any one of these input examples and it's introducing a little bit of"}, {"start": 3135.52, "end": 3141.36, "text": " entropy and because of the padding out it's actually kind of like a form of a data augmentation"}, {"start": 3141.36, "end": 3146.0, "text": " which will cover in the future and it's not kind of like augmenting the input a little bit and"}, {"start": 3146.0, "end": 3151.04, "text": " jittering it and that makes it harder for the neural nets to overfit these concrete specific"}, {"start": 3151.04, "end": 3156.8, "text": " examples so by introducing all this noise it actually like pads out the examples and it regularizes"}, {"start": 3156.8, "end": 3162.48, "text": " the neural net and that's one of the reasons why deceiving me as a second order effect this is"}, {"start": 3162.48, "end": 3167.76, "text": " actually a regularizer and that has made it harder for us to remove the use of bachelor normalization"}, {"start": 3168.72, "end": 3173.36, "text": " because basically no one likes this property that the examples in the batch are coupled"}, {"start": 3173.36, "end": 3179.2, "text": " mathematically and in the forward pass and at least all kinds of like strange results will go into"}, {"start": 3179.2, "end": 3185.84, "text": " some of that in a second as well and at least do a lot of bugs and so on and so no one likes this"}, {"start": 3185.84, "end": 3191.84, "text": " property and so people have tried to deprecate the use of bachelor normalization and move to other"}, {"start": 3191.84, "end": 3196.88, "text": " normalization techniques that do not couple the examples of a batch examples are linear normalization"}, {"start": 3196.88, "end": 3201.84, "text": " instance normalization group normalization and so on and we'll come rest we'll come rest on these"}, {"start": 3201.84, "end": 3208.0, "text": " later but basically a long story short bachelor normalization was the first kind of normalization"}, {"start": 3208.0, "end": 3212.88, "text": " later to be introduced it worked extremely well it happens to have this regularizing effect"}, {"start": 3213.44, "end": 3219.04, "text": " it's stabilized training and people have been trying to remove it and move to some of the other"}, {"start": 3219.04, "end": 3224.8, "text": " normalization techniques but it's been hard because it it just works quite well and some of the"}, {"start": 3224.8, "end": 3229.2, "text": " reason that it works quite well is again because of this regularizing effect and because of the"}, {"start": 3229.2, "end": 3234.72, "text": " because it is quite effective at controlling the activations and their distributions so that's"}, {"start": 3234.72, "end": 3240.56, "text": " kind of like the brief story of bachelor normalization and I like to show you one of the other weird"}, {"start": 3240.56, "end": 3246.0, "text": " sort of outcomes of this coupling so here's one of the strange outcomes that I only lost over"}, {"start": 3246.0, "end": 3252.08, "text": " previously when I was evaluating the loss on the validation set basically once we've trained a"}, {"start": 3252.08, "end": 3257.28, "text": " neural net we'd like to deploy it in some kind of a setting and we'd like to be able to feed in a"}, {"start": 3257.28, "end": 3262.64, "text": " single individual example and get a prediction out from our neural net but how do we do that when"}, {"start": 3262.64, "end": 3267.12, "text": " our neural net now in a forward pass estimates the statistics of the mean understand deviation"}, {"start": 3267.12, "end": 3272.08, "text": " of a batch the neural net expects batches as an input now so how do we feed in a single example"}, {"start": 3272.08, "end": 3278.0, "text": " and get sensible results out and so the proposal in the bachelor normalization paper is the following"}, {"start": 3278.7999999999997, "end": 3284.16, "text": " what we would like to do here is we would like to basically have a step after training"}, {"start": 3284.7999999999997, "end": 3290.88, "text": " that calculates and sets the bachelor mean and standard deviation a single time over the"}, {"start": 3290.88, "end": 3296.48, "text": " training set and so I wrote this code here in interest of time and we're going to call what's"}, {"start": 3296.48, "end": 3303.04, "text": " called calibrate the bachelor statistics and basically what we do is torshot no grad telling by"}, {"start": 3303.04, "end": 3307.92, "text": " torshot that none of this we will call the dot backward on and it's going to be a bit more efficient"}, {"start": 3308.8, "end": 3313.6, "text": " we're going to take the training set get the preactivations for every single training example"}, {"start": 3313.6, "end": 3317.36, "text": " and then one single time estimate the mean and standard deviation over the entire training set"}, {"start": 3318.16, "end": 3322.2400000000002, "text": " and then we're going to get b and mean and b and standard deviation and now these are fixed"}, {"start": 3322.24, "end": 3328.4799999999996, "text": " numbers estimated over the entire training set and here instead of estimating it dynamically"}, {"start": 3329.8399999999997, "end": 3336.64, "text": " we are going to instead here use b and mean and here we're just going to use b and standard deviation"}, {"start": 3338.08, "end": 3343.52, "text": " and so at test time we are going to fix these clamp them and use them during inference and now"}, {"start": 3345.52, "end": 3351.2799999999997, "text": " you see that we get basically identical result but the benefit that we've gained is that we can"}, {"start": 3351.28, "end": 3355.52, "text": " now also forward a single example because the mean and standard deviation are now fixed"}, {"start": 3355.52, "end": 3361.1200000000003, "text": " all sorts of tensors that said nobody actually wants to estimate this mean and standard deviation"}, {"start": 3361.1200000000003, "end": 3366.8, "text": " as a second stage after neural network training because everyone is lazy and so this"}, {"start": 3366.8, "end": 3371.28, "text": " specialization paper actually introduced one more idea which is that we can we can estimate"}, {"start": 3371.28, "end": 3376.0, "text": " the mean and standard deviation in a running manner running manner during training of the"}, {"start": 3376.0, "end": 3381.04, "text": " neural network and then we can simply just have a single stage of training and on the side of that"}, {"start": 3381.04, "end": 3385.52, "text": " training we are estimating the running mean as a deviation so let's see what that would look like"}, {"start": 3386.56, "end": 3391.12, "text": " let me basically take the mean here that we are estimating on the batch and let me call this"}, {"start": 3391.12, "end": 3406.4, "text": " b and mean on the i-th iteration and then here this is b and sd d b and sd d at i okay"}, {"start": 3407.04, "end": 3415.2799999999997, "text": " and the mean comes here and the sd d comes here so so far I've done nothing I've just moved around"}, {"start": 3415.2799999999997, "end": 3419.68, "text": " and I created these extra variables for the mean and standard deviation and I've put them here"}, {"start": 3419.68, "end": 3424.24, "text": " so so far nothing has changed but what we're going to do now is we're going to keep a running mean"}, {"start": 3424.24, "end": 3429.9199999999996, "text": " of both of these values during training so let me swing up here and let me create a b and mean"}, {"start": 3429.9199999999996, "end": 3437.8399999999997, "text": " underscore running and I'm going to initialize it at zeros and then b and sd d running"}, {"start": 3438.64, "end": 3447.2, "text": " which I'll initialize at once because in the beginning because of the way we initialized w1"}, {"start": 3447.2, "end": 3453.4399999999996, "text": " and b1 hpx will be roughly unit Gaussian so the mean will be roughly zero and extend deviation"}, {"start": 3453.4399999999996, "end": 3458.64, "text": " roughly one so I'm going to initialize these that way but then here I'm going to update these"}, {"start": 3459.4399999999996, "end": 3466.3199999999997, "text": " and in PyTorch these mean and standard deviation that are running they're not actually part of"}, {"start": 3466.3199999999997, "end": 3470.16, "text": " the gradient based optimization we're never going to derive gradients with respect to them"}, {"start": 3470.16, "end": 3475.3599999999997, "text": " they're they're updated on the side of training and so what we're going to do here is we're going to"}, {"start": 3475.36, "end": 3482.2400000000002, "text": " say with torsched up no grad telling PyTorch that the update here is not supposed to be building"}, {"start": 3482.2400000000002, "end": 3489.2000000000003, "text": " out a graph because there will be no dot backward but this running mean is basically going to be 0.99"}, {"start": 3490.2400000000002, "end": 3502.56, "text": " nine times the current value plus 0.001 times the best value this new mean and in the same way bnstd"}, {"start": 3502.56, "end": 3511.68, "text": " running will be mostly what it used to be but it will receive a small update in the direction of"}, {"start": 3511.68, "end": 3518.16, "text": " what the current standard deviation is and as you're seeing here this update is outside and on"}, {"start": 3518.16, "end": 3523.6, "text": " the side of the gradient based optimization and it's simply being updated not using gradient"}, {"start": 3523.6, "end": 3534.56, "text": " sent is just being updated using a jenky like smooth sort of running mean manner and so while the"}, {"start": 3534.56, "end": 3538.96, "text": " network is training and these preactivations are sort of changing and shifting around during"}, {"start": 3538.96, "end": 3543.6, "text": " during back propagation we are keeping track of the typical mean and standard deviation"}, {"start": 3543.6, "end": 3546.64, "text": " and rest of the mean them once and when I run this"}, {"start": 3546.64, "end": 3553.68, "text": " now I'm keeping track of this in a running manner and what we're hoping for of course is that the"}, {"start": 3553.68, "end": 3560.0, "text": " mean bn mean underscore running and bn mean underscore std are going to be very similar to the ones"}, {"start": 3560.0, "end": 3565.2799999999997, "text": " that we calculated here before and that way we don't need a second stage because we've sort of"}, {"start": 3565.2799999999997, "end": 3569.3599999999997, "text": " combined the two stages and we've put them on the side of each other if you want to look at it"}, {"start": 3569.3599999999997, "end": 3575.44, "text": " that way and this is how this is also implemented in the bastion realization layer in PyTorch so"}, {"start": 3575.44, "end": 3581.6, "text": " during training the exact same thing will happen and then later when you're using inference it will"}, {"start": 3581.6, "end": 3586.7200000000003, "text": " use the estimated running mean of both the mean and standard deviation of those hidden states"}, {"start": 3587.84, "end": 3592.32, "text": " so let's wait for the optimization to converge and hopefully the running mean and standard deviation"}, {"start": 3592.32, "end": 3597.36, "text": " are roughly equal to these two and then we can simply use it here and we don't need this stage"}, {"start": 3597.36, "end": 3602.7200000000003, "text": " of explicit calibration at the end okay so the optimization finished I'll rerun the explicit"}, {"start": 3602.72, "end": 3609.68, "text": " estimation and then the bn mean from the explicit estimation is here and bn mean from the running"}, {"start": 3609.68, "end": 3617.2799999999997, "text": " estimation during the during the optimization you can see it's very very similar it's not identical"}, {"start": 3617.2799999999997, "end": 3627.2799999999997, "text": " but it's pretty close and the same way bnstd is this and bnstd running is this as you can see that once"}, {"start": 3627.28, "end": 3633.28, "text": " again they are fairly similar values not identical but pretty close and so then here instead of"}, {"start": 3633.28, "end": 3641.2000000000003, "text": " being mean we can use the bn mean running instead of bnstd we can use bnstd running and hopefully"}, {"start": 3641.2000000000003, "end": 3647.84, "text": " the validation loss will not be impacted too much okay so it's basically identical and this way"}, {"start": 3647.84, "end": 3653.1200000000003, "text": " we've eliminated the need for this explicit stage of calibration because we are doing it in line"}, {"start": 3653.12, "end": 3657.2799999999997, "text": " over here okay so we're almost done with batch normalization there are only two more notes"}, {"start": 3657.2799999999997, "end": 3661.44, "text": " that I'd like to make number one I've skipped a discussion over what is this plus epsilon doing"}, {"start": 3661.44, "end": 3667.2, "text": " here this epsilon is usually like some small fixed number for example one in negative 5 by default"}, {"start": 3667.2, "end": 3672.3199999999997, "text": " and what it's doing is that it's basically preventing a division by zero in the case that the"}, {"start": 3672.3199999999997, "end": 3678.96, "text": " variance over your batch is exactly zero in that case here we normally have a division by zero"}, {"start": 3678.96, "end": 3683.52, "text": " but because of the plus epsilon this is going to become a small number in the denominator instead"}, {"start": 3683.52, "end": 3688.48, "text": " and things will be more well behaved so feel free to also add a plus epsilon here of a very small"}, {"start": 3688.48, "end": 3692.56, "text": " number it doesn't actually substantially change the result I'm going to skip it in our case just"}, {"start": 3692.56, "end": 3697.2, "text": " because this is unlikely to happen in our very simple example here and the second thing I want you"}, {"start": 3697.2, "end": 3702.48, "text": " to notice is that we're being wasteful here and it's very subtle but right here where we are"}, {"start": 3702.48, "end": 3708.88, "text": " adding the bias into each preact these biases now are actually useless because we're adding them"}, {"start": 3708.88, "end": 3715.12, "text": " to the each preact but then we are calculating the mean for every one of these neurons and subtracting"}, {"start": 3715.12, "end": 3722.0, "text": " it so whatever bias you add here is going to get subtracted right here and so these biases are"}, {"start": 3722.0, "end": 3726.1600000000003, "text": " not doing anything in fact but they're being subtracted out and they don't impact the rest of"}, {"start": 3726.1600000000003, "end": 3731.36, "text": " the calculation so if you look at b1.grad it's actually going to be zero because it being subtracted"}, {"start": 3731.36, "end": 3736.08, "text": " out and doesn't actually have any effect and so whenever you're using bachelor normalization layers"}, {"start": 3736.08, "end": 3740.08, "text": " then if you have any weight layers before like a linear or a conv or something like that"}, {"start": 3740.64, "end": 3746.24, "text": " you're better off coming here and just like not using bias so you don't want to use bias"}, {"start": 3746.24, "end": 3751.84, "text": " and then here you don't want to add it because that's that's spurious instead we have this"}, {"start": 3751.84, "end": 3756.7999999999997, "text": " bachelor normalization bias here and that bachelor normalization bias is now in charge of the"}, {"start": 3756.7999999999997, "end": 3763.92, "text": " biasing of this distribution instead of this b1 that we had here originally and so basically"}, {"start": 3763.92, "end": 3768.48, "text": " the bachelor normalization layer has its own bias and there's no need to have a bias in the layer"}, {"start": 3768.48, "end": 3773.2000000000003, "text": " before it because that bias is going to be subtracted out anyway so that's the other small detail"}, {"start": 3773.2000000000003, "end": 3778.48, "text": " to be careful with sometimes it's not going to do anything catastrophic this b1 will just be useless"}, {"start": 3778.48, "end": 3783.04, "text": " it will never get any gradient it will not learn it will stay constant and it's just wasteful"}, {"start": 3783.04, "end": 3788.88, "text": " but it doesn't actually really impact anything otherwise okay so I rearranged the code a little bit"}, {"start": 3788.88, "end": 3792.8, "text": " with comments and I just wanted to give a very quick summary of the bachelor normalization layer"}, {"start": 3792.8, "end": 3798.88, "text": " we are using bachelor normalization to control the statistics of activations in the neural net"}, {"start": 3799.6000000000004, "end": 3804.1600000000003, "text": " it is common to sprinkle bachelor normalization layer across the neural net and usually we will"}, {"start": 3804.1600000000003, "end": 3810.32, "text": " place it after layers that have multiplications like for example a linear layer or a convolutional"}, {"start": 3810.32, "end": 3817.6800000000003, "text": " layer which we may cover in the future now the bachelor normalization internally has parameters"}, {"start": 3817.68, "end": 3823.6, "text": " for the gain and the bias and these are trained using back propagation it also has two buffers"}, {"start": 3824.3999999999996, "end": 3829.2, "text": " the buffers are the mean and the standard deviation the running mean and the running mean of"}, {"start": 3829.2, "end": 3833.9199999999996, "text": " the standard deviation and these are not trained using back propagation these are trained using"}, {"start": 3833.9199999999996, "end": 3842.8799999999997, "text": " this janky update of kind of like a running mean update so these are sort of the parameters"}, {"start": 3842.88, "end": 3848.1600000000003, "text": " and the buffers of bachelor layer and then really what is doing is it's calculating the mean and"}, {"start": 3848.1600000000003, "end": 3853.6800000000003, "text": " standard deviation of the activations that are feeding into the bachelor layer over that batch"}, {"start": 3854.96, "end": 3860.8, "text": " then it's centering that batch to be unit Gaussian and then it's offsetting and scaling it by the"}, {"start": 3860.8, "end": 3867.36, "text": " learned bias and gain and then on top of that it's keeping track of the mean and standard deviation"}, {"start": 3867.36, "end": 3873.52, "text": " of the inputs and it's maintaining this running mean and standard deviation and this will later"}, {"start": 3873.52, "end": 3878.0, "text": " be used at inference so that we don't have to re-estimate the mean and standard deviation all the time"}, {"start": 3878.96, "end": 3883.28, "text": " and in addition that allows us to basically forward individual examples at test time"}, {"start": 3884.2400000000002, "end": 3887.1200000000003, "text": " so that's the bachelor normalization layer it's a fairly complicated layer"}, {"start": 3888.4, "end": 3892.56, "text": " but this is what it's doing internally now I wanted to show you a little bit of a real example"}, {"start": 3892.56, "end": 3899.44, "text": " so you can search ResNet which is a residual neural network and these are contact of neural"}, {"start": 3899.44, "end": 3905.68, "text": " arcs used for image classification and of course we haven't common dress nets in detail so I'm"}, {"start": 3905.68, "end": 3910.72, "text": " not going to explain all the pieces of it but for now just note that the image feeds into a"}, {"start": 3910.72, "end": 3915.68, "text": " ResNet on the top here and there's many many layers with repeating structure all the way to"}, {"start": 3915.68, "end": 3920.96, "text": " predictions of what's inside that image this repeating structure is made up of these blocks and"}, {"start": 3920.96, "end": 3927.12, "text": " these blocks are just sequentially stacked up in this deep neural network now the code for this"}, {"start": 3928.0, "end": 3935.28, "text": " the block basically that's used and repeated sequentially in series is called this bottleneck block"}, {"start": 3936.16, "end": 3940.64, "text": " and there's a lot here this is all PyTorch and of course we haven't covered all of it but I want"}, {"start": 3940.64, "end": 3946.16, "text": " to point out some small pieces of it here in the init is where we initialize the neural net so this"}, {"start": 3946.16, "end": 3951.3599999999997, "text": " code of block here is basically the kind of stuff we're doing here we're initializing all the layers and"}, {"start": 3951.3599999999997, "end": 3956.16, "text": " in the forward we are specifying how the neural net acts once you actually have the input so this"}, {"start": 3956.16, "end": 3964.48, "text": " code here is a long lines of what we're doing here and now these blocks are replicated and stacked"}, {"start": 3964.48, "end": 3971.52, "text": " up serially and that's what a residual network would be and so notice what's happening here come one"}, {"start": 3971.52, "end": 3978.64, "text": " these are convolutional layers and these convolutional layers basically they're the same thing as a linear"}, {"start": 3978.64, "end": 3985.2, "text": " layer except convolutional layers don't apply convolutional layers are used for images and so they"}, {"start": 3985.2, "end": 3990.96, "text": " have spatial structure and basically this linear multiplication and bias offset are done on patches"}, {"start": 3991.84, "end": 3997.92, "text": " instead of a map instead of the full input so because these images have structure spatial structure"}, {"start": 3997.92, "end": 4004.08, "text": " convolutional is just basically do wx plus b but they do it on overlapping patches of the input but"}, {"start": 4004.08, "end": 4009.92, "text": " otherwise is wx plus b then we have the normal layer which by default here is initialize to be a"}, {"start": 4009.92, "end": 4016.88, "text": " bashed norm in 2d so 2-dimensional bashed normalization layer and then we have a nonlinearity like relu so"}, {"start": 4016.88, "end": 4024.64, "text": " instead of here they use relu we are using 10h in this case but both both are just nonlinearities and"}, {"start": 4024.64, "end": 4029.92, "text": " you can just use them relatively interchangeably for very deep networks relu typically empirically"}, {"start": 4029.92, "end": 4036.0, "text": " work a bit better so see the motif that's being repeated here we have convolution, bashed normalization"}, {"start": 4036.0, "end": 4041.2799999999997, "text": " relu convolution, bashed normalization relu etc and then here this is residual connection that we"}, {"start": 4041.2799999999997, "end": 4047.68, "text": " haven't covered yet but basically that's the exact same pattern we have here we have a weight layer"}, {"start": 4047.68, "end": 4055.52, "text": " like a convolution or like a linear layer, bashed normalization and then 10h which is nonlinearity"}, {"start": 4055.52, "end": 4060.48, "text": " but basically a weight layer, a normalization layer and nonlinearity and that's the motif that"}, {"start": 4060.48, "end": 4064.96, "text": " you would be stacking up when you create these deep neural networks exactly as it's done here"}, {"start": 4065.6, "end": 4069.2799999999997, "text": " and one more thing I'd like you to notice is that here when they are initializing the"}, {"start": 4069.2799999999997, "end": 4076.7999999999997, "text": " comp layers like comp one by one the depth for that is right here and so it's initializing an nn.comp2d"}, {"start": 4076.8, "end": 4080.88, "text": " which is a convolutional layer in PyTorch and there's a bunch of keyword arguments here that I"}, {"start": 4080.88, "end": 4086.1600000000003, "text": " am not going to explain yet but you see how there's bias equals false the bias equals false is exactly"}, {"start": 4086.1600000000003, "end": 4091.52, "text": " for the same reason as bias is not used in our case you see how I raised the use of bias"}, {"start": 4092.1600000000003, "end": 4096.72, "text": " and the use of bias is spurious because after this weight layer there's the bashed normalization"}, {"start": 4096.72, "end": 4101.360000000001, "text": " and the bashed normalization subtracts that bias and that has its own bias so there's no need to"}, {"start": 4101.360000000001, "end": 4106.64, "text": " introduce these spurious parameters it wouldn't hurt performance it's just useless and so because"}, {"start": 4106.64, "end": 4111.92, "text": " they have this motif of calm bashed normalization they don't need a bias here because there's a bias"}, {"start": 4111.92, "end": 4118.08, "text": " inside here so by the way this example here is very easy to find just do a resonant PyTorch"}, {"start": 4119.360000000001, "end": 4124.88, "text": " and it's this example here so this is kind of like the stock implementation of a residual neural"}, {"start": 4124.88, "end": 4129.92, "text": " network in PyTorch and you can find that here but of course I haven't covered many of these parts yet"}, {"start": 4130.64, "end": 4135.52, "text": " and I would also like to briefly descend into the definitions of these PyTorch layers and the"}, {"start": 4135.52, "end": 4140.0, "text": " parameters that they take now instead of a convolutional layer we're going to look at a linear layer"}, {"start": 4141.280000000001, "end": 4144.88, "text": " because that's the one that we're using here this is a linear layer and I haven't covered"}, {"start": 4144.88, "end": 4149.76, "text": " cover the convolutions yet but as I mentioned convolutions are basically linear layers except on patches"}, {"start": 4151.200000000001, "end": 4156.72, "text": " so a linear layer performs a wx plus b except here they're calling the w a transpose"}, {"start": 4158.72, "end": 4163.040000000001, "text": " so the call is wx plus b very much like we did here to initialize this layer you need to know"}, {"start": 4163.04, "end": 4170.96, "text": " the fan in the fan out and that's so that they can initialize this w this is the fan in and the"}, {"start": 4170.96, "end": 4176.88, "text": " fan out so they know how big the weight matrix should be you need to also pass in whether you"}, {"start": 4176.88, "end": 4183.2, "text": " whether or not you want a bias and if you set it to false the no bias will be inside this layer"}, {"start": 4184.32, "end": 4189.2, "text": " and you may want to do that exactly like in our case if your layer is followed by a normalization"}, {"start": 4189.2, "end": 4195.679999999999, "text": " layer such as bach norm so this allows you to basically disable bias in terms of the initialization"}, {"start": 4195.679999999999, "end": 4202.08, "text": " if we swing down here this is reporting the variables used inside this linear layer and our linear"}, {"start": 4202.08, "end": 4207.679999999999, "text": " layer here has two parameters the weight and the bias in the same way they have a weight and a bias"}, {"start": 4208.5599999999995, "end": 4213.36, "text": " and they're talking about how they initialize it by default so by default PyTorch will initialize"}, {"start": 4213.36, "end": 4222.24, "text": " your weights by taking the fan in and then doing one over fan in square root and then instead of a"}, {"start": 4222.24, "end": 4228.4, "text": " normal distribution they are using a uniform distribution so it's very much the same thing but they"}, {"start": 4228.4, "end": 4232.96, "text": " are using a one instead of five over three so there's no gain being calculated here the gain is"}, {"start": 4232.96, "end": 4238.88, "text": " just one but otherwise is exactly one over the square root of fan in exactly as we have here"}, {"start": 4238.88, "end": 4246.32, "text": " so one over the square root of k is the is this scale of the weights but when they are drawing the"}, {"start": 4246.32, "end": 4251.4400000000005, "text": " numbers they're not using a Gaussian by default they're using a uniform distribution by default"}, {"start": 4251.4400000000005, "end": 4256.88, "text": " and so they draw uniformly from negative square root of k to square root of k but it's the"}, {"start": 4256.88, "end": 4263.04, "text": " exact same thing and the same motivation from for with respect to what we've seen in this lecture"}, {"start": 4263.04, "end": 4268.4800000000005, "text": " and the reason they're doing this is if you have a roughly Gaussian input this will ensure that out"}, {"start": 4268.48, "end": 4274.32, "text": " of this layer you will have a roughly Gaussian output and you you basically achieve that by"}, {"start": 4274.32, "end": 4281.36, "text": " scaling the weights by one over the square root of fan in so that's what this is doing and then the"}, {"start": 4281.36, "end": 4285.12, "text": " second thing is the bachelor normalization layer so let's look at what that looks like in PyTorch"}, {"start": 4286.16, "end": 4290.08, "text": " so here we have a one-dimensional bachelor normalization layer exactly as we are using here"}, {"start": 4290.879999999999, "end": 4294.639999999999, "text": " and there are a number of keyword arguments going into it as well so we need to know the number of"}, {"start": 4294.64, "end": 4300.8, "text": " features for us that is 200 and that is needed so that we can initialize these parameters here"}, {"start": 4300.8, "end": 4307.92, "text": " the gain the bias and the buffers for the running mean and serredivation then they need to know the"}, {"start": 4307.92, "end": 4312.8, "text": " value of epsilon here and by default this is one negative five you don't typically change this"}, {"start": 4312.8, "end": 4319.360000000001, "text": " too much then they need to know the momentum and the momentum here as they explain is basically used"}, {"start": 4319.36, "end": 4325.04, "text": " for these running mean and running standard deviation so by default the momentum here is point one"}, {"start": 4325.04, "end": 4332.719999999999, "text": " the momentum we are using here in this example is 0.001 and basically you may want to change this"}, {"start": 4332.719999999999, "end": 4338.88, "text": " sometimes and roughly speaking if you have a very large batch size then typically what you'll see"}, {"start": 4338.88, "end": 4342.96, "text": " is that when you estimate the mean and the same deviation for every single batch size if it's"}, {"start": 4342.96, "end": 4349.04, "text": " large enough you're going to get roughly the same result and so therefore you can use slightly higher"}, {"start": 4349.04, "end": 4356.08, "text": " momentum like point one but for a batch size s small is 32 the mean and understanding deviation here"}, {"start": 4356.08, "end": 4360.48, "text": " might take on slightly different numbers because there's only 32 examples we are using to estimate"}, {"start": 4360.48, "end": 4365.36, "text": " the mean and standard deviation so the value is changing around a lot and if your momentum is"}, {"start": 4365.36, "end": 4372.32, "text": " point one that that might not be good enough for this value to settle and converge to the actual mean"}, {"start": 4372.32, "end": 4377.5199999999995, "text": " and standard deviation over the entire training set and so basically if your batch size is very small"}, {"start": 4377.52, "end": 4382.4800000000005, "text": " momentum of point one is potentially dangerous and it might make it so that the running mean and"}, {"start": 4382.4800000000005, "end": 4387.280000000001, "text": " standard deviation is thrashing too much during training and it's not actually converging properly"}, {"start": 4389.4400000000005, "end": 4394.400000000001, "text": " affine equals true determines whether this batch normalization layer has these learnable affine"}, {"start": 4394.400000000001, "end": 4401.6, "text": " parameters the gain and the bias and this is almost always kept the true I'm not actually sure"}, {"start": 4401.6, "end": 4409.280000000001, "text": " why you would want to change this to false then track running stats is determining whether or not"}, {"start": 4409.280000000001, "end": 4415.360000000001, "text": " batch normalization layer of bite or chip will be doing this and one reason you may you may want"}, {"start": 4415.360000000001, "end": 4420.96, "text": " to skip the running stats is because you may want to for example estimate them at the end as a"}, {"start": 4420.96, "end": 4425.76, "text": " stage two like this and in that case you don't want the batch normalization layer to be doing all"}, {"start": 4425.76, "end": 4431.360000000001, "text": " this extra compute that you're not going to use and finally we need to know which device we're"}, {"start": 4431.36, "end": 4437.12, "text": " going to run this batch normalization on a CPU or a GPU and what the data type should be a half"}, {"start": 4437.12, "end": 4442.48, "text": " precision single precision double precision and so on so that's the batch normalization layer"}, {"start": 4442.48, "end": 4447.44, "text": " otherwise they linked to the paper is the same formula we've implemented and everything is the"}, {"start": 4447.44, "end": 4453.12, "text": " same exactly as we've done here okay so that's everything that I wanted to cover for this lecture"}, {"start": 4453.759999999999, "end": 4457.44, "text": " really what I wanted to talk about is the importance of understanding the activations and"}, {"start": 4457.44, "end": 4462.0, "text": " the gradients and their statistics in neural networks and this becomes increasingly important"}, {"start": 4462.0, "end": 4466.96, "text": " especially as you make your neural networks bigger larger and deeper we looked at the distributions"}, {"start": 4466.96, "end": 4472.16, "text": " basically at the output layer and we saw that if you have two confident misperdictions because"}, {"start": 4472.16, "end": 4476.96, "text": " the activations are too messed up at the last layer you can end up with these hockey stick losses"}, {"start": 4477.5199999999995, "end": 4481.919999999999, "text": " and if you fix this you get a better loss at the end of training because your training is not"}, {"start": 4481.919999999999, "end": 4486.719999999999, "text": " doing wasteful work then we also saw that we need to control the activations we don't want them to"}, {"start": 4486.72, "end": 4493.12, "text": " you know squash to zero or explore to infinity and because that you can run into a lot of trouble with"}, {"start": 4493.12, "end": 4498.08, "text": " all of these nonlinearities in these neural nets and basically you want everything to be fairly homogeneous"}, {"start": 4498.08, "end": 4502.88, "text": " throughout the neural net you want roughly Gaussian activations throughout the neural net let me"}, {"start": 4502.88, "end": 4508.64, "text": " talk about okay if we want roughly Gaussian activations how do we scale these weight matrices"}, {"start": 4508.64, "end": 4514.0, "text": " and biases during initialization of the neural net so that we don't get you know so everything is"}, {"start": 4514.0, "end": 4521.36, "text": " as controlled as possible so that give us a large boost in improvement and then I talked about how"}, {"start": 4521.36, "end": 4529.2, "text": " that strategy is not actually possible for much much deeper neural nets because when you have"}, {"start": 4529.2, "end": 4534.4, "text": " much deeper neural nets with lots of different types of layers it becomes really really hard to"}, {"start": 4534.4, "end": 4539.76, "text": " precisely set the weights and the biases in such a way that the activations are roughly uniform"}, {"start": 4539.76, "end": 4544.88, "text": " throughout the neural net so then I introduced the notion of the normalization layer now there are"}, {"start": 4544.88, "end": 4550.24, "text": " many normalization layers that that people use in practice, Beshaw normalization layer normalization"}, {"start": 4550.24, "end": 4554.64, "text": " this is normalization group normalization we haven't covered most of them but I've introduced"}, {"start": 4554.64, "end": 4559.76, "text": " the first one and also the one that I believe came out first and that's called Beshaw normalization"}, {"start": 4560.64, "end": 4565.12, "text": " and we saw how Beshaw normalization works this is a layer that you can sprinkle throughout your"}, {"start": 4565.12, "end": 4570.96, "text": " deep neural net and the basic idea is if you want roughly Gaussian activations well then take"}, {"start": 4570.96, "end": 4577.2, "text": " your activations and take the mean and the standard deviation and center your data and you can do"}, {"start": 4577.2, "end": 4583.76, "text": " that because the centering operation is differentiable but on top of that we actually had to add a lot"}, {"start": 4583.76, "end": 4588.5599999999995, "text": " of bells and whistles and that gave you a sense of the complexities of the Beshaw normalization layer"}, {"start": 4588.5599999999995, "end": 4593.36, "text": " because now we're centering the data that's great but suddenly we need the gain and the bias"}, {"start": 4593.36, "end": 4599.04, "text": " and now those are trainable and then because we are coupling all the training examples now suddenly"}, {"start": 4599.04, "end": 4604.48, "text": " the question is how do you do the inference or to do the inference we need to now estimate"}, {"start": 4604.48, "end": 4611.04, "text": " these mean and standard deviation once or the entire training set and then use those at inference"}, {"start": 4611.759999999999, "end": 4616.719999999999, "text": " but then no one likes to do stage two so instead we fold everything into the Beshaw normalization"}, {"start": 4616.719999999999, "end": 4621.28, "text": " layer during training and try to estimate these in the running manner so that everything is a bit"}, {"start": 4621.28, "end": 4628.8, "text": " simpler and that gives us the Beshaw normalization layer and as I mentioned no one likes this layer"}, {"start": 4629.44, "end": 4635.84, "text": " it causes a huge amount of bugs and intuitively it's because it is coupling examples"}, {"start": 4636.96, "end": 4642.5599999999995, "text": " in the form of passive and neural net and I've shocked myself in the foot with this layer"}, {"start": 4643.2, "end": 4647.2, "text": " over and over again in my life and I don't want you to suffer the same"}, {"start": 4647.2, "end": 4653.92, "text": " so basically try to avoid it as much as possible some of the other alternatives to these layers are"}, {"start": 4653.92, "end": 4659.2, "text": " for example group normalization or layer normalization and those have become more common in more"}, {"start": 4659.2, "end": 4665.2, "text": " recent deep learning but we haven't covered those yet but definitely Beshaw normalization was very"}, {"start": 4665.2, "end": 4670.32, "text": " influential at the time when it came out in roughly 2015 because it was kind of the first time"}, {"start": 4670.32, "end": 4676.72, "text": " that you could train reliably much deeper neural nets and fundamentally the reason for that is"}, {"start": 4676.72, "end": 4682.08, "text": " because this layer was very effective at controlling the statistics of the activations in the neural net"}, {"start": 4683.12, "end": 4689.12, "text": " so that's the story so far and that's all I wanted to cover and in the future lecture so"}, {"start": 4689.12, "end": 4694.16, "text": " hopefully we can start going into recurring neural nets and recurring neural nets as we'll see"}, {"start": 4694.16, "end": 4700.240000000001, "text": " are just very very deep networks because you you unrolled the loop and when you actually optimize"}, {"start": 4700.24, "end": 4707.36, "text": " these neural nets and that's where a lot of this analysis around the activations statistics"}, {"start": 4707.36, "end": 4712.96, "text": " and all these normalization layers will become very very important for good performance so we'll"}, {"start": 4712.96, "end": 4719.5199999999995, "text": " see that next time bye okay so I lied I would like us to do one more summary here as a bonus and"}, {"start": 4719.5199999999995, "end": 4724.08, "text": " I think it's useful as to have one more summary of everything I've presented in this lecture but"}, {"start": 4724.08, "end": 4728.32, "text": " also I would like us to start by torturing our code a little bit so it looks much more like what"}, {"start": 4728.32, "end": 4734.08, "text": " you would encounter in PyTorch so you'll see that I will structure our code into these modules"}, {"start": 4734.08, "end": 4741.679999999999, "text": " like a linear module and a bachelor module and I'm putting the code inside these modules so that we"}, {"start": 4741.679999999999, "end": 4745.36, "text": " can construct neural networks very much like we would construct the in PyTorch and I will go through"}, {"start": 4745.36, "end": 4751.84, "text": " this in detail so we'll create our neural net then we will do the optimization loop as we did before"}, {"start": 4752.5599999999995, "end": 4756.16, "text": " and then the one more thing that I want to do here is I want to look at the activations statistics"}, {"start": 4756.16, "end": 4761.44, "text": " both in the forward pass and in the backward pass and then here we have the evaluation and sampling"}, {"start": 4761.44, "end": 4767.84, "text": " just like before so let me rewind all the way up here and go a little bit slower so here I'm creating"}, {"start": 4767.84, "end": 4773.28, "text": " a linear layer you'll notice that Torch.nn has lots of different types of layers and one of those"}, {"start": 4773.28, "end": 4778.48, "text": " layers is the linear layer Torch.nn.nl it takes a number of input features output features whether"}, {"start": 4778.48, "end": 4783.28, "text": " or not we should have bias and then the device that we want to place this layer on and the data type"}, {"start": 4783.28, "end": 4789.679999999999, "text": " so I will omit these two but otherwise we have the exact same thing we have the fan in which is"}, {"start": 4789.679999999999, "end": 4795.44, "text": " the number of inputs, fan out the number of outputs and whether or not we want to use a bias and"}, {"start": 4795.44, "end": 4801.679999999999, "text": " internally inside this layer there's a weight and a bias if you like it it is typical to initialize"}, {"start": 4801.679999999999, "end": 4807.599999999999, "text": " the weight using say random numbers drawn from a Gaussian and then here's the coming initialization"}, {"start": 4807.6, "end": 4813.04, "text": " that we discussed already in this lecture and that's a good default and also the default that I"}, {"start": 4813.04, "end": 4819.120000000001, "text": " believe PyTorch chooses and by default the bias is usually initialized to zeros. Now when you call"}, {"start": 4819.120000000001, "end": 4825.6, "text": " this module this will basically calculate W times x plus B if you have NB and then when you also"}, {"start": 4825.6, "end": 4831.360000000001, "text": " call that parameters on this module it will return the tensors that are the parameters of this layer."}, {"start": 4831.36, "end": 4838.5599999999995, "text": " Now next we have the Bachelormalization layer so I've written that here and this is very"}, {"start": 4838.5599999999995, "end": 4846.639999999999, "text": " similar to PyTorch's and then that Bachelorm 1D layer as shown here. So I'm kind of taking these"}, {"start": 4846.639999999999, "end": 4851.839999999999, "text": " three parameters here the dimensionality the epsilon that we'll use in the division and the"}, {"start": 4851.839999999999, "end": 4856.08, "text": " momentum that we will use in keeping track of these running stats the running mean and the running"}, {"start": 4856.08, "end": 4862.24, "text": " variance. Now PyTorch actually takes quite a few more things but I'm assuming some of their settings"}, {"start": 4862.24, "end": 4867.12, "text": " so for us I'll find will be true that means that we will be using a gamma beta after denormalization."}, {"start": 4867.92, "end": 4871.6, "text": " The track running stats will be true so we will be keeping track of the running mean and the"}, {"start": 4871.6, "end": 4878.0, "text": " running variance in the in the classroom. Our device by default is the CPU and the data type by"}, {"start": 4878.0, "end": 4886.08, "text": " default is float float 32. So those are the defaults otherwise we are taking all the same parameters"}, {"start": 4886.08, "end": 4891.6, "text": " in this Bachelorm layer so first I'm just saving them. Now here's something new there's that"}, {"start": 4891.6, "end": 4896.24, "text": " training which by default is true and PyTorch and then modules also have this attribute that"}, {"start": 4896.24, "end": 4902.08, "text": " training and that's because many modules and Bachelorm is included in that have a different"}, {"start": 4902.08, "end": 4906.96, "text": " behavior whether you are training your own or not or whether you are running it in an evaluation"}, {"start": 4906.96, "end": 4911.92, "text": " mode and calculating your evaluation laws or using it for inference on some test examples."}, {"start": 4912.8, "end": 4917.04, "text": " And Bachelorm is an example of this because when we are training we are going to be using the"}, {"start": 4917.04, "end": 4922.08, "text": " mean and the variance estimated from the current batch but during inference we are using the running"}, {"start": 4922.08, "end": 4928.08, "text": " mean and running variance and so also if we are training we are updating mean and variance but if"}, {"start": 4928.08, "end": 4933.36, "text": " we are testing then these are not being updated they're kept fixed and so this flag is necessary"}, {"start": 4933.36, "end": 4938.88, "text": " and by default true just like in PyTorch. Now the parameters of Bachelorm 1D are the gamma and the"}, {"start": 4938.88, "end": 4946.799999999999, "text": " beta here and then the running mean and running variance are called buffers in PyTorch nomenclature"}, {"start": 4947.5199999999995, "end": 4954.08, "text": " and these buffers are trained using exponential moving average here explicitly and they are not"}, {"start": 4954.08, "end": 4958.5599999999995, "text": " part of the back propagation and stochastic gradient descent so they are not sort of like parameters"}, {"start": 4958.56, "end": 4964.56, "text": " of this layer and that's why when we have a parameters here we only return gamma and beta"}, {"start": 4964.56, "end": 4969.04, "text": " we do not return the mean and the variance this is trained sort of like internally here"}, {"start": 4970.160000000001, "end": 4975.6, "text": " every forward pass using exponential moving average. So that's the initialization."}, {"start": 4976.8, "end": 4981.84, "text": " Now in a forward pass if we are training then we use the mean and the variance estimated"}, {"start": 4981.84, "end": 4990.400000000001, "text": " by the batch only a block of paper here. We calculate the mean and the variance. Now up above I was"}, {"start": 4990.400000000001, "end": 4995.6, "text": " estimating the standard deviation and keeping track of the standard deviation here in the running"}, {"start": 4995.6, "end": 5000.56, "text": " standard deviation instead of running variance but let's follow the paper exactly here they"}, {"start": 5000.56, "end": 5005.2, "text": " calculate the variance which is the standard deviation squared and that's what's kept track of"}, {"start": 5005.2, "end": 5010.96, "text": " in the running variance instead of a running standard deviation but those two would be very"}, {"start": 5010.96, "end": 5017.52, "text": " very similar I believe. If we are not training then we use running mean and variance we normalize"}, {"start": 5019.04, "end": 5023.52, "text": " and then here I am calculating the output of this layer and I'm also assigning it to an"}, {"start": 5023.52, "end": 5030.16, "text": " attribute called dot out. Now dot out is something that I'm using in our modules here. This is not"}, {"start": 5030.16, "end": 5034.72, "text": " what you would find in PyTorch we are slightly deviating from it. I'm creating a dot out because I"}, {"start": 5034.72, "end": 5040.32, "text": " would like to very easily maintain all those variables so that we can create statistics of them"}, {"start": 5040.32, "end": 5046.24, "text": " and plot them but PyTorch and modules will not have a dot out attribute. And finally here we are"}, {"start": 5046.24, "end": 5052.88, "text": " updating the buffers using again as I mentioned exponential moving average given the provided momentum"}, {"start": 5052.88, "end": 5058.16, "text": " and importantly you'll notice that I'm using the torshtap no-grat context manager and I'm doing this"}, {"start": 5058.16, "end": 5063.36, "text": " because if we don't use this then PyTorch will start building out an entire computational graph"}, {"start": 5063.36, "end": 5068.32, "text": " out of these tensors because it is expecting that we will eventually call a dot backward but we"}, {"start": 5068.32, "end": 5071.679999999999, "text": " are never going to be calling that backward on anything that includes running mean and running"}, {"start": 5071.679999999999, "end": 5077.36, "text": " variance. So that's why we need to use this context manager so that we are not sort of maintaining"}, {"start": 5077.36, "end": 5082.5599999999995, "text": " them using all this additional memory. So this will make it more efficient and it's just telling"}, {"start": 5082.5599999999995, "end": 5086.4, "text": " PyTorch that they will need no backward. We just have a bunch of tensors we want to update them"}, {"start": 5086.4, "end": 5093.44, "text": " that's it and then we return. Okay now scrolling down we have the 10H layer. This is very very"}, {"start": 5093.44, "end": 5100.16, "text": " similar to torshtap 10H and it doesn't do too much it just calculates 10H as you might expect."}, {"start": 5100.48, "end": 5106.5599999999995, "text": " So that's torshtap 10H and there's no parameters in this layer but because these are layers"}, {"start": 5107.44, "end": 5114.96, "text": " it now becomes very easy to sort of like stack them up into basically just a list and we can do all"}, {"start": 5114.96, "end": 5119.759999999999, "text": " the initializations that we're used to. So we have the initial sort of embedding matrix we have"}, {"start": 5119.76, "end": 5124.8, "text": " our layers and we can call them sequentially and then again with torshtap no grad there's some"}, {"start": 5124.8, "end": 5129.76, "text": " initializations here. So we want to make the output softmax a bit less confident like we saw"}, {"start": 5130.320000000001, "end": 5135.280000000001, "text": " and in addition to that because we are using a six layer multi layer perceptron here so you see how"}, {"start": 5135.280000000001, "end": 5142.0, "text": " I'm stacking linear 10H linear 10H etc. I'm going to be using the gain here and I'm going to play"}, {"start": 5142.0, "end": 5146.08, "text": " with this in a second so you'll see how when we change this what happens to the statistics."}, {"start": 5146.08, "end": 5151.92, "text": " Finally the primers are basically the embedding matrix and all the parameters in all the layers"}, {"start": 5151.92, "end": 5157.36, "text": " and notice here I'm using a double list comprehension if you want to call it that but for every layer"}, {"start": 5157.36, "end": 5162.48, "text": " in layers and for every parameter in each of those layers we are just stacking up all those"}, {"start": 5162.48, "end": 5170.8, "text": " piece all those parameters. Now in total we have 46,000 parameters and I'm telling PyTorch that all"}, {"start": 5170.8, "end": 5179.92, "text": " of them require gradient. Then here we have everything here we are actually mostly used to."}, {"start": 5180.64, "end": 5185.2, "text": " We are sampling batch we are doing a forward pass the forward pass now is just the linear"}, {"start": 5185.2, "end": 5190.400000000001, "text": " application of all the layers in order followed by the cross entropy and then in the backward pass"}, {"start": 5190.400000000001, "end": 5194.64, "text": " you'll notice that for every single layer I now iterate over all the outputs and I'm telling"}, {"start": 5194.64, "end": 5200.16, "text": " PyTorch to retain the gradient of them and then here we are already used to all the all the"}, {"start": 5200.16, "end": 5205.44, "text": " gradients set to none do the backward to fill in the gradients do an update using the cast gradient"}, {"start": 5205.44, "end": 5212.0, "text": " send and then track some statistics and then I am going to break after a single iteration."}, {"start": 5212.0, "end": 5216.96, "text": " Now here in this cell in this diagram I am visualizing the histogram the histograms of the"}, {"start": 5216.96, "end": 5223.36, "text": " forward pass activations and I'm specifically doing it at the 10 each layers. So iterating over all"}, {"start": 5223.36, "end": 5231.12, "text": " the layers except for the very last one which is basically just the softmax layer. If it is a 10"}, {"start": 5231.12, "end": 5235.44, "text": " each layer and I'm using a 10 each layer just because they have a finite output negative 1 to 1"}, {"start": 5235.44, "end": 5240.0, "text": " and so it's very easy to visualize here so you see negative 1 to 1 and it's a finite range and"}, {"start": 5240.0, "end": 5246.88, "text": " easy to work with. I take the out tensor from that layer into t and then I'm calculating the mean"}, {"start": 5246.88, "end": 5252.48, "text": " this 10 deviation and the percent saturation of t and the way I define the percent saturation is"}, {"start": 5252.48, "end": 5258.08, "text": " that t dot absolute value is greater than 0.97 so that means we are here at the tails of the 10"}, {"start": 5258.08, "end": 5262.799999999999, "text": " each and remember that when we are in the tails of the 10 each that will actually stop gradients"}, {"start": 5262.799999999999, "end": 5269.919999999999, "text": " so we don't want this to be too high. Now here I'm calling torshot histogram and then I am plotting"}, {"start": 5269.919999999999, "end": 5274.16, "text": " this histogram. So basically what this is doing is that every different type of layer and they"}, {"start": 5274.16, "end": 5281.04, "text": " all have a different color we are looking at how many values in these tensors take on any of the"}, {"start": 5281.04, "end": 5288.24, "text": " values below on this axis here. So the first layer is fairly saturated here at 20 percent so you"}, {"start": 5288.24, "end": 5293.6, "text": " can see that it's got tails here but then everything sort of stabilizes and if we had more layers"}, {"start": 5293.6, "end": 5298.72, "text": " here it would actually just stabilize at around the 10 deviation of about 0.65 and the saturation"}, {"start": 5298.72, "end": 5304.24, "text": " will be roughly 5 percent and the reason that this stabilizes and gives us a nice distribution here"}, {"start": 5304.24, "end": 5313.04, "text": " is because gain is set to 5 over 3. Now here this gain you see that by default we initialize"}, {"start": 5313.04, "end": 5317.92, "text": " with 1 over square root of fennel but then here during initialization I come in and I iterate"}, {"start": 5317.92, "end": 5323.92, "text": " our all the layers and if it's a linear layer I boost that by the gain. Now we saw that one"}, {"start": 5324.5599999999995, "end": 5331.2, "text": " so basically if we just do not use a gain then what happens? If I retraw this you will see that"}, {"start": 5331.2, "end": 5338.0, "text": " the standard deviation is shrinking and the saturation is coming to 0 and basically what's happening"}, {"start": 5338.0, "end": 5343.679999999999, "text": " is the first layer is you know pretty decent but then further layers are just kind of like shrinking"}, {"start": 5343.679999999999, "end": 5349.92, "text": " down to 0 and it's happening slowly but it's shrinking to 0 and the reason for that is when you"}, {"start": 5349.92, "end": 5357.84, "text": " just have a sandwich of linear layers alone then a then initializing our weights in this manner"}, {"start": 5357.84, "end": 5363.12, "text": " we saw previously would have conserved the standard deviation of 1 but because we have this"}, {"start": 5363.12, "end": 5370.4800000000005, "text": " interspersed 10H layers in there these 10H layers are squashing functions and so they take your"}, {"start": 5370.4800000000005, "end": 5376.64, "text": " distribution and they slightly squash it and so some gain is necessary to keep expanding it"}, {"start": 5377.2, "end": 5384.32, "text": " to fight the squashing so it just turns out that 5 over 3 is a good value so if we have something"}, {"start": 5384.32, "end": 5390.88, "text": " too small like 1 we saw that things will come towards 0 but if it's something too high let's do 2"}, {"start": 5392.4, "end": 5400.32, "text": " then here we see that well let me do something a bit more extreme because so it's a bit more visible"}, {"start": 5400.32, "end": 5407.679999999999, "text": " let's try 3 okay so we see here that the saturation is not going to be too large okay so 3 would"}, {"start": 5407.68, "end": 5416.16, "text": " create weight-o-saturated activations so 5 over 3 is a good setting for a sandwich of linear layers"}, {"start": 5416.16, "end": 5421.84, "text": " with 10H activations and it roughly stabilizes the standard deviation at a reasonable point"}, {"start": 5422.56, "end": 5428.08, "text": " now honestly I have no idea where 5 over 3 came from in PyTorch when we were looking at the"}, {"start": 5428.08, "end": 5434.320000000001, "text": " counting initialization I see empirically that it stabilizes the sandwich of linear and 10H"}, {"start": 5434.32, "end": 5438.719999999999, "text": " and that the saturation is in a good range but I don't actually know if this came out of some"}, {"start": 5438.719999999999, "end": 5444.16, "text": " math formula I tried searching briefly for where this comes from but I wasn't able to find anything"}, {"start": 5444.719999999999, "end": 5449.36, "text": " but certainly we see that empirically these are very nice ranges our saturation is roughly 5%"}, {"start": 5449.36, "end": 5456.0, "text": " which is a pretty good number and this is a good setting of the gain in this context similarly"}, {"start": 5456.0, "end": 5461.36, "text": " we can do the exact same thing with the gradients so here is a very same loop if it's a 10H"}, {"start": 5461.36, "end": 5465.5199999999995, "text": " but instead of taking the layer that out I'm taking the grad and then I'm also showing the mean"}, {"start": 5465.5199999999995, "end": 5470.48, "text": " and the standard deviation and I'm plotting the histogram of these values and so you'll see"}, {"start": 5470.48, "end": 5475.2, "text": " that the gradient distribution is fairly reasonable and in particular what we're looking for is that"}, {"start": 5475.2, "end": 5480.5599999999995, "text": " all the different layers in this sandwich has roughly the same gradient things are not shrinking"}, {"start": 5480.5599999999995, "end": 5486.0, "text": " or exploding so we can for example come here and we can take a look at what happens if this gain"}, {"start": 5486.0, "end": 5494.24, "text": " was way too small so this was 0.5 then you see the first of all the activations are shrinking to zero"}, {"start": 5494.24, "end": 5498.88, "text": " but also the gradients are doing something weird the gradients started out here and then now they're"}, {"start": 5498.88, "end": 5507.04, "text": " like expanding out and similarly if we for example have a two-hymogain select three then we see"}, {"start": 5507.04, "end": 5511.36, "text": " that also the gradients have there's some asymmetry going on where as you go into deeper and deeper"}, {"start": 5511.36, "end": 5516.4, "text": " layers the activations are also changing and so that's not what we want and in this case we saw"}, {"start": 5516.4, "end": 5522.4, "text": " that without use of besterm as we are going through right now we have to very carefully set those"}, {"start": 5522.4, "end": 5528.4, "text": " gains to get nice activations in both the forward pass and the backward pass now before we move on"}, {"start": 5528.4, "end": 5533.92, "text": " to bestermalization I would also like to take a look at what happens when we have no 10H units here"}, {"start": 5533.92, "end": 5540.799999999999, "text": " so erasing all the 10H nonlinearities but keeping the gain at 5 over 3 we now have just a giant"}, {"start": 5540.8, "end": 5546.400000000001, "text": " linear sandwich so let's see what happens to the activations as we saw before the correct gain"}, {"start": 5546.400000000001, "end": 5554.08, "text": " here is one that is the standard deviation preserving gain so 1.667 is too high and so what's"}, {"start": 5554.08, "end": 5560.72, "text": " going to happen now is the following I have to change this to be linear so we are because there's"}, {"start": 5560.72, "end": 5569.52, "text": " no more 10H layers and let me change this to linear as well so what we're seeing is the activations"}, {"start": 5569.52, "end": 5575.84, "text": " started out on the blue and have by layer 4 become very diffuse so what's happening to the"}, {"start": 5575.84, "end": 5583.120000000001, "text": " activations is this and with the gradients on the top layer the activation the gradient statistics"}, {"start": 5583.120000000001, "end": 5588.240000000001, "text": " are the purple and then they diminish as you go down deeper in the layers and so basically"}, {"start": 5588.240000000001, "end": 5592.400000000001, "text": " you have an asymmetry like in the neural net and you might imagine that if you have very deep"}, {"start": 5592.400000000001, "end": 5598.0, "text": " neural networks say like 50 layers or something like that this just this is not a good place to be"}, {"start": 5598.0, "end": 5604.8, "text": " also that's why before best normalization this was an incredibly tricky to to set in particular"}, {"start": 5604.8, "end": 5610.8, "text": " if this is too large of a gain this happens and if it's too little of a gain then this happens"}, {"start": 5610.8, "end": 5620.0, "text": " also the opposite of that basically happens here we have a shrinking and a diffusion depending on"}, {"start": 5620.0, "end": 5624.96, "text": " which direction you look at it from and so certainly this is not what you want and in this case the"}, {"start": 5624.96, "end": 5630.8, "text": " correct setting of the gain is exactly one just like we're doing at initialization and then we see"}, {"start": 5630.8, "end": 5638.0, "text": " that the statistics for the forward and the backward pass are well behaved and so the reason I want"}, {"start": 5638.0, "end": 5644.24, "text": " to show you this is the basically like getting neuralness to train before these normalization layers"}, {"start": 5644.24, "end": 5648.96, "text": " and before the use of advanced optimizers like Adam which we still have to cover and residual"}, {"start": 5648.96, "end": 5654.88, "text": " connections and so on training neuralness basically look like this it's like a total balancing act"}, {"start": 5654.88, "end": 5659.04, "text": " you have to make sure that everything is precisely orchestrated and you have to care about the"}, {"start": 5659.04, "end": 5663.84, "text": " activations and the gradients and their statistics and then maybe you can train something but it was"}, {"start": 5663.84, "end": 5668.0, "text": " basically impossible to train very deep networks and this is fundamentally the reason for that"}, {"start": 5668.0, "end": 5673.6, "text": " you'd have to be very very careful with your initialization the other point here is"}, {"start": 5673.6, "end": 5678.4800000000005, "text": " you might be asking yourself by the way I'm not sure if I covered this why do we need these 10H"}, {"start": 5678.96, "end": 5684.56, "text": " layers at all why do we include them and then have to worry about the gain and the reason for"}, {"start": 5684.56, "end": 5689.280000000001, "text": " that of course is that if you just have a stack of linear layers then certainly we're getting very"}, {"start": 5689.280000000001, "end": 5695.120000000001, "text": " easily nice activations and so on but this is just a massive linear sandwich and it turns out"}, {"start": 5695.120000000001, "end": 5700.8, "text": " that it collapses to a single linear layer in terms of its representation power so if you were to"}, {"start": 5700.8, "end": 5705.04, "text": " plot the output as a function of the input you're just getting a linear function no matter how many"}, {"start": 5705.04, "end": 5710.72, "text": " linear layers you stack up you still just end up with a linear transformation all the WX plus B's"}, {"start": 5710.72, "end": 5715.84, "text": " just collapse into a large WX plus B with slightly different W's as slightly different B"}, {"start": 5717.52, "end": 5721.92, "text": " but interestingly even though the forward pass collapses to just a linear layer because of"}, {"start": 5721.92, "end": 5728.56, "text": " backpapigation and the dynamics of the backward pass the optimization is really is not identical"}, {"start": 5728.56, "end": 5735.52, "text": " you actually end up with all kinds of interesting dynamics in the backward pass because of the"}, {"start": 5735.52, "end": 5741.68, "text": " the way the chain rule is calculating it and so optimizing a linear layer by itself and optimizing"}, {"start": 5741.68, "end": 5746.160000000001, "text": " a sandwich of 10 linear layers in both cases those are just a linear transformation in the forward"}, {"start": 5746.160000000001, "end": 5751.120000000001, "text": " pass but the training dynamics would be different and there's entire papers that analyze in fact"}, {"start": 5751.120000000001, "end": 5756.72, "text": " like infinitely layered linear layers and so on and so there's a lot of things to that you can play"}, {"start": 5756.72, "end": 5766.88, "text": " with there but basically the tenational linearities allow us to turn this sandwich from just a linear"}, {"start": 5768.4800000000005, "end": 5774.4800000000005, "text": " function into a neural network that can in principle approximate any arbitrary function"}, {"start": 5775.52, "end": 5781.280000000001, "text": " okay so now I've reset the code to use the linear ten each sandwich like before and I've reset"}, {"start": 5781.28, "end": 5786.8, "text": " everything so the gains five over three we can run a single step of optimization and we can look"}, {"start": 5786.8, "end": 5791.44, "text": " at the activations statistics of the forward pass and the backward pass but I've added one more"}, {"start": 5791.44, "end": 5794.96, "text": " plot here that I think is really important to look at when you're training your neural nets and"}, {"start": 5794.96, "end": 5800.08, "text": " to consider and ultimately what we're doing is we're updating the parameters of the neural net"}, {"start": 5800.08, "end": 5805.44, "text": " so we care about the parameters and their values and their gradients so here what I'm doing is"}, {"start": 5805.44, "end": 5808.639999999999, "text": " I'm actually iterating over all the parameters available and that I'm only"}, {"start": 5808.64, "end": 5814.64, "text": " restricting it to the two-dimensional parameters which are basically the weights of these linear layers"}, {"start": 5814.64, "end": 5820.96, "text": " and I'm skipping the biases and I'm skipping the gammas and the betas and the best room just for"}, {"start": 5820.96, "end": 5825.76, "text": " simplicity but you can also take a look at those as well but what's happening with the weights is"}, {"start": 5826.64, "end": 5833.84, "text": " instructive by itself so here we have all the different weights their shapes so this is the embedding"}, {"start": 5833.84, "end": 5838.160000000001, "text": " layer the first linear layer all the way to the very last linear layer and then we have the"}, {"start": 5838.16, "end": 5843.84, "text": " mean the standard deviation of all these primers the histogram and you can see that actually it"}, {"start": 5843.84, "end": 5848.72, "text": " doesn't look that amazing so there's some trouble in paradise even though these gradients looked okay"}, {"start": 5848.72, "end": 5853.68, "text": " there's something weird going on here I'll get to that in a second and the last thing here is the"}, {"start": 5853.68, "end": 5859.04, "text": " gradient to data ratio so sometimes I'll have to visualize this as well because what this could"}, {"start": 5859.04, "end": 5864.88, "text": " see a sense of is what is the scale of the gradient compared to the scale of the actual values"}, {"start": 5864.88, "end": 5871.36, "text": " and this is important because we're going to end up taking a step update that is the learning"}, {"start": 5871.36, "end": 5877.2, "text": " rate times the gradient onto the data and so the gradient has two larger magnitudes if the numbers"}, {"start": 5877.2, "end": 5882.400000000001, "text": " and there are two large compared to the numbers in data then you'd be in trouble but in this case"}, {"start": 5882.400000000001, "end": 5889.84, "text": " the gradient to data is our low numbers so the values inside grad are 1000 times smaller than the values"}, {"start": 5889.84, "end": 5897.12, "text": " inside data in these weights most of them now notably that is not true about the last layer"}, {"start": 5897.12, "end": 5901.2, "text": " and so the last layer actually here the output layer is a bit of a trouble maker in the way that"}, {"start": 5901.2, "end": 5909.28, "text": " this is currently arranged because you can see that the last layer here in pink takes on values"}, {"start": 5909.28, "end": 5916.8, "text": " there are much larger than some of the values inside inside the neural net so the standard deviations"}, {"start": 5916.8, "end": 5923.76, "text": " are roughly 1-3 throughout except for the last but last new layer which actually has roughly 1-2"}, {"start": 5923.76, "end": 5930.0, "text": " standard deviation of gradients and so the gradients on the last layer are currently about 100 times"}, {"start": 5930.0, "end": 5936.24, "text": " greater sorry 10 times greater than all the other weights inside the neural net and so this"}, {"start": 5936.24, "end": 5941.360000000001, "text": " problematic because in the simple stochastic gradient in the sense setup you would be training"}, {"start": 5941.36, "end": 5947.44, "text": " this last layer about 10 times faster than you would be training the other layers at initialization now"}, {"start": 5947.44, "end": 5951.92, "text": " this actually like kind of fixes itself a little bit if you train for a bit longer so for example"}, {"start": 5951.92, "end": 5959.36, "text": " if I agree then 1000 only then do a break let me initialize and then let me do it 1000 steps"}, {"start": 5960.0, "end": 5966.24, "text": " and after 1000 steps we can look at the for a pass okay so you see how the neurons are a bit"}, {"start": 5966.24, "end": 5971.44, "text": " are saturating a bit and we can also look at the backward pass but otherwise they look good they're"}, {"start": 5971.44, "end": 5977.5199999999995, "text": " about equal and there's no shrinking to zero or exploding to infinities and you can see that here"}, {"start": 5977.5199999999995, "end": 5982.8, "text": " in the weights things are also stabilizing a little bit so the tails of the last pink layer"}, {"start": 5982.8, "end": 5988.8, "text": " are actually coming in during the optimization but certainly this is like a little bit of troubling"}, {"start": 5988.8, "end": 5992.96, "text": " especially if you are using a very simple update rule like stochastic gradient descent instead of"}, {"start": 5992.96, "end": 5997.6, "text": " a modern optimizer like atom now I'd like to show you one more plot that I usually look at when"}, {"start": 5997.6, "end": 6003.28, "text": " I train your own works and basically the gradient to data ratio is not actually that informative"}, {"start": 6003.28, "end": 6008.4, "text": " because what matters at the end is not the gradient to data ratio but the update to the data ratio"}, {"start": 6008.4, "end": 6013.52, "text": " because that is the amount by which we will actually change the data in these tensors so coming up"}, {"start": 6013.52, "end": 6020.32, "text": " here what I'd like to do is I'd like to introduce a new update to data ratio it's going to be less"}, {"start": 6020.32, "end": 6025.04, "text": " than we're going to build it out every single iteration and here I'd like to keep track of basically"}, {"start": 6025.84, "end": 6035.36, "text": " the ratio every single iteration so without any ingredients I'm comparing the update which is"}, {"start": 6035.36, "end": 6040.88, "text": " learning rate times the time is the gradient that is the update that we're going to apply to every"}, {"start": 6040.88, "end": 6045.92, "text": " parameter associated with random world of parameters and then I'm taking the basically standard"}, {"start": 6045.92, "end": 6053.52, "text": " deviation of the update we're going to apply and divided by the actual content the data of that"}, {"start": 6053.52, "end": 6060.0, "text": " parameter and its standard deviation so this is the ratio of basically how great are the updates"}, {"start": 6060.0, "end": 6064.56, "text": " to the values in these tensors then we're going to take a log of it and actually I'd like to take a"}, {"start": 6064.56, "end": 6071.6, "text": " log 10 just so it's a nice serviceualization so we're going to be basically looking at the"}, {"start": 6071.6, "end": 6080.0, "text": " exponents of this division here and then that item to pop out the float and we're going to be"}, {"start": 6080.0, "end": 6084.8, "text": " keeping track of this for all the parameters and adding it to this UD tensor so now let me"}, {"start": 6084.8, "end": 6091.280000000001, "text": " re-initialize and run a thousand iterations we can look at the activations the gradients"}, {"start": 6091.92, "end": 6096.0, "text": " and the parameter gradients as we did before but now I have one more plot here to introduce"}, {"start": 6096.0, "end": 6101.76, "text": " now what's happening here is we're every interval of parameters and I'm constraining it again"}, {"start": 6101.76, "end": 6108.16, "text": " like I did here to just to weights so the number of dimensions in these sensors is two and then I'm"}, {"start": 6108.16, "end": 6117.52, "text": " basically plotting all of these update ratios over time so when I plot this I plot those ratios"}, {"start": 6117.52, "end": 6122.16, "text": " and you can see that they evolve over time during initialization that they concern values and then"}, {"start": 6122.16, "end": 6126.72, "text": " these updates are like start stabilizing usually during training then the other thing that I'm"}, {"start": 6126.72, "end": 6131.76, "text": " plotting here is I'm plotting here like an approximate value that is a rough guide for what it"}, {"start": 6131.76, "end": 6136.24, "text": " roughly should be and it should be like roughly one in negative three and so that means that"}, {"start": 6136.24, "end": 6142.639999999999, "text": " basically there's some values in this tensor and they take on certain values and the updates"}, {"start": 6142.639999999999, "end": 6149.04, "text": " to them at every single iteration are no more than roughly one thousand of the actual magnitude"}, {"start": 6149.04, "end": 6156.64, "text": " in those tensors if this was much larger like for example if this was if the log of this was like"}, {"start": 6156.64, "end": 6161.04, "text": " same negative one this is actually updating those values quite a lot they're undergoing a lot of"}, {"start": 6161.04, "end": 6167.68, "text": " change but the reason that the final rate the final layer here is an outlier is because this layer"}, {"start": 6167.68, "end": 6177.12, "text": " was artificially shrugged down to keep the softmax income unconfident so here you see how we multiply"}, {"start": 6177.12, "end": 6183.2, "text": " the weight by point one in the initialization to make the last layer prediction less confident"}, {"start": 6184.16, "end": 6189.84, "text": " that made that artificially made the values inside that tensor way too low and that's why we're"}, {"start": 6189.84, "end": 6195.76, "text": " getting temporarily a very high ratio but you see that that stabilizes over time once that weight"}, {"start": 6195.76, "end": 6201.36, "text": " starts to learn starts to learn but basically I like to look at the evolution of this update ratio"}, {"start": 6201.36, "end": 6208.08, "text": " for all my parameters usually and I like to make sure that it's not too much above one negative three"}, {"start": 6208.08, "end": 6214.799999999999, "text": " roughly so around negative three on this log plot if it's below negative three usually that means"}, {"start": 6214.799999999999, "end": 6219.599999999999, "text": " that parameters are not training fast enough so if our learning rate was very low let's do that"}, {"start": 6219.599999999999, "end": 6226.24, "text": " experiment let's initialize and then let's actually do a learning rate of say one in negative three"}, {"start": 6226.24, "end": 6236.88, "text": " here so 0.001 if your learning rate is way too low this plot will typically reveal it so you see"}, {"start": 6236.88, "end": 6244.719999999999, "text": " how all of these updates are way too small so the size of the update is basically 10,000 times"}, {"start": 6246.4, "end": 6251.84, "text": " in magnitude to the size of the numbers in that tensor in the first place so this is a symptom"}, {"start": 6251.84, "end": 6257.12, "text": " of training way too slow so this is another way to sometimes start to learning rate and to get"}, {"start": 6257.12, "end": 6261.52, "text": " a sense of what that learning rate should be and ultimately this is something that you would keep track of"}, {"start": 6265.04, "end": 6270.64, "text": " if anything the learning rate here is a little bit on the higher side because you see that"}, {"start": 6271.76, "end": 6276.56, "text": " we're above the black line of negative three we're somewhere around negative 2.5 it's like okay"}, {"start": 6277.04, "end": 6281.68, "text": " and but everything is like somewhat stabilizing and so this looks like a pretty decent setting of"}, {"start": 6281.68, "end": 6286.96, "text": " of learning rates and so on but this is something to look at and when things are miscalibrated you"}, {"start": 6286.96, "end": 6292.8, "text": " will you will see very quickly so for example everything looks pretty well behaved right but just"}, {"start": 6292.8, "end": 6297.280000000001, "text": " as a comparison when things are not properly calibrated what does that look like let me come up here"}, {"start": 6297.84, "end": 6304.0, "text": " and let's say that for example what do we do let's say that we forgot to apply this"}, {"start": 6304.0, "end": 6308.8, "text": " a fan in normalization so the weights inside the linear layers are just sample from a Gaussian"}, {"start": 6308.8, "end": 6315.4400000000005, "text": " in all the stages what happens to our how do we notice that something's off well the activation"}, {"start": 6315.4400000000005, "end": 6320.320000000001, "text": " plot will tell you whoa your neurons are way too saturated the gradients are going to be all messed up"}, {"start": 6321.360000000001, "end": 6325.92, "text": " the histogram for these weights are going to be all messed up as well and there's a lot of"}, {"start": 6325.92, "end": 6332.0, "text": " asymmetry and then if we look here I suspect it's all going to be also pretty messed up so you see"}, {"start": 6332.0, "end": 6337.360000000001, "text": " there's a lot of discrepancy in how fast these layers are learning and some of them are learning"}, {"start": 6337.36, "end": 6344.48, "text": " way too fast so negative 1 negative 1.5 those are very large numbers in terms of this ratio again"}, {"start": 6344.48, "end": 6349.679999999999, "text": " you should be somewhere around negative three and not much more about that so this is how"}, {"start": 6349.679999999999, "end": 6355.04, "text": " miscalibration so if your neurons are going to manifest and these kinds of plots here are a good way of"}, {"start": 6356.08, "end": 6363.839999999999, "text": " sort of bringing those miscalibration sort of to your attention and so you can address them"}, {"start": 6363.84, "end": 6369.2, "text": " okay so so far we've seen that when we have this linear 10-H sandwich we can actually precisely"}, {"start": 6369.2, "end": 6374.32, "text": " calibrate the gains and make the activations the gradients and the parameters and the updates all"}, {"start": 6374.32, "end": 6380.56, "text": " look pretty decent but it definitely feels a little bit like balancing of a pencil on your finger"}, {"start": 6381.12, "end": 6386.64, "text": " and that's because this gain has to be very precisely calibrated so now let's introduce"}, {"start": 6386.64, "end": 6394.08, "text": " bashfulization layers into the fix into the mix and let's let's see how that helps fix the problem so"}, {"start": 6394.08, "end": 6401.4400000000005, "text": " here I'm going to take the bashful monday class and I'm going to start placing it inside and as I"}, {"start": 6401.4400000000005, "end": 6407.360000000001, "text": " mentioned before the standard typical placey would place it is between the linear layer so right"}, {"start": 6407.360000000001, "end": 6412.72, "text": " after it before the non-linearity but people have definitely played with that and in fact you can"}, {"start": 6412.72, "end": 6418.64, "text": " get very similar results even if you place it after the non-linearity and the other thing that I"}, {"start": 6418.64, "end": 6423.04, "text": " wanted to mention is it's totally fine to also place it at the end after the last linear layer"}, {"start": 6423.04, "end": 6430.4800000000005, "text": " and before the last function so this is potentially fine as well and in this case this would be"}, {"start": 6430.4800000000005, "end": 6438.64, "text": " output would be woke up size now because the last layer is bashful we would not be changing to wait"}, {"start": 6438.64, "end": 6444.64, "text": " to make the softmax less confident we'd be changing the gamma because gamma remember in the"}, {"start": 6444.64, "end": 6449.92, "text": " bash room is the variable that multiplicatively interacts with the output of that normalization"}, {"start": 6452.64, "end": 6458.64, "text": " so we can initialize this sandwich now we can train and we can see that the activations"}, {"start": 6459.360000000001, "end": 6464.8, "text": " are going to of course look very good and they are going to necessarily look good because now before"}, {"start": 6464.8, "end": 6471.4400000000005, "text": " every single 10-H layer there is a normalization in the bash room so this is unsurprisingly all"}, {"start": 6472.16, "end": 6477.4400000000005, "text": " looks pretty good it's going to be standard deviation of roughly 0.65 2% and roughly equals"}, {"start": 6477.4400000000005, "end": 6483.52, "text": " standard deviation throughout the entire layers so everything looks very homogeneous the gradients"}, {"start": 6483.52, "end": 6492.72, "text": " look good the weights look good and their distributions and then the updates also look pretty"}, {"start": 6492.72, "end": 6499.280000000001, "text": " reasonable we're going above negative 3 a little bit but not by too much so all the parameters"}, {"start": 6499.280000000001, "end": 6508.0, "text": " are training in roughly the same rate here but now what we gained is we are going to be slightly less"}, {"start": 6511.4400000000005, "end": 6516.64, "text": " brittle with respect to the gain of these so for example I can make the gain B say 0.2 here"}, {"start": 6516.64, "end": 6524.400000000001, "text": " which is much more personal over than what we had with the 10-H but as we'll see the activations"}, {"start": 6524.400000000001, "end": 6530.240000000001, "text": " will actually be exactly unaffected and that's because again this explicit normalization the gradients"}, {"start": 6530.240000000001, "end": 6535.4400000000005, "text": " are going to look okay the weight gradients are going to look okay but actually the updates will"}, {"start": 6535.4400000000005, "end": 6541.92, "text": " change and so even though the forward and backward paths to a very large extent look okay because"}, {"start": 6541.92, "end": 6546.56, "text": " of the backward paths of the bash room and how the scale of the incoming activations interact"}, {"start": 6546.56, "end": 6554.8, "text": " in the bash room and its backward paths this is actually changing the scale of the updates on"}, {"start": 6554.8, "end": 6561.4400000000005, "text": " these parameters so the gradients of these weights are affected so we still don't get a completely"}, {"start": 6561.4400000000005, "end": 6568.8, "text": " free pass to pass in arbitrary weights here but everything else is significantly more robust in"}, {"start": 6568.8, "end": 6574.4800000000005, "text": " terms of the forward backward and the weight gradients it's just that you may have to retune your"}, {"start": 6574.48, "end": 6580.16, "text": " learning rate if you are changing sufficiently the scale of the activations that are coming into"}, {"start": 6580.16, "end": 6586.5599999999995, "text": " the bash rooms so here for example this we changed the gains of these linear layers to be"}, {"start": 6586.5599999999995, "end": 6592.5599999999995, "text": " greater and we're seeing that the updates are coming out lower as a result and then finally we can"}, {"start": 6592.5599999999995, "end": 6597.759999999999, "text": " also if we are using bash rooms we don't actually need to necessarily let me reset this to one"}, {"start": 6597.759999999999, "end": 6604.16, "text": " so there's no gain we don't necessarily even have to normalize back then in sometimes so if I take"}, {"start": 6604.16, "end": 6609.599999999999, "text": " out the fan in so these are just now random Gaussian we'll see that because of bash rooms this"}, {"start": 6609.599999999999, "end": 6617.28, "text": " will actually be relatively well behaved so this this is look of course in the forward pass look good"}, {"start": 6617.28, "end": 6625.12, "text": " the gradients look good the backward the weight updates look okay a little bit of fat tails and some"}, {"start": 6625.12, "end": 6633.36, "text": " delayers and this looks okay as well but as you as you can see we're significantly below negative"}, {"start": 6633.36, "end": 6638.32, "text": " three so we'd have to bump up the learning rate of this bachelor so that we are training more"}, {"start": 6638.32, "end": 6643.2, "text": " properly and in particular looking at this roughly looks like we have to 10x the learning rate"}, {"start": 6643.2, "end": 6650.5599999999995, "text": " to get to about 1 e negative 3 so we'd come here and we would change this to be update of 1.0"}, {"start": 6651.36, "end": 6652.719999999999, "text": " and if I initialize"}, {"start": 6652.72, "end": 6664.72, "text": " then we'll see that everything still of course looks good and now we are roughly here and we expect"}, {"start": 6664.72, "end": 6670.400000000001, "text": " this to be an okay training run so long story short we are significantly more robust to the gain"}, {"start": 6670.400000000001, "end": 6675.52, "text": " of these linear layers whether or not we have to apply the fan in and then we can change the gain"}, {"start": 6676.320000000001, "end": 6682.0, "text": " but we actually do have to worry a little bit about the update scales and making sure that the"}, {"start": 6682.0, "end": 6686.96, "text": " learning rate is properly calibrated here but the activations of the forward backward pass and the"}, {"start": 6686.96, "end": 6692.24, "text": " updates are all are looking significantly more well behaved except for the global scale that is"}, {"start": 6692.24, "end": 6698.08, "text": " potentially being adjusted here okay so now let me summarize there are three things I was hoping to"}, {"start": 6698.08, "end": 6702.48, "text": " achieve with this section number one I wanted to introduce you to bachelor normalization which is"}, {"start": 6702.48, "end": 6707.92, "text": " one of the first modern innovations that we're looking into that helped stabilize very deep neural"}, {"start": 6707.92, "end": 6713.76, "text": " networks and their training and I hope you understand how the bachelor normalization works and how"}, {"start": 6713.76, "end": 6719.76, "text": " it would be used in neural network number two I was hoping to pie torchify some wire code and wrap"}, {"start": 6719.76, "end": 6727.28, "text": " it up into these modules so like linear bachelor mondi 10h etc these are layers or modules and they"}, {"start": 6727.28, "end": 6733.92, "text": " can be stacked up into neural nets like Lego building blocks and these layers actually exist in"}, {"start": 6733.92, "end": 6739.4400000000005, "text": " pie torch and if you import torch and then then you can actually the way I've constructed it you"}, {"start": 6739.4400000000005, "end": 6746.56, "text": " can simply just use pie torch by pre-pending nn dot to all these different layers and actually"}, {"start": 6746.56, "end": 6751.28, "text": " everything will just work because the API that I have developed here is identical to the API that"}, {"start": 6751.28, "end": 6756.96, "text": " pie torch uses and the implementation also is basically as far as I'm aware identical to the one"}, {"start": 6756.96, "end": 6762.64, "text": " in pie torch and number three I try to introduce you to the diagnostic tools that you would use to"}, {"start": 6762.64, "end": 6767.52, "text": " understand whether your neural network is in a good state dynamically so we are looking at the"}, {"start": 6767.52, "end": 6772.72, "text": " statistics and histograms and activation of the forward pass activation activations the backward"}, {"start": 6772.72, "end": 6777.52, "text": " pass gradients and then also we're looking at the weights that are going to be updated as part of"}, {"start": 6777.52, "end": 6782.8, "text": " stochasticity in the send and we're looking at their means standard deviations and also the ratio"}, {"start": 6782.8, "end": 6789.84, "text": " of gradients to data or even better the updates to data and we saw that typically we don't actually"}, {"start": 6789.84, "end": 6794.56, "text": " look at it as a single snapshot frozen in time at some particular iteration typically people look"}, {"start": 6794.56, "end": 6799.84, "text": " at this as a over time just like I've done here and they look at these update to data ratios and they"}, {"start": 6799.84, "end": 6806.08, "text": " make sure everything looks okay and in particular I said that one in negative three or basically"}, {"start": 6806.08, "end": 6812.0, "text": " negative three on the log scale is a good rough heuristic for what you want this ratio to be and if"}, {"start": 6812.0, "end": 6817.12, "text": " it's way too high then probably the learning rate or the updates are too big and if it's way too"}, {"start": 6817.12, "end": 6821.04, "text": " small that the learning rate is probably too small so that's just some of the things that you"}, {"start": 6821.04, "end": 6827.2, "text": " may want to play with when you try to get your neural network to work well very well now there's"}, {"start": 6827.2, "end": 6831.5199999999995, "text": " a number of things I did not try to achieve I did not try to beat our previous performance as an"}, {"start": 6831.5199999999995, "end": 6837.36, "text": " example by introducing the bachelor layer actually I did try and I found that you I used the"}, {"start": 6837.36, "end": 6841.76, "text": " learning rate finding mechanism that I've described before I tried to train the bachelor layer"}, {"start": 6841.76, "end": 6846.64, "text": " a bachelor neural nut and I actually ended up with results that are very very similar to what we"}, {"start": 6846.64, "end": 6852.96, "text": " obtained before and that's because our performance now is not bottlenecked by the optimization"}, {"start": 6852.96, "end": 6857.6, "text": " which is what bachelor is helping with the performance at the stage is bottlenecked by what I"}, {"start": 6857.6, "end": 6863.68, "text": " suspect is the context length of our context so currently we are taking three characters to"}, {"start": 6863.68, "end": 6867.6, "text": " predict the fourth one and I think we need to go beyond that and we need to look at more powerful"}, {"start": 6867.6, "end": 6873.200000000001, "text": " architectures like recurrent neural networks and transformers in order to further push the"}, {"start": 6873.2, "end": 6879.5199999999995, "text": " block probabilities that we're achieving on this day as it and I also did not try to have a full"}, {"start": 6879.5199999999995, "end": 6884.32, "text": " explanation of all of these activations, the gradients and the backward pass and the statistics of"}, {"start": 6884.32, "end": 6888.4, "text": " all these gradients and so you may have found some of the parts here on intuitive and maybe you're"}, {"start": 6888.4, "end": 6893.36, "text": " slightly confused about okay if I change the gain here how come that we need a different"}, {"start": 6893.36, "end": 6896.88, "text": " learning rate and I didn't go into the full detail because you'd have to actually look at the"}, {"start": 6896.88, "end": 6901.28, "text": " backward pass of all these different layers and get an intuitive understanding of how that works"}, {"start": 6901.28, "end": 6906.16, "text": " and I did not go into that in this lecture the purpose really was just to introduce you to the"}, {"start": 6906.16, "end": 6910.48, "text": " diagnostic tools and what they look like but there's still a lot of work remaining on the intuitive"}, {"start": 6910.48, "end": 6916.08, "text": " level to understand the initialization the backward pass and how all that interacts but you"}, {"start": 6916.08, "end": 6922.88, "text": " shouldn't feel too bad because honestly we are getting to the cutting edge of where the field is"}, {"start": 6922.88, "end": 6928.16, "text": " we certainly haven't I would say solved initialization and we haven't solved back propagation"}, {"start": 6928.16, "end": 6931.92, "text": " and these are still very much an active area of research people are still trying to figure out"}, {"start": 6931.92, "end": 6935.28, "text": " where's the best way to initialize these networks what is the best update rule to use"}, {"start": 6936.5599999999995, "end": 6940.72, "text": " and so on so none of this is really solved and we don't really have all the answers to all the"}, {"start": 6942.0, "end": 6946.96, "text": " to you know all these cases but at least you know we're making progress and at least we have some"}, {"start": 6946.96, "end": 6953.599999999999, "text": " tools to tell us whether or not things are on the right track for now so I think we've made"}, {"start": 6953.6, "end": 6961.04, "text": " positive progress in this lecture and I hope you enjoyed that and I will see you next time"}]