We implement a bigram character-level language model, which we will further complexify in followup videos into a modern Transformer language model, like GPT. In this video, the focus is on (1) introducing torch.Tensor and its subtleties and use in efficiently evaluating neural networks and (2) the overall framework of language modeling that includes model training, sampling, and the evaluation of a loss (e.g. the negative log likelihood for classification).

Links:
- makemore on github: https://github.com/karpathy/makemore
- jupyter notebook I built in this video: https://github.com/karpathy/nn-zero-to-hero/blob/master/lectures/makemore/makemore_part1_bigrams.ipynb
- my website: https://karpathy.ai
- my twitter: https://twitter.com/karpathy
- (new) Neural Networks: Zero to Hero series Discord channel: https://discord.gg/Hp2m3kheJn , for people who'd like to chat more and go beyond youtube comments

Useful links for practice:
- Python + Numpy tutorial from CS231n https://cs231n.github.io/python-numpy-tutorial/ . We use torch.tensor instead of numpy.array in this video. Their design (e.g. broadcasting, data types, etc.) is so similar that practicing one is basically practicing the other, just be careful with some of the APIs - how various functions are named, what arguments they take, etc. - these details can vary.
- PyTorch tutorial on Tensor https://pytorch.org/tutorials/beginner/basics/tensorqs_tutorial.html
- Another PyTorch intro to Tensor https://pytorch.org/tutorials/beginner/nlp/pytorch_tutorial.html

Exercises:
E01: train a trigram language model, i.e. take two characters as an input to predict the 3rd one. Feel free to use either counting or a neural net. Evaluate the loss; Did it improve over a bigram model?
E02: split up the dataset randomly into 80% train set, 10% dev set, 10% test set. Train the bigram and trigram models only on the training set. Evaluate them on dev and test splits. What can you see?
E03: use the dev set to tune the strength of smoothing (or regularization) for the trigram model - i.e. try many possibilities and see which one works best based on the dev set loss. What patterns can you see in the train and dev set loss as you tune this strength? Take the best setting of the smoothing and evaluate on the test set once and at the end. How good of a loss do you achieve?
E04: we saw that our 1-hot vectors merely select a row of W, so producing these vectors explicitly feels wasteful. Can you delete our use of F.one_hot in favor of simply indexing into rows of W?
E05: look up and use F.cross_entropy instead. You should achieve the same result. Can you think of why we'd prefer to use F.cross_entropy instead?
E06: meta-exercise! Think of a fun/interesting exercise and complete it.

Chapters:
00:00:00 intro
00:03:03 reading and exploring the dataset
00:06:24 exploring the bigrams in the dataset
00:09:24 counting bigrams in a python dictionary
00:12:45 counting bigrams in a 2D torch tensor ("training the model")
00:18:19 visualizing the bigram tensor
00:20:54 deleting spurious (S) and (E) tokens in favor of a single . token
00:24:02 sampling from the model
00:36:17 efficiency! vectorized normalization of the rows, tensor broadcasting 
00:50:14 loss function (the negative log likelihood of the data under our model)
01:00:50 model smoothing with fake counts
01:02:57 PART 2: the neural network approach: intro
01:05:26 creating the bigram dataset for the neural net
01:10:01 feeding integers into neural nets? one-hot encodings
01:13:53 the "neural net": one linear layer of neurons implemented with matrix multiplication
01:18:46 transforming neural net outputs into probabilities: the softmax
01:26:17 summary, preview to next steps, reference to micrograd
01:35:49 vectorized loss
01:38:36 backward and update, in PyTorch
01:42:55 putting everything together
01:47:49 note 1: one-hot encoding really just selects a row of the next Linear layer's weight matrix
01:50:18 note 2: model smoothing as regularization loss
01:54:31 sampling from the neural net
01:56:16 conclusion
 Hi everyone, hope you're well. And next up what I'd like to do is I'd like to build out Makemore. Like micrograd before it, Makemore is a repository that I have on my GitHub web page. You can look at it. But just like with micrograd, I'm going to build it out step by step and I'm going to spell everything out. So we're going to build it out slowly and together. Now, what is Makemore? Makemore, as the name suggests, makes more of things that you give it. So here's an example. Names.TXT is an example dataset to make more. And when you look at names.TXT, you'll find that it's a very large dataset of names. So here's lots of different types of names. In fact, I believe there are 32,000 names that I've sort of found randomly on the government website. And if you trade Makemore on this dataset, it will learn to make more of things like this. And in particular, in this case, that will mean more things that sound name-like, but are actually unique names. And maybe if you have a baby and you're trying to assign a name, maybe you're looking for a cool new sounding unique name, Makemore might help you. So here are some examples of generations from the neural network. Once we train it on our dataset. So here's some examples of unique names that it will generate. Don't tell, I rot, Zendi, and so on. And so all these sort of sound name-like, but they're not, of course, names. So under the hood, Makemore is a character-level language model. So what that means is that it is treating every single line here as an example. And within each example, it's treating them all as sequences of individual characters. So R-E-E-S-E is this example. And that's the sequence of characters. And that's the level on which we are building out Makemore. And what it means to be a character-level language model then is that it's just sort of modeling those sequences of characters and it knows how to predict the next character in the sequence. Now, we're actually going to implement a large number of character-level language models in terms of the neural networks that are involved in predicting the next character in a sequence. So very simple, bi-gram and bag of word models, multilaylor perceptrons, recurring neural networks, all the way to modern transformers. In fact, a transformer that we will build will be basically the equivalent transformer to GPT2 if you have heard of GPT. So that's kind of a big deal. It's a modern network and by the end of the series, you will actually understand how that works on the level of characters. Now, to give you a sense of the extensions here, after characters, we will probably spend some time on the word level so that we can generate documents of words, not just little segments of characters, but we can generate entire much larger documents. And then we're probably going to go into images and image text networks, such as Dali, stable diffusion, and so on. But for now, we have to start here, careful level language modeling. Let's go. So like before, we are starting with a completely blank GPNodebit page. The first thing is I would like to basically load up the dataset, names.txt. So we're going to open up names.txt for reading. And we're going to read in everything into a massive string. And then because it's a massive string, we'd only like the individual words and put them in the list. So let's call split lines on that string to get all of our words as a Python list of strings. So basically we can look at, for example, the first 10 words. And we have that it's a list of Emma, Olivia, Eva, and so on. And if we look at the top of the page here, that is indeed what we see. So that's good. This list actually makes me feel that this is probably sorted by frequency. But okay, so these are the words. Now we'd like to actually learn a little bit more about this dataset. Let's look at the total number of words. We expect this to be roughly 32,000. And then what is the, for example, shortest word. So min of length of each word for W in words. So the shortest word will be length 2. And max of length W for W in words. So the longest word will be 15 characters. So let's now think through our very first language model. As I mentioned, a character level and good model is predicting the next character in a sequence given already some concrete sequence of characters before it. Now what we have to realize here is that every single word here, like Isabella, is actually quite a few examples packed in to that single word. Because what is an existence of a word like Isabella and the dataset telling us really? It's saying that the character I is a very likely character to come first in a sequence of a name. The character S is likely to come after I. The character A is likely to come after I.S. The character B is very likely to come after I.S.A. And someone all the way to A following Isabella. And then there's one more example actually packed in here. And that is that after there's Isabella, the word is very likely to end. So that's one more sort of explicit piece of information that we have here, that we have to be careful with. And so there's a lot packed into a single individual word in terms of the statistical structure of what's likely to follow in these character sequences. And then of course we don't have just an individual word. We actually have 32,000 of these. And so there's a lot of structure here to model. Now in the beginning, what I'd like to start with is I'd like to start with building a program language model. Now in a by-gram language model, we're always working with just two characters at a time. So we're only looking at one character that we are given and we're trying to predict the next character in the sequence. So what characters are likely to follow are what characters are likely to follow. Hey, and so on. And we're just modeling that kind of a little local structure. And we're forgetting the fact that we may have a lot more information. We're always just looking at the previous character to predict the next one. So it's a very simple and weak language model, but I think it's a great place to start. So now let's begin by looking at these by-grams in our data set and what they look like. And these by-grams again are just two characters in a row. So for WNWords, HW here is an individual word string. We want to iterate for, we want to iterate this word with consecutive characters. So two characters at a time, sliding it through the word. Now, a interesting nice way, cute way to this in Python, by the way, is doing something like this. For character on character two, in zip of W and W at one. One call. Print, character on character two. And let's not do all the words. Let's just do the first three words. And I'm going to show you in a second how this works. But from now, basically as an example, let's just do the very first word alone. You see how we have a M up. And this will just print EM, M-M, M-A. And the reason this works is because W is the string M up. W at one column is the string M-A. And zip takes two iterators. And it pairs them up and then creates an iterator over the tuples of their consecutive entries. And if any one of these lists is shorter than the other, then it will just halt and return. So basically that's why we return EM, M-M, M-M, M-A. But then because this iterator's second one here runs out of elements, zip just ends. And that's why we only get these tuples. So pretty cute. So these are the consecutive elements in the first word. Now we have to be careful because we actually have more information here than just these three examples. As I mentioned, we know that E is very likely to come first. And we know that A in this case is coming last. So one way to do this is basically we're going to create a special array here, our characters. And we're going to hallucinate a special start token here. I'm going to call it like special start. So this is a list of one element. And plus W. And then plus a special end character. And the reason I'm wrapping a list of W here is because W is a string, M-A, list of W will just have the individual characters in the list. And then doing this again now, but not iterating over W's, but over the characters. We'll give us something like this. So E is likely, so this is a bygram of the start character and E. And this is a bygram of the A in the special end character. And now we can look at, for example, what this looks like for Olivia or Eva. And indeed, we can actually, especially this for the entire dataset. But we won't print that. That's going to be too much. But these are the individual character bygrams and we can print them. Now, in order to learn the statistics about which characters are likely to follow other characters, the simplest way in the bygram language models is to simply do it by counting. So we're basically just going to count how often any one of these combinations occurs in the training set in these words. So we're going to need some kind of a dictionary that's going to maintain some counts for every one of these bygrams. So let's use a dictionary B. And this will map these bygrams. So bygram is a tuple of character on character two. And then B at bygram will be B dot get of bygram, which is basically the same as B at bygram. But in the case that bygram is not in the dictionary B, we would like to buy default or term zero plus one. So this will basically add up all the bygrams and count how often they occur. Let's get rid of printing or rather let's keep the printing and let's just inspect what B is in this case. And we see that many bygrams occur just a single time. This one allegedly occluder three times. So A was an ending character three times. And that's true for all of these words. All of Emma, Olivia and Eva and with A. So that's why this occurred three times. Now let's do it for all the words. Oops, I should not have printed it. I'm going to erase that. Let's kill this. Let's just run and now B will have the statistics of the entire data set. So these are the counts across all the words of the individual bygrams. And we could, for example, look at some of the most common ones and least common ones. This kind of grows in Python, but the way to do this, the simplest way I like is we just use B dot items. B dot items returns the tuples of key value. In this case, the keys are the character bygrams and the values are the counts. And so then what we want to do is we want to do sort it of this. But by default, sort is on the first on the first item of a tuple. But we want to sort by the values, which are the second element of a tuple that is the key value. So we want to use the key equals lambda that takes the key value and returns the key value at one, not at zero, but at one, which is the count. So we want to sort by the count of these elements. And actually, we wanted to go backwards. So here what we have is the bygram q and r occurs only a single time, d z occurred only a single time. And when we sort this the other way around, we're going to see the most likely bygrams. So we see that n was very often an ending character many, many times. And apparently n always always follows an a and that's a very likely combination as well. So this is kind of the individual counts that we achieve over the entire data set. Now it's actually going to be significantly more convenient for us to keep this information in a two dimensional array instead of a high-thond dictionary. So we're going to store this information in a 2D array and the rows are going to be the first character of the bygram and the columns are going to be the second character. And each entry in the two dimensional array will tell us how often that first character follows the second character in the data set. So in particular, the array representation that we're going to use or the library is that of PyTorch. And PyTorch is a deep learning neural framework, but part of it is also this torch.tensor which allows us to create multi-dimensional arrays and manipulate them very efficiently. So let's import PyTorch, which you can do by import Torch. And then we can create a race. So let's create a array of zeros. And we give it a size of this array. Let's create a 3 by 5 array as an example. And this is a 3 by 5 array of zeros. And by default, you'll notice a dot d type, which is short for data type, is flow 32. So these are single precision floating point numbers. Because we are going to represent counts, let's actually use d type as Torch.t in 32. So these are 32 bit integers. So now you see that we have integer data inside this tensor. Now, tensors allow us to really manipulate all the individual entries and do it very efficiently. So for example, if we want to change this bit, we have to index into the tensor. And in particular, here, this is the first row. And the, because it's zero indexed. So this is row index one and column index zero, one, two, three. So a at one comma three, we can set that to one. And then a will have a one over there. We can of course also do things like this. So now a will be to over there. And also we can, for example, say a zero zero is five. And then a will have a five over here. So that's how we can index into the arrays. Now, of course, the array that we are interested in is much much bigger. So for our purposes, we have 26 letters of the alphabet. And then we have two special characters as and e. So we want 26 plus two or 28 by 28 array. And let's call it the capital N because it's going to represent the sort of the counts. Let me raise this stuff. So that's the array that starts at zeroes, 28 by 28. And now let's copy paste this here. But instead of having an dictionary B, which we're going to erase, we now have an N. Now the problem here is that we have these characters, which are strings, but we have to now basically index into a array. And we have to index using integers. So we need some kind of a lockup table from characters to integers. So let's construct such a character array. And the way we're going to do this is we're going to take all the words, which is a list of strings. We're going to concatenate all of it into a massive string. So this is just simply the entire data set as a single string. We're going to pass this to the set constructor, which takes this massive string and throws out duplicates because sets do not allow duplicates. So set of this will just be the set of all the lowercase characters. And there should be a total of 26 of them. And now we actually don't want a set. We want a list. But we don't want a list sorted in some weird arbitrary way. We wanted to be sorted from a to Z. So sorted list. So those are our characters. Now we want is this lookup table, as I mentioned. So let's create a special S to I. I will call it. S is string or character. And this will be an S to I mapping for I S in a numerate of these characters. So enumerate basically gives us this iterator over the integer index and the actual element of the list. And then we are mapping the character to the integer. So S to I is a mapping from a to zero, B to one, etc. All the way from Z to 25. And that's going to be useful here. But we actually also have to specifically set that S will be 26. And S to I at E will be 27. Right? Because Z was 25. So those are the lookups. And now we can come here and we can map both character one and character to to their integers. So this will be S to I at character one. And I X to will be S to I of character two. And now we should be able to do this line, but using our array. So in it, I X one, I X to this is the two dimensional array indexing. I showed you before. And honestly, just plus equals one because everything starts at zero. So this should work and give us a large 28 by 20 array of all these counts. So if we print in this is the array, but of course it looks ugly. So let's erase this ugly mess. And let's try to visualize it a bit more nicer. So for that, we're going to use a library called matplotlib. So matplotlib allows us to create figures. So we can do things like PILT I'm show off the counter ray. So this is the 20 by 28 array. And this is structure, but even this, I would say is still pretty ugly. So we're going to try to create a much nicer visualization of it. And I wrote a bunch of code for that. The first thing we're going to need is we're going to need to invert this array here, this dictionary. So S to I is mapping from S to I. And in I to S, we're going to reverse this dictionary. So it rate of all the items and just reverse that array. So I to S maps inversely from zero to a want to be, etc. So we'll need that. And then here's the code that I came up with to try to make this a little bit nicer. We create a figure we plot and then we do and then we visualize a bunch of things here. Let me just run it so you get a sense of what it is. So you see here that we have the array spaced out and every one of these is basically like B follows G zero times B follows H 41 times. So a follows J 175 times. And so what you can see that I'm doing here is first I show that entire array. And then I iterate over all the individual little cells here. And I create a character string here, which is the inverse mapping I to S of the integer I and the integer J. So that's the diagrams in a character representation. And then I plot just the diagram text and then I plot the number of times that this diagram occurs. Now the reason that there's a dot item here is because when you index into these arrays, these are torch tensors. You see that we still get a tensor back. So the type of this thing you think it would be just an integer long 49 but it's actually a torched up tensor. And so if you do dot item, then it will pop out that individual integer. So it will just be 149. So that's what's happening there. And these are just some options to make it look nice. So what is this structure of this array? We have all these counts and we see that some of them occur often and some of them do not occur often. Now if you scrutinize this carefully, you will notice that we're not actually being very clever. That's because when you come over here, you'll notice that for example, we have an entire row of completely zeroes. And that's because the end character is never possibly going to be the first character of a diagram because we're always placing these end tokens all at the end of the diagram. Similarly, we have entire column zeros here because the S character will never possibly be the second element of a diagram because we always start with S and we end with E and we only have the words in between. So we have an entire column of zeros and entire row of zeros. And in this little two by two matrix here as well, the only one that can possibly happen is if S directly follows E. That can be non-zero if we have a word that has no letters. So in that case, there's no letters in a word. It's an empty word and we just have S follows E. But the other ones are just not possible. And so we're basically wasting space and not only that, but the S and the E are getting very crowded here. I was using these brackets because there's convention and natural language processing to use these kinds of brackets to denote special tokens. But we're going to use something else. So let's fix all this and make it prettier. We're not actually going to have two special tokens. We're only going to have one special token. So we're going to have n by n array of 27 by set 27 instead. Instead of having two, we will just have one and I will call it a dot. Okay. Let me swing this over here. Now, one more thing that I would like to do is I would actually like to make this special character half position zero. And I would like to offset all the other letters off. I find that a little bit more pleasing. So we need a plus one here so that the first character, which is A, will start at one. So S to I will now be a starts at one and dot is zero. And I to us, of course, we're not changing this because I to us just creates reverse mapping and this will work fine. So one is a to us B zero is dot. So we reverse that here. We have a dot and a dot. This should work fine. Make sure I started zeros count. And then here we don't go up to 28. We go up to 27. And this should just work. Okay. So we see that dot dot never happened. It's at zero because we don't have empty words. Then this row here now is just very simply the counts for all the first letters. So G J starts a word H starts word I starts a word, etc. And then these are all the ending characters. And in between we have the structure of what characters follow each other. So this is the counts array of our entire data set. So this array actually has all the information necessary for us to actually sample from this by gram character level language model. And roughly speaking we're going to do is we're just going to start following these probabilities and these counts. And we're going to start sampling from the model. So in the beginning, of course, we start with the dot, the start token dot. So to sample the first character of a name, we're looking at this row here. So we see that we have the counts and those counts externally are telling us how often any one of these characters is to start a word. So if we take this n and we grab the first row, we can do that by using just indexing at zero and then using this notation colon for the rest of that row. So n zero colon is indexing into the zero row and then grabbing all the columns. And so this will give us a one dimensional array of the first row. So zero for four ten, you know, zero for four ten, one three oh six one five four two, etc. It's just the first row. The shape of this is 27, just the row of 27. And the other way that you can do this also is you just you don't actually give this you just grab the zero row like this. This is equivalent. Now these are the counts and now what we'd like to do is we'd like to basically sample from this. Since these are the raw counts, we actually have to convert this to probabilities. So we create a probability vector. So we'll take n of zero and we'll actually convert this to float first. Okay, so these integers are converted to float, float, equine numbers. And the reason we're creating floats is because we're about to normalize these counts. So to create a probability distribution here, we want to divide, we basically want to the peak, p divide p dot sum. And now we get a vector of smaller numbers and these are now probabilities. So of course, because we divided by the sum, the sum of p now is one. So this is a nice proper probability distribution. It sums to one and this is giving us the probability for any single character to be the first character of a word. So now we can try to sample from this distribution to sample from these distributions. We're going to use torshtut multinomial, which I've pulled up here. So torshtut multinomial returns samples from the multinomial probability distribution, which is a complicated way of saying, you give me probabilities and I will give you integers, which are sampled according to the probability distribution. So this is the signature of the method. And to make everything deterministic, we're going to use a generator object in PyTorch. So this makes everything deterministic. So when you run this on your computer, you're going to get the exact same results that I'm getting here on my computer. So let me show you how this works. Here's the deterministic way of creating a torch generator object, seeding it with some number that we can agree on. So that seeds a generator, gets us an object g. And then we can pass that g to a function that creates here random numbers, torshtut random, creates random numbers, three of them. And it's using this generator object to, as a source of randomness. So without normalizing it, I can just print. This is sort of like numbers between zero and one that are random according to this thing. And whenever I run it again, I'm always going to get the same result because I keep using the same generator object, which I'm seeding here. And then if I divide to normalize, I'm going to get a nice probability distribution of just three elements. And then we can use torshtut multinomial to draw samples from it. So this is what that looks like. So torshtut multinomial will take the torshtensor of probability distributions. Then we can ask for a number of samples like C20. Replacement equals true means that when we draw an element, we can draw it, and then we can put it back into the list of eligible indices to draw again. And we have to specify replacement as true because by default, for some reason, it's false. So I think it's just something to be careful with. And the generator is passed in here. So we are going to always get deterministic results, the same results. So if I run these two, we're going to get a bunch of samples from this distribution. Now you'll notice here that the probability for the first element in this tensor is 60%. So in these 20 samples, we'd expect 60% of them to be zero. We'd expect 30% of them to be one. And because the element index 2 has only 10% probability, very few of these samples should be two. And indeed, we only have a small number of twos. And we can sample as many as we would like. And the more we sample, the more these numbers should, roughly, have the distribution here. So we should have lots of zeros, half as many ones. And we should have three times s few, sorry, s few ones, and three times s few twos. So you see that we have very few twos. We have some ones and most of them are zero. So that's what torsion multilimals doing. For us here, we are interested in this row. We've created this p here. And now we can sample from it. So if we use the same seed, and then we sample from this distribution, let's just get one sample. Then we see that the sample is, say, 13. So this will be the index. And let's, you see how it's a tensor that wraps 13. We again have to use dot item to pop out that integer. And now index would be just number 13. And of course, we can map the I2S of IX to figure out exactly which character we're sampling here. We're sampling M. So we're saying that the first character is in our generation. And just look at the row here. M was drawn, and we can see that M actually starts a large number of words. M started 2,500 words out of 32,000 words. So almost a bit less than 10% of the words start with M. So this was actually fairly likely character to draw. So that would be the first character of our word. And now we can continue to sample more characters. Because now we know that M started. Now M is already sampled. So now to draw the next character, we will come back here. And we will work for the row that starts with M. So you see M and we have a row here. So we see that M dot is 516, M A is this many, M B is this many, etc. So these are the counts for the next row. And that's the next character that we are going to now generate. So I think we are ready to actually just write out a loop. So we are going to start to get a sense of how this is going to go. We always begin at index 0, because that's the start token. And then while true, we are going to grab the row corresponding to index that we are currently on. So that's n array at ix. Converted to float is rp. Then we normalize this p to sum to 1. I accidentally ran the infinite loop. We normalize p to sum to 1. Then we need this generator object. And we are going to initialize up here. And we are going to draw a single sample from this distribution. And then this is going to tell us what index is going to be next. If the index sampled is 0, then that's now the end token. So we will break. Otherwise, we are going to print s2i of ix. i2s of ix. And that's pretty much it. We're just... this should work. Okay, more. So that's the name that we've sampled. We started with m. The next step was o, then r, and then dot. And this dot, we printed here as well. So let's not do this a few times. So let's actually create an out list here. And instead of printing, we're going to append. So out that append this character. And then here, let's just print it at the end. So let's just join up all the outs. And we're just going to print more. Now we're always getting the same result because of the generator. So who want to do this a few times? We can go for i and range 10. We can sample 10 names. And we can just do that 10 times. And these are the names that we're getting out. 10, 20. I'll be honest with you, this doesn't look right. So I've started a few minutes to convince myself that it actually is right. The reason these samples are so terrible is that by-gram language model is actually just like really terrible. We can generate a few more here. And you can see that they're kind of like their name, like a little bit, like yanu, irailly, etc. But they're just like totally messed up. And I mean the reason that this is so bad, like we're generating h as a name. But you have to think through it from the model's eyes. It doesn't know that this h is the very first h. All it knows is that h was previously. And now how likely is h the last character? Well, it's somewhat likely. And so it just makes it last character. It doesn't know that there were other things before it or there were not other things before it. And so that's why I'm generating all these like nonsense names. And the other way to do this is to convince yourself that it is actually doing something reasonable, even though it's so terrible, is these little piece here are 27, right? Like 27. So how about if we did something like this? Instead of having any structure whatsoever. How about if p was just a torch dot ones of 27? By default, this is a float 32. So this is fine. Divide 27. So what I'm doing here is this is the uniform distribution, which will make everything equally likely. And we can sample from that. So let's see if that doesn't need better. Okay. So it's this is what you have from a model that is completely untrained where everything is equally likely. So it's obviously garbage. And then if we have a trained model, which is trained on just by grams, this is what we get. So you can see that it is more name like it is actually working. It's just by gram is so terrible and we have to do better. Now next I would like to fix an inefficiency that we have going on here. Because what we're doing here is we're always fetching a row of n from the counts matrix up ahead. And we're always doing the same things. We're converting to float and we're dividing. And we're doing this every single iteration of the slope. And we just keep normalizing these rows over and over again. And it's extremely inefficient and wasteful. So what I'd like to do is I'd like to actually prepare a matrix capital p that will just have the probabilities in it. So in other words, it's going to be the same as the capital n matrix here of counts. But every single row will have the row of probabilities that is normalized to one, indicating the probability distribution for the next character given the character before it. As defined by which row we're in. So basically what we'd like to do is we'd like to just do it up front here. And then we would like to just use that row here. So here we would like to just do p equals p of ix instead. Okay. The other reason I want to do this is not just for efficiency, but also I would like us to practice these and dimensional tensors. And I'd like us to practice their manipulation. And especially something that's called broadcasting that we'll go into in a second. We're actually going to have to become very good at these tensor manipulations because we're going to build out all the way to transformers. We're going to be doing some pretty complicated array operations for efficiency. And we need to really understand that and be very good at it. So in doing what we want to do is we first want to grab the floating point copy of n. And I'm mimicking the line here basically. And then we want to divide all the rows so that they sum to one. So we'd like to do something like this. p divide p dot sum. But now we have to be careful because p dot sum actually produces a sum. Sorry. P equals n dot float copy. p dot sum produces a sums up all of the counts of this entire matrix n. And gives us a single number of just the summation of everything. So that's not the way we want to divide. We want to simultaneously and in parallel divide all the rows by their respective sums. So what we have to do now is we have to go into documentation for tors.sum. And we can scroll down here to a definition that is relevant to us, which is where we don't only provide an input array that we want to sum. But we also provide the dimension along which we want to sum. And in particular, we want to sum up over rows. Now one more argument that I want you to pay attention to here is the keep them as false. If keep them is true and the output tensor is of the same size as input, except of course the dimension along which you summed, which will become just one. But if you pass in, keep them as false, then this dimension is squeezed out. And so tors.sum not only does the sum and collapses dimension to be of size one, but in addition, it does what's called a squeeze where it squeezes out, it squeezes out that dimension. So basically what we want here is we instead want to do p dot sum of sum axis. And in particular, notice that p dot shape is 27 by 27. So when we sum up across axis zero, then we would be taking the zero dimension and we would be summing across it. So when keep them as true, then this thing will not only give us the counts across along the columns, but notice that basically the shape of this is one by 27. We just get a row vector and the reason we get a row vector here again is because we pass in zero dimension. So this zero dimension becomes one and we've done a sum and we get a row. And so basically we've done the sum this way vertically and arrived at just a single one by 27 vector of counts. What happens when you take out keep them is that we just get 27. So it squeezes out that dimension and we just get one dimensional vector of size 27. Now we don't actually want one by 27 row vector because that gives us the counts or the sums across the columns. We actually want to sum the other way along dimension one and you'll see that the shape of this is 27 by 1. So it's a column vector. It's a 27 by 1 vector of counts. And that's because what's happened here is that we're going horizontally and this 27 by 27 matrix becomes a 27 by 1 array. Now you'll notice by the way that the actual numbers of these counts are identical. And that's because this special array of counts here comes from by grams to the sticks and actually it just so happens by chance. Or because of the way this array is constructed that the sums along the columns or along the rows horizontally or vertically is identical. But actually what we want to do in this case is we want to sum across the rows horizontally. So what we want here is be that some of one with keep them true 27 by 1 column vector. And now what we want to do is we want to divide by that. Now we have to be careful here again. Is it possible to take what's a p. shape you see here is 27 by 27. Is it possible to take a 27 by 27 array and divide it by what is a 27 by 1 array? Is that an operation that you can do? And whether or not you can perform this operation is determined by what's called broadcasting rules. So if you just search broadcasting semantics in torch, you'll notice that there's a special definition for what's called broadcasting that for whether or not these two arrays can be combined in a binary operation like division. So the first condition is each tensor has at least one dimension, which is the case for us. And then when iterating over the dimension sizes starting at the trailing dimension, the dimension sizes must either be equal, one of them is one or one of them does not exist. So let's do that. We need to align the two arrays and their shapes, which is very easy because both of these shapes have two elements. So they're aligned. Then we iterate over from the right and going to the left. Each dimension must be either equal, one of them is a one or one of them does not exist. So in this case, they're not equal, but one of them is a one. So this is fine. And then this dimension, they're both equal. So this is fine. So all the dimensions are fine. And therefore the this operation is broadcastable. So that means that this operation is allowed. And what is it that these arrays do when you divide 27 by 27 by 27 by one? What it does is that it takes this dimension one and it stretches it out. It copies it to match 27 here. In this case. So in our case, it takes this column vector, which is 27 by one. And it copies it 27 times to make these both be 27 by 27 internally. You can think of it that way. And so it copies those counts. And then it does an element wise division, which is what we want. Because these counts we want to divide by them on every single one of these columns in this matrix. So this actually we expect will normalize every single row. And we can check that this is true by taking the first row, for example, and taking it some. We expect this to be one because it's not normalized. And then we expect this now, because if we actually correctly normalize all the rows, we expect to get the exact same result here. So let's run this. It's the exact same result. So this is correct. So now I would like to scare you a little bit. You actually have to like basically encourage you very strongly to read through broadcasting semantics. And I encourage you to treat this with respect. And it's not something to play fast and smooth. It's something to really respect, really understand and look up maybe some tutorials for broadcasting and practice it and be careful with it because you can very quickly run into bugs. Let me show you what I mean. You see how here we have p.1. Keep them this true. The shape of this is 27 by 1. Let me take out this line just so we have the n and then we can see the counts. We can see that this is all the counts across all the rows. And it's 27 by 1 column vector. Now suppose that I tried to do the following, but I erased keep them this true here. What does that do? If keep them is not true, it's false. Then remember according to documentation, it gets rid of this dimension 1. It squeezes it out. So basically we just get all the same counts, the same result except the shape of it is not 27 by 1. It is just 27 by 1 disappears. But all the counts are the same. So you'd think that this divide that would work. First of all, can we even write this? Is it even expected to run? Is it broadcastable? Let's determine if this result is broadcastable. p.1 is shape. It's 27. This is 27 by 27. So 27 by 27 broadcasting into 27. So now suppose broadcasting number 1 align all the dimensions on the right done. Now iteration over all the dimensions started from the right going to the left. All the dimensions must either be equal. One of them must be 1 or one then does not exist. So here they are or equal. Here the dimension does not exist. So internally what broadcasting will do is it will create a 1 here and then we see that one of them is a 1 and this will get copied and this will run. This will broadcast. Okay, so you'd expect this to work because we are this broadcast and this we can divide this. Now if I run this, you'd expect it to work but it doesn't. Now you actually get garbage. You got a wrong result because this is actually a bug. This keeps them equal to true. Makes it work. This is a bug. In both cases we are doing the correct counts. We are summing up across the rows but keep them saving us and making it work. So in this case, I'd like you to encourage you to potentially like pause this video at this point and try to think about why this is buggy and why the keep them was necessary here. Okay, so the reason to do for this is I'm trying to hint it here when I was giving you a bit of a hint on how this works. This 27 vector internally inside the broadcasting, this becomes a 1 by 27 and 1 by 27 is a row vector. Right? And now we are dividing 27 by 27 by 1 by 27 and torch will replicate this dimension. So basically it will take this row vector and it will copy it vertically now 27 times so the 27 by 27 lies exactly and element wise divides. And so basically what's happening here is we're actually normalizing the columns instead of normalizing the rows. So you can check what's happening here is that p at 0 which is the first row of p dot sum is not 1 it's 7. It is the first column as an example that sums to 1. So to summarize where does the issue come from? The issue comes from the silent adding of the dimension here because in broadcasting rules you align on the right and go from right to left and if the dimension doesn't exist you create it. So that's where the problem happens. We still did the counts correctly. We did the counts across the rows and we got the counts on the right here as a column vector. But because the key things was true this dimension was discarded and now we just have a vector 27. And because of broadcasting the way it works this vector of 27 suddenly becomes a row vector. And then this row vector gets replicated vertically and that every single point we are dividing by the count in the opposite direction. So this thing just doesn't work. This needs to be keep them as true in this case. So then then we have that p at 0 is normalized. And conversely the first column you'd expect to potentially not be normalized. And this is what makes it work. So pretty subtle and hopefully this helps to scare you that you should have respect for broadcasting. Be careful. Check your work and understand how it works under the hood and make sure that it's broadcasting in the direction that you like. Otherwise you're going to introduce very subtle bugs, very hard to find bugs. And just be careful. One more note on efficiency. We don't want to be doing this here because this creates a completely new tensor that we store into p. We prefer to use in place operations if possible. So this would be an in place operation has the potential to be faster. It doesn't create new memory under the hood. And then let's erase this. We don't need it. And let's also just do fewer. Just so I'm not wasting space. Okay. So we're actually in the pre-good spot now. We trained a bi-gram language model and we trained it really just by counting how frequently any pairing occurs and then normalizing so that we get a nice property distribution. So really these elements of this array p are really the parameters of our bi-gram language model given us and summarizing the statistics of these diagrams. So we trained a model and then we know how to sample from a model. We just iteratively sampled the next character and feed it in each time and get a next character. Now what I'd like to do is I'd like to somehow evaluate the quality of this model. We'd like to somehow summarize the quality of this model into a single number. How good is it at predicting the training set? And as an example, so in the training set, we can evaluate now the training loss and this training loss is telling us about sort of the quality of this model in a single number just like we saw in micrograd. So let's try to think through the quality of the model and how we would evaluate it. Basically what we're going to do is we're going to copy paste this code that we previously used for counting. Okay. And let me just print these bi-grams first. We're going to use f strings and I'm going to print character one followed by character two. These are the bi-grams. And then I didn't want to do it for all the words. Let's just do first three words. So here we have Emma, Olivia and Eva bi-grams. Now what we'd like to do is we'd like to basically look at the probability that the model assigns to every one of these bi-grams. So in other words, we can look at the probability, which is summarized in the matrix B of Ix1, Ix2. And then we can print it here as probability. And because these probabilities are way too large, let me percent or call on.4F to like truncated a bit. So what do we have here? We're looking at the probabilities that the model assigns to every one of these bi-grams in the dataset. And so we can see some of them are 4% 3% etc. Just to have a measuring stick in our mind, by the way, we have 27 possible characters or tokens. And if everything was equally likely, then you'd expect all these probabilities to be 4% roughly. So anything above 4% means that we've learned something useful from these bi-grams statistics. And you see that roughly some of these are 4%, but some of them are as high as 40%, 35%. And so on. So you see that the model actually assigned a pretty high probability to whatever's in the training set. And so that's a good thing. Basically, if you have a very good model, you'd expect that these probabilities should be near one, because that means that your model is correctly predicting what's going to come next, especially in the training set where you trained your model. So now we'd like to think about how can we summarize these probabilities into a single number that measures the quality of this model. Now when you look at the literature into maximum likelihood estimation and statistical modeling and so on, you'll see that what's typically used here is something called the likelihood. And the likelihood is the product of all of these probabilities. And so the product of all of these probabilities is the likelihood. And it's really telling us that the probability of the entire dataset assigned by the model that we've trained. And that is a measure of quality. So the product of these should be as high as possible when you are training the model and when you have a good model, your product of these probabilities should be very high. Now because the product of these probabilities and is an unwieldy thing to work with, you can see that all of them are between 0 and 1. So your product of these probabilities will be a very tiny number. So for convenience, what people work with usually is not the likelihood, but they work with what's called the log likelihood. So the product of these is the likelihood to get the log likelihood. We just have to take the log of the probability. And so the log of the probability here, I have the log of x from 0 to 1. The log is a, you see here monotonic transformation of the probability. Where if you pass in 1, you get 0. So probability 1 gets your log probability of 0. And then as you go lower and lower probability, the log will grow more and more negative until all the way to negative infinity at 0. So here we have a log prob, which is religious, the torshtod log of probability. Let's print it out to get a sense of what that looks like. Log prob also point for f. So as you can see, when we plug in numbers that are very close, some of our higher numbers, we get closer and closer to 0. And then if we plug in very bad probabilities, we get more and more negative number. That's bad. And the reason we work with this is for large extent convenience, right, because we have mathematically that if you have some product, a times b times c, of all these probabilities, right, the likelihood is the product of all these probabilities. Then the log of these is just log of a plus log of b plus log of c. If you remember your logs from your high school or undergrad and so on. So we have that basically the likelihood of the product probabilities, the what likelihood is just the sum of the logs of the individual probabilities. So log likelihood starts at 0 and then log likelihood here, we can just accumulate simply. And then the end we can print this print the log likelihood, f strings, maybe you're familiar with this. So what likelihood is negative 38. Now we actually want. So how high can log likelihood get? It can go to zero. So when all the probabilities are one log likelihood to zero. And then when all the probabilities are lower, this will grow more and more negative. Now we don't actually like this because what we'd like is a loss function and a loss function has the semantics that low is good because we're trying to minimize the loss. So we actually need to invert this and that's what gives us something called the negative log likelihood. Negative log likelihood is just negative of the log likelihood. These are f strings, by the way, if you'd like to look this up, negative log likelihood equals. So the negative log likelihood now is just negative of it. And so the negative log likelihood is a very nice loss function because the lowest it can get is zero. And the higher it is, the worse off the predictions are that you're making. And then one more modification to this that sometimes people do is that for convenience, they actually like to normalize by they like to make it an average instead of a sum. And so here, let's just keep some counts as well. So n plus equals one starts at zero. And then here we can have sort of like a normalized log likelihood. If we just normalize it by the count, then we will sort of get the average log likelihood. So this would be usually our loss function here is this we would this is what we would use. So our loss function for the training set assigned by the model is 2.4. That's the quality of this model. And the lower it is, the better off we are. And the higher it is, the worse off we are. And the job of our training is to find the parameters that minimize the negative log likelihood loss. And that would be like a high quality model. Okay, so to summarize, I actually wrote it out here. So our goal is to maximize likelihood, which is the product of all the probabilities assigned by the model. And we want to maximize this likelihood with respect to the model parameters. And in our case, the model parameters here are defined in the table. These numbers, the probabilities are the model parameters, sort of in our background language model so far. But you have to keep in mind that here we are storing everything in a table format, the probabilities. But what's coming up as a brief preview is that these numbers will not be kept explicitly. But these numbers will be calculated by neural network. So that's coming up. And we want to change and tune the parameters of these neural networks. We want to change these parameters to maximize the likelihood, the product of the probabilities. Now, maximizing the likelihood is equivalent to maximizing the log likelihood because log is a monotonic function. Here's the graph of log. And basically, all it is doing is it's just scaling your... You can look at it as just a scaling of the loss function. And so the optimization problem here and here are actually equivalent because this is just a scaling. You can look at it that way. And so these are two identical optimization problems. Maximizing the log likelihood is equivalent to minimizing the negative log likelihood. And then in practice, people actually minimize the average negative log likelihood to get numbers like 2.4. And then this summarizes the quality of your model. And we'd like to minimize it and make it as small as possible. And the lowest it can get is zero. And the lower it is, the better off your model is because it's signing high probabilities to your data. Now let's estimate the probability over the entire training set just to make sure that we get something around 2.4. Let's run this over the entire... Oops! Let's take out the print statement as well. Okay, 2.45 or the entire training set. Now what I'd like to show you is that you can actually evaluate the probability for any word that you want. Like for example, if we just test a single word, Andre, and bring back the print statement, then you see that Andre is actually kind of like an unlikely word. Like on average, we take three log probability to represent it. And roughly that's because EJ apparently is very uncommon as an example. Now, think through this. I'm going to take Andre and I append Q, and I test the probability of it. Andre Q. We actually get infinity. And that's because JQ has a 0% probability according to our model. So the log likelihood... So the log of 0 will be negative infinity. We get infinite loss. So this is kind of undesirable, right? Because we plugged in a string that could be like a somewhat reasonable name. But basically what this is saying is that this model is exactly 0% likely to predict this name. And our loss is infinity on this example. And really, the reason for that is that J is followed by Q 0 times, where is Q? JQ is 0. And so JQ is 0% likely. So this is actually kind of gross and people don't like this too much. To fix this, there's a very simple fix that people like to do to sort of smooth out your model a little bit. And it's called model smoothing. And roughly what's happening is that we will add some eight counts. So imagine adding a count of one to everything. So we add a count of one like this. And then we recalculate the probabilities. And that's model smoothing. And you can add as much as you like. You can add five and that will give you a smoother model. And the more you add here, the more uniform model you're going to have. And the less you add, the more peaked model you're going to have. Of course, so one is like a pretty decent count to add. And that will ensure that there will be no zeros in our probability matrix P. And so this will of course change the generations a little bit. In this case, it didn't but it in principle it could. But what that's going to do now is that nothing will be infinity unlikely. So now our model will predict some other probability. And we see that JQ now has a very small probability. So the model still finds it's very surprising that this was a word or a by gram. But we don't get negative infinity. So it's kind of like a nice fix that people like to apply sometimes and it's called models moving. Okay, so we've now trained a respectable by gram character level language model. And we saw that we both sort of trained the model by looking at the counts of all the by grams. And normalizing the rows to get probability distributions. So we saw that we can also then use those parameters of this model to perform sampling of new words. So we sample new names according to those distributions. And we also saw that we can evaluate the quality of this model. And the quality of this model is summarized in a single number, which is the negative log likelihood. And the lower this number is the better the model is because it is giving high probabilities to the actual next characters in all the by grams in our training set. So that's all well and good. But we've arrived at this model explicitly by doing something that felt sensible. We were just performing counts. And then we were normalizing those counts. Now what I would like to do is I would like to take an alternative approach. We will end up in a very, very similar position, but the approach will look very different. Because I would like to cast the problem of by gram character level language modeling into the neural network framework. And in neural network framework, we're going to approach things slightly differently, but again, end up in a very similar spot. I'll go into that later. Now, our neural network is going to be a still a by gram character level language model. So it receives a single character as an input. Then there's neural network with some weights or some parameters w. And it's going to output the probability distribution over the next character in a sequence. It's going to make guesses as to what is likely to follow this character that was input to the model. And then in addition to that, we're going to be able to evaluate any setting of the parameters of the neural net, because we have a loss function, the negative lot likelihood. So we're going to take a look at its probability distributions, and we're going to use the labels, which are basically just the identity of the next character in that by gram, the second character. So knowing what the second character actually comes next in the by gram allows us to then look at what, how high of probability the model assigns to that character. And then we of course want the probability to be very high. And that is another way of saying that the loss is low. So we're going to use gradient based optimization then to tune the parameters of this network, because we have the loss function, and we're going to minimize it. So we're going to tune the weights so that the neural net is correctly predicting the probabilities for the next character. So let's get started. The first thing I want to do is I want to compile the training set of this neural network. So create the training set of all the by grams. And here I'm going to copy paste this code, because this code iterates over all the by grams. So here we start with the words, we iterate over all the by grams. And previously, as you recall, we did the counts. But now we're not going to do counts. We're just creating a training set. Now this training set will be made up of two lists. We have the inputs and the targets, the labels. And these by grams will denote x, y. Those are the characters, right? And so we're given the first character of the by gram, and then we're trying to predict the next one. Both of these are going to be integers. So here we'll take x's that append is just x1, y's that append, x2. And then here we actually don't want lists of integers. We will create tensors out of these. So x's is torched up tensor of x's, and y's is torched up tensor of y's. And then we don't actually want to take all the words just yet, because I want everything to be manageable. So let's just do the first word, which is m, i. And then it's clear what these x's and y's would be. Here, let me print character one character two. Just so you see what's going on here. So the by grams of these characters is dot e, e m, m, m, a dot. So this single word, as I mentioned, has one, two, three, four, five examples for our neural network. There are five separate examples in m, and those examples I show my is here. When the input to the neural neural network is integer zero, the desired label is integer five, which corresponds to e. When the input to the neural network is five, we want its weights to be arranged so that 13 gets a very high probability. When 13 is put in, we want 13 to have a high probability. When 13 is put in, we also want one to have a high probability. When one is input, we want zero to have a very high probability. So there are five separate input examples to a neural net in this data set. I wanted to add a tangent of a note of caution to be careful with a lot of the APIs of some of these frameworks. You saw me silently use torched dot tensor with a lowercase t, and the output looked right. But you should be aware that there's actually two ways of constructing a tensor. There's a torched dot lowercase tensor, and there's also a torched dot capital tensor class, which you can also construct. So you can actually call both. You can also do torched dot capital tensor, and you get a nexus and wise as well. So that's not confusing at all. There are threads on what is the difference between these two. And unfortunately, the docs are just like not clear on the difference. And when you look at the docs of lowercase tensor, construct tensor with no undergrad history by copying data. It's just like it doesn't, it doesn't make sense. So the actual difference, as far as I can tell, is explained eventually in this random thread that you can google. And really, it comes down to, I believe, that, where is this? Torched dot tensor refers to the d type, the data type automatically, while torched dot tensor just returns a flow tensor. I would recommend to stick to torched dot lowercase tensor. So, indeed, we see that when I construct this with a capital T, the data type here of x is flow 32. But torched dot lowercase tensor, you see how it's now x dot d type is now integer. So, it's advised that you use lowercase t, and you can read more about it if you like in some of these threads. But basically, I'm pointing out some of these things because I want to caution you, and I want you to get used to reading a lot of documentation, and reading through a lot of Q and A's and threads like this. And, you know, some of the stuff is unfortunately not easy and not very well documented, and you have to be careful out there. What we want here is integers, because that's what makes sense. And so, lowercase tensor is what we are using. Okay, now we want to think through how we're going to feed in these examples into a neural network. Now, it's not quite as straightforward as plugging it in, because these examples right now are integers. So, there's like a 0, 5, or 13. It gives us the index of the character, and you can't just plug an integer index into a neural net. These neural nets are sort of made up of these neurons, and these neurons have weights. And, as you saw in micrograd, these weights act multiplicatively on the inputs, wx, plus b, there's 10HS, and so on. And so, it doesn't really make sense to make an input neuron take on integer values that you feed in, and then multiply on with weights. So, instead, a common way of encoding integers is what's called one-hot encoding. In one-hot encoding, we take an integer like 13, and we create a vector that is all zeros, except for the 13th dimension, which we turn to a 1. And then that vector can feed into a neural net. Now, conveniently, PyTorch actually has something called the one-hot function inside Torch and in Functional. It takes a tensor made up of integers. Long is an integer, and it also takes a number of classes, which is how large you want your vector to be. So, here, let's import Torch. and end.functional.sf. This is a common way of importing it. And then let's do f.1 hot, and we feed in the integers that we want to encode. So, we can actually feed in the entire array of x's, and we can tell that num class is 27. So, it doesn't have to try to guess it. It may have guessed that it's only 13, and would give us an incorrect result. So, this is the one-hot. Let's call this x-inc for x-incoded. And then we see that x-incoded.shape is 5x27. And we can also visualize it at plt.imShowofxinc to make it a little bit more clear because this is a little messy. So, we see that we've encoded all the five examples into vectors. We have five examples, so we have five rows, and each row here is now an example into a neural mat. And we see that the appropriate bit is turned on as a one, and everything else is zero. So, here, for example, the zero-th bit is turned on, the fifth bit is turned on, 13th bits are turned on for both of these examples, and then the first bit here is turned on. So, that's how we can encode integers into vectors, and then these vectors can feed in to neural mats. One more issue to be careful with here, by the way, is let's look at the data type of echoing. We always want to be careful with data types. What would you expect x-incodings data type to be? When we're plugging numbers into neural mats, we don't want them to be integers. We want them to be floating point numbers that can take on various values. But the D-type here is actually 64-bit integer. And the reason for that, I suspect, is that one hot received a 64-bit integer here, and it returned to the same data type. And when you look at the signature of one hot, it doesn't even take a desired data type of the output tensor. And so, we can't, in a lot of functions in tortuary, be able to do something like D-type equals torshtotflot32, which is what we want, but one hot does not support that. So instead, we're going to want to cast this to float like this. So that these, everything is the same. Everything looks the same, but the D-type is float32. And floats can feed into neural mats. So now let's construct our first neuron. This neuron will look at these input vectors. And as you remember from micrograd, these neurons basically perform a very simple function, Wx plus B, where Wx is a dot product. So we can achieve the same thing here. Let's first define the weights of this neuron, basically. We're the initial weights at initialization for this neuron. Let's initialize them with torshtotrandin. Torshtotrandin is fills a tensor with random numbers, drawn from a normal distribution. And a normal distribution has a probability density function like this. And so most of the numbers drawn from this distribution will be around zero, but some of them will be as high as almost three and so on. And very few numbers will be above three in magnitude. So we need to take a size as an input here. And I'm going to use size as to be 27 by one. So 27 by one, and then let's visualize W. So W is a column vector of 27 numbers. And these weights are then multiplied by the inputs. So now to perform this multiplication, we can take x encoding, and we can multiply it with W. This is a matrix multiplication operator in PyTorch. And the output of this operation is 5 by 1. The reason it's 5 by 5 is the following. We took x encoding, which is 5 by 27, and we multiplied it by 27 by 1. And in matrix multiplication, you see that the output will become 5 by 1, because these 27 will multiply and add. So basically what we're seeing here out of this operation is we are seeing the five activations of this neuron on these five inputs. And we've evaluated all of them in parallel. We didn't feed in just a single input to the single neuron. We fed in simultaneously all the five inputs into the same neuron. And in parallel, PyTorch has evaluated the Wx plus B, but here is just Wx. There's no bias. It is value W times x for all of them, independently. Now instead of a single neuron, though, I would like to have 27 neurons. And I'll show you in a second why I've gone 27 neurons. So instead of having just a one here, which is indicating this presence of one single neuron, we can use 27. And then when W is 27 by 27, this will in parallel evaluate all the 27 neurons on all five inputs, giving us a much better, much, much bigger result. So now what we've done is 5 by 27 multiplied, 27 by 27, and the output of this is now 5 by 27. So we can see that the shape of this is 5 by 27. So what is every element here telling us? It's telling us for every one of 27 neurons that we created. What is the firing rate of those neurons on every one of those five examples? So the element, for example, 3 comma 13 is giving us the firing rate of the 13th neuron looking at the third input. And the way this was achieved is by a dot product between the third input and the 13th column of this W matrix here. So using a major multiplication, we can very efficiently evaluate the dot product between lots of input examples in a batch. And lots of neurons where all of those neurons have weights in the columns of those W's. And in major multiplication, we're just doing those dot products in parallel. Just to show you that this is the case, we can take X and we can take the third row. And we can take the W and take its 13th column. And then we can do X and get 3, element wise multiply with W at 13 and sum that up. This WX plus B. Well, there's no plus B. It's just WX dot product. And that's this number. So you see that this is just being done efficiently by the matrix multiplication operation for all the input examples and for all the output neurons of this first layer. Okay, so we fed our 27 dimensional inputs into a first layer of a neural net that has 27 neurons. Right? So we have 27 inputs and now we have 27 neurons. These neurons perform W times X. They don't have a bias and they don't have a non-linearity like 10H. We're going to leave them to be a linear layer. In addition to that, we're not going to have any other layers. This is going to be it. It's just going to be the dumbest, smallest, simplest neural net, which is just a single linear layer. And now I'd like to explain what I want those 27 outputs to be. Intuitively, what we're trying to produce here for every single input example is we're trying to produce some kind of a probability distribution for the next character in a sequence. And there's 27 of them. But we have to come up with like precise semantics for exactly how we're going to interpret these 27 numbers that these neural state come on. Now intuitively, you see here that these numbers are negative and some of them are positive, etc. And that's because these are coming out of the neural net layer initialized with these normal distribution parameters. But what we want is we want something like we had here, like each row here told us the counts. And then we normalize the counts to get probabilities. And we want something similar to come out of the neural net. But what we just have right now is just some negative and positive numbers. Now, we want those numbers to somehow represent the probabilities for the next character. But you see that probabilities, they have a special structure. They're positive numbers and they sum to 1. And so that doesn't just come out of a neural net. And then they can't be counts because these counts are positive and counts are integers. So counts are also not really a good thing to output from a neural net. So instead what the neural net is going to output and how we are going to interpret the 27 numbers is that these 27 numbers are giving us log counts, basically. So instead of giving us counts directly, lock in this table, they're giving us log counts. And to get the counts, we're going to take the log counts and we're going to exponentially eat them. Now, exponentiation takes the following form. It takes numbers that are negative or they are positive. It takes the entire real line. And then if you plug in negative numbers, you're going to get e to the x, which is always below 1. So you're getting numbers lower than 1. And if you plug in numbers greater than 0, you're getting numbers greater than 1 all the way growing to the infinity. And this here grows to 0. So basically we're going to take these numbers here. And instead of them being positive and negative in all of the place, we're going to interpret them as log counts. And then we're going to element wise, exponentiate these numbers. Exponentiating them now gives us something like this. And you see that these numbers now, because they went through an exponent, all the negative numbers turned into numbers below 1, like 0,338. And all the positive numbers originally turned into even more positive numbers, sort of greater than 1. So like for example, 7 is some positive number over here. That is greater than 0. But exponentiated outputs here basically give us something that we can use and interpret as the equivalent of counts originally. So you see these counts here, 1, 12, 7, 51, 1, etc. The neural net is kind of now predicting counts. And these counts are positive numbers. They can never be below 0. So that makes sense. And they can now take on various values depending on the settings of W. So let me break this down. We're going to interpret these to be the log counts. In other words, for this, that is often used is so called logits. These are logits log counts. And these will be sort of the counts, logits exponentiated. And this is equivalent to the n matrix, sort of the n array that we used previously. Remember this was the n. This is the array of counts. And each row here are the counts for the next character, sort of. So those are the counts. And now the probabilities are just the counts normalized. And so I'm not going to find the same. But basically I'm not going to scroll all the place. We've already done this. We want to count that sum along the first dimension. And we want to keep them as true. We've went over this. And this is how we normalize the rows of our counts matrix to get our probabilities. So now these are the probabilities. And these are the counts that we have currently. And now when I show the probabilities, you see that every row here, of course, will sum to 1. Because they're normalized. And the shape of this is 5 by 27. And so really what we've achieved is for every one of our five examples, we now have a row that came out of a neural net. And because of the transformations here, we made sure that this output of this neural net now are probabilities, or we can interpret to be probabilities. So our WX here gave us logits. And then we interpret those to be log counts. We exponentiate to get something that looks like counts. And then we normalize those counts to get a probability distribution. And all of these are differentiable operations. So what we've done now is we are taking inputs. We have differentiable operations that we can back propagate through. And we're getting out probability distributions. So, for example, for the zero example that fed in, right, which was the zero example here was a one-half vector of zero. And it basically corresponded to feeding in this example here. So we're feeding an adot into a neural net. And the way we fed the dot into a neural net is that we first got its index. Then we one hot encoded it. Then it went into the neural net. And out came this distribution of probabilities. And its shape is 27, there's 27 numbers. And we're going to interpret this as the neural net's assignment for how likely every one of these characters, 27 characters are to come next. And as we tune the weights W, we're going to be, of course, getting different probabilities out for any character that you input. And so now the question is just, can we optimize and find a good W such that the probabilities coming out are pretty good. And the way we measure pretty good is by the loss function. Okay, so I organized everything into a single summary so that hopefully it's a bit more clear. So it starts here. With an input dataset, we have some inputs through the neural net. And we have some labels for the correct next character in a sequence. These are integers. Here, I'm using torsion generators now so that you see the same numbers that I see. And I'm generating 27 neurons weights. And each neuron here receives 27 inputs. Then here we're going to plug in all the input examples x's into a neural net. So here, this is a forward pass. First, we have to encode all of the inputs into one hot representations. So we have 27 classes. We pass in these integers. And x-nc becomes a array that is 5 by 27. Zeroes, except for a few ones. We then multiply this in the first layer of a neural net to get low-gets. Expand and shade the low-gets to get fake counts, sort of, and normalize these counts to get probabilities. So these last two lines, by the way, here, are called the softmax, which I pulled up here. Softmax is a very often used layer in a neural net that takes these z's, which are low-gets. Expand and shade them. And the rise in normalizes. It's a way of taking outputs of a neural net layer. And these outputs can be positive or negative. And it outputs probability distributions. It outputs something that is always sums to one in our positive numbers, just like probabilities. So it's going to look like a normalization function if you want to think of it that way. And you can put it on top of any other linear layer inside a neural net. And it basically makes a neural net output probabilities. That's very often used. And we used it as well here. So this is the forward pass. And that's how we made a neural net output probability. Now, you'll notice that all of these, this entire forward pass is made up of differentiable layers. Everything here, we can back propagate through. And we saw some of the back propagation in micrograd. This is just multiplication and addition. All that's happening here is just multiply and add. And we know how to back propagate through them. Expandentiation, we know how to back propagate through. And then here we are summing. And sum is easily back propagated as well. And division as well. So everything here is the differential operation. And we can back propagate through. Now, we achieve these probabilities, which are five by 27 for every single example. We have a vector of probabilities that sum to one. And then here I wrote a bunch of stuff to sort of like break down the examples. So we have five examples making up Emma, right? And there are five by grams inside Emma. So by gram example, a by gram example one is that E is the beginning character right after dot. And the indexes for these are zero and five. So then we feed in a zero. That's the input as a neural net. We get probabilities from the neural net that are 27 numbers. And then the label is five because he actually comes after dot. So that's the label. And then we use this label five to index into the probability distribution here. So this index five here is zero one two three four five. It's this number here, which is here. So that's basically the probability assigned by the neural net to the actual correct character. You see that the net work currently thinks that this next character that E following dot is only one percent likely, which is of course not very good, right? Because this actually is a training example. And the network thinks that this is currently very, very unlikely. But that's just because we didn't get very lucky in generating a good setting of W. So right now this network thinks it's unlikely and zero point zero one is not a good outcome. So the log likelihood then is very negative. And the negative log likelihood is very positive. And so four is a very high negative log likelihood. And that means we're going to have a high loss. Because what is the loss? The loss is just the average negative log likelihood. And the second character is E and you see here that also the network thought that M following E is very unlikely one percent. Of the for M following M it thought it was 2% and for a following M it actually thought it was 7% likely. So just by chance, this one actually has a pretty good probability. And therefore pretty low negative log likelihood. And finally here it thought this was 1% likely. So overall our average negative log likelihood, which is the loss, the total loss that summarizes basically the how well this network currently works. At least on this one word, not on the full data suggest the one word is 3.76. Which is actually very fairly high loss. This is not a very good setting of W's. Now here's what we can do. We're currently getting 3.76. We can actually come here and we can change our W. We can resample it. So let me just add one to have a different seed. And then we get a different W. And then we can rerun this. And with this different seed, with this different setting of W's, we now get 3.37. So this is a much better W. And it's better because the probabilities just happen to come out higher for the characters that actually are next. And so you can imagine actually just resampling this, you know, we can try two. So, okay, this was not very good. Let's try one more. We can try three. Okay, this was terrible setting because we have a very high loss. So anyway, I'm going to erase this. What I'm doing here, which is just guess and check of randomly assigning parameters and seeing if the network is good. That is amateur hour. That's not how you optimize in your alert. The way you optimize in your alert is you start with some random guess. And we're going to commit to this one, even though it's not very good. But not the big deal is we have a loss function. So this loss is made up only of the furniture operations. And we can minimize the loss by tuning W's by computing the gradients of the loss with respect to these W matrices. And so then we can tune W to minimize the loss and find a good setting of W using gradient based optimization. So let's see how that will work. Now, things are actually going to look almost identical to what we had with micrograd. So here I pulled up the lecture from micrograd, the notebook. It's from this repository. And when I scroll all the way to the end where we left off with micrograd, we had something very, very similar. We had a number of input examples. In this case, we had four input examples inside X's. And we had their targets. These are targets. Just like here we have our X's now, but we have five of them. And they're now integers instead of vectors. But we're going to convert our integers to vectors, except our vectors will be 27 large instead of three large. And then here what we did is first we did a forward pass where we ran a neural net on all the inputs to get predictions. Our neural net at the time this n effects was a net of multilayer perceptron. Our neural net is going to look different because our neural net is just a single layer. Single linear layer followed by a softmax. So that's our neural net. And the loss here was the mean squared error. So we simply subtracted the prediction from the ground truth and squared it and some they roll up. And that was the loss. And loss was the single number that summarized the quality of the neural net. And when loss is low, like almost zero, that means the neural net is predicting correctly. So we had a single number that that summarized the performance of the neural net. And everything here was differentiable and was stored in massive compute graph. And then we iterated over all the parameters we made sure that the gradients are set to zero. And we called lost a backward and lost a backward initiated back propagation at the final output node of loss. Right. So yeah, remember these expressions. We had lost all the way at the end. We start that propagation and we went all the way back. And we made sure that we populated all the parameters dot grad. So that grad started at zero, but back propagation filled it in. And then in the update, we iterated over the parameters and we simply did a parameter update where every single element of our parameters was nudged in the opposite direction of the gradient. And so we're going to do the exact same thing here. So I'm going to pull this up on the side here. So that we have it available and we're actually going to do the exact same thing. So this was the forward pass. So where we did this. And props is our white bread. So now we have to evaluate the loss, but we're not using the mean square there. We're using the negative log likelihood because we are doing classification. We're not doing regression as it's called. So here we want to calculate loss. Now the way we calculated is is just this average negative log likelihood. Now this probs here. Has a shape of five by 27. And so to get all the we basically want to pluck out the probabilities at the correct indices here. So in particular, because the labels are stored here in the array wise, basically what we're after is for the first example, we're looking at probability of five. Right, at the index five. For the second example at the second row or row index one, we are interested in the probability of science to index 13. At the second example, we also have 13. At the third row, we want one. And at the last row, which is four, we want zero. So these are the probabilities we're interested in. Right. And you can see that they're not amazing as we saw above. So these are the probabilities we want, but we want like a more efficient way to access these probabilities, not just listing them out in a tuple like this. So it turns out that the way to this in PyTorch, one of the ways at least, is we can basically pass in all of these, sorry about that, all of these integers and vectors. So the these ones, you see how they're just zero, one, two, three, four. We can actually create that using MP, not MP, sorry, torch dot arrange of five zero, one, two, three, four. So we can index here with torch dot arrange of five. And here we index with wise. And you see that that gives us exactly these numbers. So that plucks out the probabilities of that the neural network assigns to the correct next character. Now we take those probabilities and we don't we actually look at the log probability. So we want to dot log. And then we want to just average that up. So take the mean of all that. And then it's the negative average log likelihood that is the loss. So the loss here is 3.7 something and you see that this loss 3.76, 3.76 is exactly as we've obtained before. But this is a vectorized form of that expression. So we get the same loss. And the same loss we can consider sort of as part of this forward pass and we've achieved here now loss. Okay, so we made our way all the way to loss. We define the forward pass. We forwarded the network and the loss. Now we're ready to do backward pass. So backward pass. We want to first make sure that all the gradients are reset. So they're at zero. Now in pie torch, you can set the gradients to be zero. But you can also just set it to none and setting it to none is more efficient and pie torch will interpret none as like a lack of a gradient and is the same as zeros. So this is a way to set to zero the gradient. And now we do lost the backward. Before we do lost that backward, we need one more thing. If you remember from micro-grad, pie torch actually requires that we pass in requires grad is true. So that we tell pie torch that we are interested in calculating gradient for this lead tensor by default. This is false. So let me recalculate with that and then setting none and lost that backward. Now something magical happened when lost the backward was run because pie torch just like micro-grad, when we did the forward pass here, it keeps track of all the operations under the hood. It builds a full computational graph. Just like the graphs we produced in micro-grad, those graphs exist inside pie torch. And so it knows all the dependencies and all the mathematical operations of everything. And when you then calculate the loss, we can call a dot backward on it. And that backward then fills in the gradients of all the intermediates all the way back to W's, which are the parameters of our neural net. So now we can do WL grad and we see that it has structure. There's stuff inside it. And these gradients, every single element here, so W.shape is 27 by 27, W grads shape is the same, 27 by 27. And every element of W.grad is telling us the influence of that weight on the loss function. So for example, this number all the way here, if this element, the zero zero element of W, because the gradient is positive, it's telling us that this has a positive influence on the loss. Slightly nudging, W slightly taking W zero zero and adding a small h to it would increase the loss, mildly, because this gradient is positive. Some of these gradients are also negative. So that's telling us about the gradient information. And we can use this gradient information to update the weights of this neural network. So let's not do the update. It's going to be very similar to what we had in micrograd. We need no loop over all the parameters because we only have one parameter tensor and that is W. So we simply do W dot data plus equals. We can actually copy this almost exactly negative 0.1 times W dot grad. And that would be the update to the tensor. So that updates the tensor. And because the tensor is updated, we would expect that now the loss should decrease. So here, if I print loss. It was 3.76, right? So we've updated the W here. So if I recalculate forward pass, loss now should be slightly lower. So 3.76 goes to 3.74. And then we can again set to set grad to none and backward update. And now the parameters changed again. So if we recalculate the forward pass, we expect a lower loss again 3.72. And this is again doing the we're now doing reading the set. And when we achieve a low loss, that will mean that the network is assigning high probabilities to the correct next characters. Okay, so I rearranged everything and I put it all together from scratch. So here is where we construct our data set of diagrams. You see that we are still iterating only over the first word, Emma. I'm going to change that in a second. I added a number that counts the number of elements in axis so that we explicitly see that number of examples is 5. Because currently we're just working with Emma. There's five bygones there. And here I added a loop of exactly what we had before. So we had 10 iterations of very decent of forward pass, backward pass, and an update. And so running these two cells, initialization and creating the scent, gives us some improvement on the last function. But now I want to use all the words. And there's not five, but 228,000 bygones now. However, this should require no modification whatsoever. Everything should just run because all the code we wrote doesn't carry their five bygones or 228,000 bygones. And with everything we should just work. So you see that this will just run. But now we are optimizing over the entire training set of all the bygones. And you see now that we are decreasing very slightly. So actually we can probably afford a larger learning rate. And probably for even larger learning rate. Even 50 seems to work on this very, very simple example. So let me re-enact the lies and let's run 100 iterations. See what happens. Okay. We seem to be coming up to some pretty good losses here. 2.47. Let me run 100 more. What is the number that we expect by the way in the loss? We expect to get something around what we had originally actually. So all the way back, if you remember, in the beginning of this video, when we optimized just by counting, our loss was roughly 2.47 after we had its moving. But before its moving, we had roughly 2.45, likely it. Sorry, loss. And so that's actually roughly the vicinity of what we expect to achieve. But before we achieved it by counting, and here we are achieving roughly the same result, but with gradient-based optimization. So we come to about 2.46, 2.45, etc. And that makes sense because fundamentally we're not taking any additional information. We're still just taking in the previous character and trying to predict the next one, but instead of doing it explicitly by counting and normalizing, we are doing it with gradient-based learning. And it just so happens that the explicit approach happens to very well optimize the loss function. Without any need for a gradient-based optimization, because the setup for bi-gram language models is so straightforward, so simple, we can just afford to estimate as probably is directly and maintain them in a table. But the gradient-based approach is significantly more flexible. So we've actually gained a lot because what we can do now is we can expand this approach and complexify the neural net. So currently we're just taking a single character and feeding into a neural net, and the neural is extremely simple. But we're about to iterate on this substantially. We're going to be taking multiple previous characters, and we're going to be feeding them into increasingly more complex neural nets. But fundamentally, the output of the neural net will always just be low jits. And those low jits will go through the exact same transformation. We are going to take them through a softmax, calculate the loss function, and the negative log likelihood, and do gradient-based optimization. And so actually, as we complexify the neural nets, and work all the way up to transformers, none of this will really fundamentally change. None of this will fundamentally change. The only thing that will change is the way we do the forward pass, or we've taken some previous characters and calculated logits for the next character in a sequence. That will become more complex, and I will use the same machinery to optimize it. And it's not obvious how we would have extended this bi-gram approach into the case where there are many more characters at the input. Because eventually these tables would get way too large, because there's way too many combinations of what previous characters could be. If you only have one previous character, we can just keep everything in a table, the counts. But if you have the last 10 characters that are input, we can't actually keep everything in a table anymore. So this is fundamentally an unscatable approach, and the neural network approach is significantly more scalable, and it's something that actually we can improve on over time. So that's where we will be digging next. I wanted to point out two more things. Number one, I want you to notice that this x and k here, this is made up of one hot vectors, and then those one hot vectors are multiplied by this w matrix. And we think of this as a multiple neurons being forwarded in a fully connected manner. But actually what's happening here is that, for example, if you have a one hot vector here that has a one at say the fifth dimension, then because of the way the matrix multiplication works, multiplying that one hot vector with w actually ends up plucking out the fifth row of w. A lot of logits would become just the fifth row of w, and that's because of the way the matrix multiplication works. So that's actually what ends up happening. So, but that's actually exactly what happened before. Because remember, all the way up here, we have a by-gram, we took the first character, and then that first character indexed into a row of this array here. And that row gave us the probability distribution for the next character. So the first character was used as a lookup into a matrix here to get the probability distribution. Well, that's actually exactly what's happening here. Because we're taking the index. We're encoding it as one hot and multiplying it by w. So, logits literally becomes the appropriate row of w. And that gets just as before, expedited to create the counts, and then normalized and becomes probability. So, this w here is literally the same as this array here. But, w, remember, is the log counts, not the counts. So it's more precise to say that w, expedentially, w.x is this array. But this array was filled in by counting, and by basically, popularly in the counts of bi-grams, whereas in the gradient base framework, we initialize it randomly, and then we let the loss guide us to arrive at the exact same array. So, this array, exactly here, is basically the array w at the end of optimization, except we arrive at it, piece by piece by following the loss. And that's why we also obtain the same loss function at the end. And the second note is, if I come here, remember the smoothing where we added fake counts to our counts in order to smooth out and make more uniform the distributions of these probabilities. And that prevented us from assigning zero probability to any one bi-gram. Now, if I increase the count here, what's happening to the probability? As I increase the count, probability becomes more and more uniform, right? Because these counts go only up to like 900 or whatever. So if I'm adding plus a million to every single number here, you can see how the row, and its probability, then, when we divide, it's just going to become more and more close to exactly even probability in a form distribution. It turns out that the gradient base framework has an equivalent to smoothing. In particular, think through these w's here, which we initialize randomly. We could also think about initializing w's to be zero. If all the entries of w are zero, then you'll see that logits will become all zero. And then, expd. And those logits becomes all one. And then, the probabilities turn out to be exactly uniform. So basically, when w's are all equal to each other, or say, especially zero, then the probabilities come out completely uniform. So, trying to incentivize w to be near zero is basically equivalent to label smoothing. And the more you incentivize that in a loss function, the more smooth distribution you're going to achieve. So this brings us to something that's called regularization, where we can actually augment the loss function to have a small component that we call a regularization loss. In particular, what we're going to do is we can take w. And we can, for example, square all of its entries. And then we can, oops, sorry about that. We can take all the entries of w and we can sum them. And because we're squaring, there will be no signs anymore. Natives and positives all get squashed to be positive numbers. And then, the way this works is you achieve zero loss if w is exactly or zero. But if w has non-zero numbers, you accumulate loss. And so we can actually take this and we can add it on here. So we can do something like loss plus w square dot sum. Or let's actually, instead of sum, let's take a mean. Because otherwise the sum gets too large. So mean is like a little bit more manageable. And then we have a regularization loss here. I'll say 0.01 times. Or something like that. You can choose the regularization strength. And then we can just optimize this. And now this optimization actually has two components. Not only is it trying to make all the probabilities work out. But in addition to that, there's an additional component that simultaneously tries to make all w's b zero. Because if w's are non-zero, you feel a loss. And so minimizing this, the only way to achieve that is for w to b zero. And so you can think of this as adding like a spring force, or like a gravity force, that pushes w to b zero. So w wants to b zero. And the probabilities want to be uniform. But they also simultaneously want to match up your probabilities as indicated by the data. And so the strength of this regularization is exactly controlling the amount of counts that you add here. Adding a lot more counts here corresponds to increasing this number. Because the more you increase it, the more this part of the loss function dominates this part. And the more these weights will be unable to grow. Because as they grow, they accumulate way too much loss. And so if this is strong enough, then we are not able to overcome the force of this loss. And we will never, and basically everything will be uniform predictions. So I thought that's kind of cool. Okay, and lastly, before we wrap up, I wanted to show you how you would sample from this neural net model. And I copy-pasted the sampling code from before, where remember that we sampled five times. And all we did was start at zero, we grabbed the current ix row of p. And that was our probability row, from which we sampled the next index, and just accumulated that and break when zero. And running this gave us these results. I still have the p in memory, so this is fine. Now, this p doesn't come from the row of p. Instead it comes from this neural net. First, we take ix, and we encode it into a one-hot row of x-ank. This x-ank multiplies rw, which really just plugs out the row of w corresponding to ix. Really, that's what's happening. And that gets our logits, and then we normalize those logits, and we actually exponential to get counts, and then we normalize to get the distribution, and then we can sample from the distribution. So if I run this, kind of anti-climatic or climatic, depending on how you look at it, but we get the exact same result. And that's because this is the identical model. Not only does it achieve the same loss, but as I mentioned, these are identical models, but we came to this answer in a very different way, and it's got a very different interpretation. But fundamentally, this is basically the same model, and gives the same samples here. And so, that's kind of cool. Okay, so we've actually covered a lot of ground. We introduced the bi-gram character level language model. We saw how we can train the model, how we can sample from the model, and how we can evaluate the quality of the model, using the negative log likelihood loss. And then we actually train the model in two completely different ways, that actually get the same result and the same model. In the first way, we just count it up, the frequency of all the bi-grams, and normalized. In the second way, we used the negative log likelihood loss, as a guide, to optimizing the counts matrix, or the counts array, so that the loss is minimized, in the gradient-based framework. And we saw that both of them give the same result, and that's it. Now, the second one of these, the gradient-based framework, is much more flexible. And right now, our neural network is super simple. We're taking a single previous character, and we're taking it through a single linear layer, to calculate the logits. This is about to complexify. So, in the follow-up videos, we're going to be taking more and more of these characters, and we're going to be feeding them into a neural net. But this neural net will still output the exact same thing. The neural net will output logits. And these logits will still be normalized in the exact same way, and all the loss, and everything else, and the gradient-based framework, everything stays identical. It's just that this neural net will now complexify all the way to transformers. So, that's going to be pretty awesome, and I'm looking forward to it for now. Bye.
[{"start": 0.0, "end": 2.0, "text": " Hi everyone, hope you're well."}, {"start": 2.0, "end": 6.0, "text": " And next up what I'd like to do is I'd like to build out Makemore."}, {"start": 6.0, "end": 11.0, "text": " Like micrograd before it, Makemore is a repository that I have on my GitHub web page."}, {"start": 11.0, "end": 12.0, "text": " You can look at it."}, {"start": 12.0, "end": 16.0, "text": " But just like with micrograd, I'm going to build it out step by step"}, {"start": 16.0, "end": 18.0, "text": " and I'm going to spell everything out."}, {"start": 18.0, "end": 20.0, "text": " So we're going to build it out slowly and together."}, {"start": 20.0, "end": 22.0, "text": " Now, what is Makemore?"}, {"start": 22.0, "end": 27.0, "text": " Makemore, as the name suggests, makes more of things that you give it."}, {"start": 27.0, "end": 29.0, "text": " So here's an example."}, {"start": 29.0, "end": 32.0, "text": " Names.TXT is an example dataset to make more."}, {"start": 32.0, "end": 38.0, "text": " And when you look at names.TXT, you'll find that it's a very large dataset of names."}, {"start": 38.0, "end": 41.0, "text": " So here's lots of different types of names."}, {"start": 41.0, "end": 47.0, "text": " In fact, I believe there are 32,000 names that I've sort of found randomly on the government website."}, {"start": 47.0, "end": 54.0, "text": " And if you trade Makemore on this dataset, it will learn to make more of things like this."}, {"start": 54.0, "end": 60.0, "text": " And in particular, in this case, that will mean more things that sound name-like,"}, {"start": 60.0, "end": 62.0, "text": " but are actually unique names."}, {"start": 62.0, "end": 65.0, "text": " And maybe if you have a baby and you're trying to assign a name,"}, {"start": 65.0, "end": 69.0, "text": " maybe you're looking for a cool new sounding unique name, Makemore might help you."}, {"start": 69.0, "end": 73.0, "text": " So here are some examples of generations from the neural network."}, {"start": 73.0, "end": 76.0, "text": " Once we train it on our dataset."}, {"start": 76.0, "end": 79.0, "text": " So here's some examples of unique names that it will generate."}, {"start": 79.0, "end": 85.0, "text": " Don't tell, I rot, Zendi, and so on."}, {"start": 85.0, "end": 90.0, "text": " And so all these sort of sound name-like, but they're not, of course, names."}, {"start": 90.0, "end": 94.0, "text": " So under the hood, Makemore is a character-level language model."}, {"start": 94.0, "end": 99.0, "text": " So what that means is that it is treating every single line here as an example."}, {"start": 99.0, "end": 105.0, "text": " And within each example, it's treating them all as sequences of individual characters."}, {"start": 105.0, "end": 109.0, "text": " So R-E-E-S-E is this example."}, {"start": 109.0, "end": 111.0, "text": " And that's the sequence of characters."}, {"start": 111.0, "end": 114.0, "text": " And that's the level on which we are building out Makemore."}, {"start": 114.0, "end": 117.0, "text": " And what it means to be a character-level language model then"}, {"start": 117.0, "end": 120.0, "text": " is that it's just sort of modeling those sequences of characters"}, {"start": 120.0, "end": 123.0, "text": " and it knows how to predict the next character in the sequence."}, {"start": 123.0, "end": 128.0, "text": " Now, we're actually going to implement a large number of character-level language models"}, {"start": 128.0, "end": 132.0, "text": " in terms of the neural networks that are involved in predicting the next character in a sequence."}, {"start": 132.0, "end": 136.0, "text": " So very simple, bi-gram and bag of word models, multilaylor perceptrons,"}, {"start": 136.0, "end": 140.0, "text": " recurring neural networks, all the way to modern transformers."}, {"start": 140.0, "end": 145.0, "text": " In fact, a transformer that we will build will be basically the equivalent transformer to GPT2"}, {"start": 145.0, "end": 147.0, "text": " if you have heard of GPT."}, {"start": 147.0, "end": 149.0, "text": " So that's kind of a big deal."}, {"start": 149.0, "end": 152.0, "text": " It's a modern network and by the end of the series,"}, {"start": 152.0, "end": 155.0, "text": " you will actually understand how that works on the level of characters."}, {"start": 155.0, "end": 159.0, "text": " Now, to give you a sense of the extensions here,"}, {"start": 159.0, "end": 162.0, "text": " after characters, we will probably spend some time on the word level"}, {"start": 162.0, "end": 167.0, "text": " so that we can generate documents of words, not just little segments of characters,"}, {"start": 167.0, "end": 170.0, "text": " but we can generate entire much larger documents."}, {"start": 170.0, "end": 175.0, "text": " And then we're probably going to go into images and image text networks,"}, {"start": 175.0, "end": 178.0, "text": " such as Dali, stable diffusion, and so on."}, {"start": 178.0, "end": 182.0, "text": " But for now, we have to start here, careful level language modeling."}, {"start": 182.0, "end": 183.0, "text": " Let's go."}, {"start": 183.0, "end": 186.0, "text": " So like before, we are starting with a completely blank GPNodebit page."}, {"start": 186.0, "end": 191.0, "text": " The first thing is I would like to basically load up the dataset, names.txt."}, {"start": 191.0, "end": 195.0, "text": " So we're going to open up names.txt for reading."}, {"start": 195.0, "end": 199.0, "text": " And we're going to read in everything into a massive string."}, {"start": 199.0, "end": 201.0, "text": " And then because it's a massive string,"}, {"start": 201.0, "end": 204.0, "text": " we'd only like the individual words and put them in the list."}, {"start": 204.0, "end": 211.0, "text": " So let's call split lines on that string to get all of our words as a Python list of strings."}, {"start": 211.0, "end": 216.0, "text": " So basically we can look at, for example, the first 10 words."}, {"start": 216.0, "end": 221.0, "text": " And we have that it's a list of Emma, Olivia, Eva, and so on."}, {"start": 221.0, "end": 227.0, "text": " And if we look at the top of the page here, that is indeed what we see."}, {"start": 227.0, "end": 229.0, "text": " So that's good."}, {"start": 229.0, "end": 235.0, "text": " This list actually makes me feel that this is probably sorted by frequency."}, {"start": 235.0, "end": 238.0, "text": " But okay, so these are the words."}, {"start": 238.0, "end": 241.0, "text": " Now we'd like to actually learn a little bit more about this dataset."}, {"start": 241.0, "end": 243.0, "text": " Let's look at the total number of words."}, {"start": 243.0, "end": 246.0, "text": " We expect this to be roughly 32,000."}, {"start": 246.0, "end": 249.0, "text": " And then what is the, for example, shortest word."}, {"start": 249.0, "end": 253.0, "text": " So min of length of each word for W in words."}, {"start": 253.0, "end": 258.0, "text": " So the shortest word will be length 2."}, {"start": 258.0, "end": 261.0, "text": " And max of length W for W in words."}, {"start": 261.0, "end": 264.0, "text": " So the longest word will be 15 characters."}, {"start": 264.0, "end": 267.0, "text": " So let's now think through our very first language model."}, {"start": 267.0, "end": 272.0, "text": " As I mentioned, a character level and good model is predicting the next character in a sequence"}, {"start": 272.0, "end": 276.0, "text": " given already some concrete sequence of characters before it."}, {"start": 276.0, "end": 280.0, "text": " Now what we have to realize here is that every single word here, like Isabella,"}, {"start": 280.0, "end": 285.0, "text": " is actually quite a few examples packed in to that single word."}, {"start": 285.0, "end": 289.0, "text": " Because what is an existence of a word like Isabella and the dataset telling us really?"}, {"start": 289.0, "end": 296.0, "text": " It's saying that the character I is a very likely character to come first in a sequence"}, {"start": 296.0, "end": 298.0, "text": " of a name."}, {"start": 298.0, "end": 303.0, "text": " The character S is likely to come after I."}, {"start": 303.0, "end": 307.0, "text": " The character A is likely to come after I.S."}, {"start": 307.0, "end": 310.0, "text": " The character B is very likely to come after I.S.A."}, {"start": 310.0, "end": 314.0, "text": " And someone all the way to A following Isabella."}, {"start": 314.0, "end": 317.0, "text": " And then there's one more example actually packed in here."}, {"start": 317.0, "end": 321.0, "text": " And that is that after there's Isabella,"}, {"start": 321.0, "end": 323.0, "text": " the word is very likely to end."}, {"start": 323.0, "end": 327.0, "text": " So that's one more sort of explicit piece of information that we have here,"}, {"start": 327.0, "end": 329.0, "text": " that we have to be careful with."}, {"start": 329.0, "end": 334.0, "text": " And so there's a lot packed into a single individual word in terms of the statistical structure"}, {"start": 334.0, "end": 337.0, "text": " of what's likely to follow in these character sequences."}, {"start": 337.0, "end": 340.0, "text": " And then of course we don't have just an individual word."}, {"start": 340.0, "end": 342.0, "text": " We actually have 32,000 of these."}, {"start": 342.0, "end": 344.0, "text": " And so there's a lot of structure here to model."}, {"start": 344.0, "end": 349.0, "text": " Now in the beginning, what I'd like to start with is I'd like to start with building a"}, {"start": 349.0, "end": 353.0, "text": " program language model. Now in a by-gram language model,"}, {"start": 353.0, "end": 357.0, "text": " we're always working with just two characters at a time."}, {"start": 357.0, "end": 361.0, "text": " So we're only looking at one character that we are given"}, {"start": 361.0, "end": 364.0, "text": " and we're trying to predict the next character in the sequence."}, {"start": 364.0, "end": 369.0, "text": " So what characters are likely to follow are what characters are likely to follow."}, {"start": 369.0, "end": 370.0, "text": " Hey, and so on."}, {"start": 370.0, "end": 373.0, "text": " And we're just modeling that kind of a little local structure."}, {"start": 373.0, "end": 377.0, "text": " And we're forgetting the fact that we may have a lot more information."}, {"start": 377.0, "end": 380.0, "text": " We're always just looking at the previous character to predict the next one."}, {"start": 380.0, "end": 384.0, "text": " So it's a very simple and weak language model, but I think it's a great place to start."}, {"start": 384.0, "end": 388.0, "text": " So now let's begin by looking at these by-grams in our data set and what they look like."}, {"start": 388.0, "end": 391.0, "text": " And these by-grams again are just two characters in a row."}, {"start": 391.0, "end": 393.0, "text": " So for WNWords,"}, {"start": 393.0, "end": 396.0, "text": " HW here is an individual word string."}, {"start": 396.0, "end": 399.0, "text": " We want to iterate for,"}, {"start": 399.0, "end": 403.0, "text": " we want to iterate this word with consecutive characters."}, {"start": 403.0, "end": 406.0, "text": " So two characters at a time, sliding it through the word."}, {"start": 406.0, "end": 410.0, "text": " Now, a interesting nice way, cute way to this in Python, by the way,"}, {"start": 410.0, "end": 412.0, "text": " is doing something like this."}, {"start": 412.0, "end": 419.0, "text": " For character on character two, in zip of W and W at one."}, {"start": 419.0, "end": 421.0, "text": " One call."}, {"start": 421.0, "end": 424.0, "text": " Print, character on character two."}, {"start": 424.0, "end": 426.0, "text": " And let's not do all the words."}, {"start": 426.0, "end": 427.0, "text": " Let's just do the first three words."}, {"start": 427.0, "end": 429.0, "text": " And I'm going to show you in a second how this works."}, {"start": 429.0, "end": 431.0, "text": " But from now, basically as an example,"}, {"start": 431.0, "end": 433.0, "text": " let's just do the very first word alone."}, {"start": 433.0, "end": 436.0, "text": " You see how we have a M up."}, {"start": 436.0, "end": 439.0, "text": " And this will just print EM, M-M, M-A."}, {"start": 439.0, "end": 443.0, "text": " And the reason this works is because W is the string M up."}, {"start": 443.0, "end": 447.0, "text": " W at one column is the string M-A."}, {"start": 447.0, "end": 450.0, "text": " And zip takes two iterators."}, {"start": 450.0, "end": 456.0, "text": " And it pairs them up and then creates an iterator over the tuples of their consecutive entries."}, {"start": 456.0, "end": 459.0, "text": " And if any one of these lists is shorter than the other,"}, {"start": 459.0, "end": 463.0, "text": " then it will just halt and return."}, {"start": 463.0, "end": 469.0, "text": " So basically that's why we return EM, M-M, M-M, M-A."}, {"start": 469.0, "end": 473.0, "text": " But then because this iterator's second one here runs out of elements,"}, {"start": 473.0, "end": 475.0, "text": " zip just ends."}, {"start": 475.0, "end": 477.0, "text": " And that's why we only get these tuples."}, {"start": 477.0, "end": 479.0, "text": " So pretty cute."}, {"start": 479.0, "end": 482.0, "text": " So these are the consecutive elements in the first word."}, {"start": 482.0, "end": 487.0, "text": " Now we have to be careful because we actually have more information here than just these three examples."}, {"start": 487.0, "end": 492.0, "text": " As I mentioned, we know that E is very likely to come first."}, {"start": 492.0, "end": 495.0, "text": " And we know that A in this case is coming last."}, {"start": 495.0, "end": 502.0, "text": " So one way to do this is basically we're going to create a special array here, our characters."}, {"start": 502.0, "end": 508.0, "text": " And we're going to hallucinate a special start token here."}, {"start": 508.0, "end": 512.0, "text": " I'm going to call it like special start."}, {"start": 512.0, "end": 514.0, "text": " So this is a list of one element."}, {"start": 514.0, "end": 518.0, "text": " And plus W."}, {"start": 518.0, "end": 521.0, "text": " And then plus a special end character."}, {"start": 521.0, "end": 526.0, "text": " And the reason I'm wrapping a list of W here is because W is a string, M-A,"}, {"start": 526.0, "end": 531.0, "text": " list of W will just have the individual characters in the list."}, {"start": 531.0, "end": 538.0, "text": " And then doing this again now, but not iterating over W's, but over the characters."}, {"start": 538.0, "end": 540.0, "text": " We'll give us something like this."}, {"start": 540.0, "end": 545.0, "text": " So E is likely, so this is a bygram of the start character and E."}, {"start": 545.0, "end": 549.0, "text": " And this is a bygram of the A in the special end character."}, {"start": 549.0, "end": 554.0, "text": " And now we can look at, for example, what this looks like for Olivia or Eva."}, {"start": 554.0, "end": 558.0, "text": " And indeed, we can actually, especially this for the entire dataset."}, {"start": 558.0, "end": 561.0, "text": " But we won't print that. That's going to be too much."}, {"start": 561.0, "end": 565.0, "text": " But these are the individual character bygrams and we can print them."}, {"start": 565.0, "end": 570.0, "text": " Now, in order to learn the statistics about which characters are likely to follow other characters,"}, {"start": 570.0, "end": 574.0, "text": " the simplest way in the bygram language models is to simply do it by counting."}, {"start": 574.0, "end": 582.0, "text": " So we're basically just going to count how often any one of these combinations occurs in the training set in these words."}, {"start": 582.0, "end": 587.0, "text": " So we're going to need some kind of a dictionary that's going to maintain some counts for every one of these bygrams."}, {"start": 587.0, "end": 590.0, "text": " So let's use a dictionary B."}, {"start": 590.0, "end": 596.0, "text": " And this will map these bygrams. So bygram is a tuple of character on character two."}, {"start": 596.0, "end": 604.0, "text": " And then B at bygram will be B dot get of bygram, which is basically the same as B at bygram."}, {"start": 604.0, "end": 613.0, "text": " But in the case that bygram is not in the dictionary B, we would like to buy default or term zero plus one."}, {"start": 613.0, "end": 618.0, "text": " So this will basically add up all the bygrams and count how often they occur."}, {"start": 618.0, "end": 627.0, "text": " Let's get rid of printing or rather let's keep the printing and let's just inspect what B is in this case."}, {"start": 627.0, "end": 630.0, "text": " And we see that many bygrams occur just a single time."}, {"start": 630.0, "end": 633.0, "text": " This one allegedly occluder three times."}, {"start": 633.0, "end": 635.0, "text": " So A was an ending character three times."}, {"start": 635.0, "end": 638.0, "text": " And that's true for all of these words."}, {"start": 638.0, "end": 641.0, "text": " All of Emma, Olivia and Eva and with A."}, {"start": 641.0, "end": 646.0, "text": " So that's why this occurred three times."}, {"start": 646.0, "end": 651.0, "text": " Now let's do it for all the words."}, {"start": 651.0, "end": 655.0, "text": " Oops, I should not have printed it."}, {"start": 655.0, "end": 657.0, "text": " I'm going to erase that."}, {"start": 657.0, "end": 659.0, "text": " Let's kill this."}, {"start": 659.0, "end": 664.0, "text": " Let's just run and now B will have the statistics of the entire data set."}, {"start": 664.0, "end": 668.0, "text": " So these are the counts across all the words of the individual bygrams."}, {"start": 668.0, "end": 673.0, "text": " And we could, for example, look at some of the most common ones and least common ones."}, {"start": 673.0, "end": 679.0, "text": " This kind of grows in Python, but the way to do this, the simplest way I like is we just use B dot items."}, {"start": 679.0, "end": 685.0, "text": " B dot items returns the tuples of key value."}, {"start": 685.0, "end": 691.0, "text": " In this case, the keys are the character bygrams and the values are the counts."}, {"start": 691.0, "end": 698.0, "text": " And so then what we want to do is we want to do sort it of this."}, {"start": 698.0, "end": 705.0, "text": " But by default, sort is on the first on the first item of a tuple."}, {"start": 705.0, "end": 710.0, "text": " But we want to sort by the values, which are the second element of a tuple that is the key value."}, {"start": 710.0, "end": 724.0, "text": " So we want to use the key equals lambda that takes the key value and returns the key value at one, not at zero, but at one, which is the count."}, {"start": 724.0, "end": 730.0, "text": " So we want to sort by the count of these elements."}, {"start": 730.0, "end": 733.0, "text": " And actually, we wanted to go backwards."}, {"start": 733.0, "end": 741.0, "text": " So here what we have is the bygram q and r occurs only a single time, d z occurred only a single time."}, {"start": 741.0, "end": 746.0, "text": " And when we sort this the other way around, we're going to see the most likely bygrams."}, {"start": 746.0, "end": 752.0, "text": " So we see that n was very often an ending character many, many times."}, {"start": 752.0, "end": 759.0, "text": " And apparently n always always follows an a and that's a very likely combination as well."}, {"start": 759.0, "end": 765.0, "text": " So this is kind of the individual counts that we achieve over the entire data set."}, {"start": 765.0, "end": 774.0, "text": " Now it's actually going to be significantly more convenient for us to keep this information in a two dimensional array instead of a high-thond dictionary."}, {"start": 774.0, "end": 785.0, "text": " So we're going to store this information in a 2D array and the rows are going to be the first character of the bygram and the columns are going to be the second character."}, {"start": 785.0, "end": 793.0, "text": " And each entry in the two dimensional array will tell us how often that first character follows the second character in the data set."}, {"start": 793.0, "end": 799.0, "text": " So in particular, the array representation that we're going to use or the library is that of PyTorch."}, {"start": 799.0, "end": 810.0, "text": " And PyTorch is a deep learning neural framework, but part of it is also this torch.tensor which allows us to create multi-dimensional arrays and manipulate them very efficiently."}, {"start": 810.0, "end": 815.0, "text": " So let's import PyTorch, which you can do by import Torch."}, {"start": 815.0, "end": 821.0, "text": " And then we can create a race. So let's create a array of zeros."}, {"start": 821.0, "end": 827.0, "text": " And we give it a size of this array. Let's create a 3 by 5 array as an example."}, {"start": 827.0, "end": 837.0, "text": " And this is a 3 by 5 array of zeros. And by default, you'll notice a dot d type, which is short for data type, is flow 32."}, {"start": 837.0, "end": 846.0, "text": " So these are single precision floating point numbers. Because we are going to represent counts, let's actually use d type as Torch.t in 32."}, {"start": 846.0, "end": 850.0, "text": " So these are 32 bit integers."}, {"start": 850.0, "end": 861.0, "text": " So now you see that we have integer data inside this tensor. Now, tensors allow us to really manipulate all the individual entries and do it very efficiently."}, {"start": 861.0, "end": 873.0, "text": " So for example, if we want to change this bit, we have to index into the tensor. And in particular, here, this is the first row. And the, because it's zero indexed."}, {"start": 873.0, "end": 884.0, "text": " So this is row index one and column index zero, one, two, three. So a at one comma three, we can set that to one."}, {"start": 884.0, "end": 893.0, "text": " And then a will have a one over there. We can of course also do things like this. So now a will be to over there."}, {"start": 893.0, "end": 903.0, "text": " And also we can, for example, say a zero zero is five. And then a will have a five over here. So that's how we can index into the arrays."}, {"start": 903.0, "end": 910.0, "text": " Now, of course, the array that we are interested in is much much bigger. So for our purposes, we have 26 letters of the alphabet."}, {"start": 910.0, "end": 919.0, "text": " And then we have two special characters as and e. So we want 26 plus two or 28 by 28 array."}, {"start": 919.0, "end": 926.0, "text": " And let's call it the capital N because it's going to represent the sort of the counts. Let me raise this stuff."}, {"start": 926.0, "end": 940.0, "text": " So that's the array that starts at zeroes, 28 by 28. And now let's copy paste this here. But instead of having an dictionary B, which we're going to erase, we now have an N."}, {"start": 940.0, "end": 949.0, "text": " Now the problem here is that we have these characters, which are strings, but we have to now basically index into a array."}, {"start": 949.0, "end": 955.0, "text": " And we have to index using integers. So we need some kind of a lockup table from characters to integers."}, {"start": 955.0, "end": 962.0, "text": " So let's construct such a character array. And the way we're going to do this is we're going to take all the words, which is a list of strings."}, {"start": 962.0, "end": 978.0, "text": " We're going to concatenate all of it into a massive string. So this is just simply the entire data set as a single string. We're going to pass this to the set constructor, which takes this massive string and throws out duplicates because sets do not allow duplicates."}, {"start": 978.0, "end": 988.0, "text": " So set of this will just be the set of all the lowercase characters. And there should be a total of 26 of them."}, {"start": 988.0, "end": 999.0, "text": " And now we actually don't want a set. We want a list. But we don't want a list sorted in some weird arbitrary way. We wanted to be sorted from a to Z."}, {"start": 999.0, "end": 1005.0, "text": " So sorted list. So those are our characters."}, {"start": 1005.0, "end": 1013.0, "text": " Now we want is this lookup table, as I mentioned. So let's create a special S to I. I will call it."}, {"start": 1013.0, "end": 1024.0, "text": " S is string or character. And this will be an S to I mapping for I S in a numerate of these characters."}, {"start": 1024.0, "end": 1035.0, "text": " So enumerate basically gives us this iterator over the integer index and the actual element of the list. And then we are mapping the character to the integer."}, {"start": 1035.0, "end": 1044.0, "text": " So S to I is a mapping from a to zero, B to one, etc. All the way from Z to 25."}, {"start": 1044.0, "end": 1056.0, "text": " And that's going to be useful here. But we actually also have to specifically set that S will be 26. And S to I at E will be 27. Right? Because Z was 25."}, {"start": 1056.0, "end": 1065.0, "text": " So those are the lookups. And now we can come here and we can map both character one and character to to their integers. So this will be S to I at character one."}, {"start": 1065.0, "end": 1079.0, "text": " And I X to will be S to I of character two. And now we should be able to do this line, but using our array. So in it, I X one, I X to this is the two dimensional array indexing."}, {"start": 1079.0, "end": 1094.0, "text": " I showed you before. And honestly, just plus equals one because everything starts at zero. So this should work and give us a large 28 by 20 array of all these counts."}, {"start": 1094.0, "end": 1105.0, "text": " So if we print in this is the array, but of course it looks ugly. So let's erase this ugly mess. And let's try to visualize it a bit more nicer."}, {"start": 1105.0, "end": 1116.0, "text": " So for that, we're going to use a library called matplotlib. So matplotlib allows us to create figures. So we can do things like PILT I'm show off the counter ray."}, {"start": 1116.0, "end": 1130.0, "text": " So this is the 20 by 28 array. And this is structure, but even this, I would say is still pretty ugly. So we're going to try to create a much nicer visualization of it. And I wrote a bunch of code for that."}, {"start": 1130.0, "end": 1143.0, "text": " The first thing we're going to need is we're going to need to invert this array here, this dictionary. So S to I is mapping from S to I. And in I to S, we're going to reverse this dictionary."}, {"start": 1143.0, "end": 1154.0, "text": " So it rate of all the items and just reverse that array. So I to S maps inversely from zero to a want to be, etc. So we'll need that."}, {"start": 1154.0, "end": 1160.0, "text": " And then here's the code that I came up with to try to make this a little bit nicer."}, {"start": 1160.0, "end": 1172.0, "text": " We create a figure we plot and then we do and then we visualize a bunch of things here. Let me just run it so you get a sense of what it is."}, {"start": 1172.0, "end": 1185.0, "text": " So you see here that we have the array spaced out and every one of these is basically like B follows G zero times B follows H 41 times."}, {"start": 1185.0, "end": 1197.0, "text": " So a follows J 175 times. And so what you can see that I'm doing here is first I show that entire array. And then I iterate over all the individual little cells here."}, {"start": 1197.0, "end": 1209.0, "text": " And I create a character string here, which is the inverse mapping I to S of the integer I and the integer J. So that's the diagrams in a character representation."}, {"start": 1209.0, "end": 1223.0, "text": " And then I plot just the diagram text and then I plot the number of times that this diagram occurs. Now the reason that there's a dot item here is because when you index into these arrays, these are torch tensors."}, {"start": 1223.0, "end": 1232.0, "text": " You see that we still get a tensor back. So the type of this thing you think it would be just an integer long 49 but it's actually a torched up tensor."}, {"start": 1232.0, "end": 1245.0, "text": " And so if you do dot item, then it will pop out that individual integer. So it will just be 149. So that's what's happening there. And these are just some options to make it look nice."}, {"start": 1245.0, "end": 1254.0, "text": " So what is this structure of this array? We have all these counts and we see that some of them occur often and some of them do not occur often."}, {"start": 1254.0, "end": 1264.0, "text": " Now if you scrutinize this carefully, you will notice that we're not actually being very clever. That's because when you come over here, you'll notice that for example, we have an entire row of completely zeroes."}, {"start": 1264.0, "end": 1274.0, "text": " And that's because the end character is never possibly going to be the first character of a diagram because we're always placing these end tokens all at the end of the diagram."}, {"start": 1274.0, "end": 1287.0, "text": " Similarly, we have entire column zeros here because the S character will never possibly be the second element of a diagram because we always start with S and we end with E and we only have the words in between."}, {"start": 1287.0, "end": 1298.0, "text": " So we have an entire column of zeros and entire row of zeros. And in this little two by two matrix here as well, the only one that can possibly happen is if S directly follows E."}, {"start": 1298.0, "end": 1307.0, "text": " That can be non-zero if we have a word that has no letters. So in that case, there's no letters in a word. It's an empty word and we just have S follows E."}, {"start": 1307.0, "end": 1315.0, "text": " But the other ones are just not possible. And so we're basically wasting space and not only that, but the S and the E are getting very crowded here."}, {"start": 1315.0, "end": 1325.0, "text": " I was using these brackets because there's convention and natural language processing to use these kinds of brackets to denote special tokens. But we're going to use something else."}, {"start": 1325.0, "end": 1333.0, "text": " So let's fix all this and make it prettier. We're not actually going to have two special tokens. We're only going to have one special token."}, {"start": 1333.0, "end": 1344.0, "text": " So we're going to have n by n array of 27 by set 27 instead. Instead of having two, we will just have one and I will call it a dot."}, {"start": 1344.0, "end": 1350.0, "text": " Okay. Let me swing this over here."}, {"start": 1350.0, "end": 1356.0, "text": " Now, one more thing that I would like to do is I would actually like to make this special character half position zero."}, {"start": 1356.0, "end": 1362.0, "text": " And I would like to offset all the other letters off. I find that a little bit more pleasing."}, {"start": 1362.0, "end": 1369.0, "text": " So we need a plus one here so that the first character, which is A, will start at one."}, {"start": 1369.0, "end": 1382.0, "text": " So S to I will now be a starts at one and dot is zero. And I to us, of course, we're not changing this because I to us just creates reverse mapping and this will work fine."}, {"start": 1382.0, "end": 1386.0, "text": " So one is a to us B zero is dot."}, {"start": 1386.0, "end": 1392.0, "text": " So we reverse that here. We have a dot and a dot."}, {"start": 1392.0, "end": 1402.0, "text": " This should work fine. Make sure I started zeros count. And then here we don't go up to 28. We go up to 27."}, {"start": 1402.0, "end": 1411.0, "text": " And this should just work."}, {"start": 1411.0, "end": 1416.0, "text": " Okay. So we see that dot dot never happened. It's at zero because we don't have empty words."}, {"start": 1416.0, "end": 1423.0, "text": " Then this row here now is just very simply the counts for all the first letters."}, {"start": 1423.0, "end": 1433.0, "text": " So G J starts a word H starts word I starts a word, etc. And then these are all the ending characters."}, {"start": 1433.0, "end": 1437.0, "text": " And in between we have the structure of what characters follow each other."}, {"start": 1437.0, "end": 1441.0, "text": " So this is the counts array of our entire data set."}, {"start": 1441.0, "end": 1449.0, "text": " So this array actually has all the information necessary for us to actually sample from this by gram character level language model."}, {"start": 1449.0, "end": 1455.0, "text": " And roughly speaking we're going to do is we're just going to start following these probabilities and these counts."}, {"start": 1455.0, "end": 1458.0, "text": " And we're going to start sampling from the model."}, {"start": 1458.0, "end": 1464.0, "text": " So in the beginning, of course, we start with the dot, the start token dot."}, {"start": 1464.0, "end": 1470.0, "text": " So to sample the first character of a name, we're looking at this row here."}, {"start": 1470.0, "end": 1479.0, "text": " So we see that we have the counts and those counts externally are telling us how often any one of these characters is to start a word."}, {"start": 1479.0, "end": 1493.0, "text": " So if we take this n and we grab the first row, we can do that by using just indexing at zero and then using this notation colon for the rest of that row."}, {"start": 1493.0, "end": 1502.0, "text": " So n zero colon is indexing into the zero row and then grabbing all the columns."}, {"start": 1502.0, "end": 1506.0, "text": " And so this will give us a one dimensional array of the first row."}, {"start": 1506.0, "end": 1513.0, "text": " So zero for four ten, you know, zero for four ten, one three oh six one five four two, etc."}, {"start": 1513.0, "end": 1514.0, "text": " It's just the first row."}, {"start": 1514.0, "end": 1519.0, "text": " The shape of this is 27, just the row of 27."}, {"start": 1519.0, "end": 1526.0, "text": " And the other way that you can do this also is you just you don't actually give this you just grab the zero row like this."}, {"start": 1526.0, "end": 1528.0, "text": " This is equivalent."}, {"start": 1528.0, "end": 1535.0, "text": " Now these are the counts and now what we'd like to do is we'd like to basically sample from this."}, {"start": 1535.0, "end": 1539.0, "text": " Since these are the raw counts, we actually have to convert this to probabilities."}, {"start": 1539.0, "end": 1542.0, "text": " So we create a probability vector."}, {"start": 1542.0, "end": 1550.0, "text": " So we'll take n of zero and we'll actually convert this to float first."}, {"start": 1550.0, "end": 1554.0, "text": " Okay, so these integers are converted to float, float, equine numbers."}, {"start": 1554.0, "end": 1559.0, "text": " And the reason we're creating floats is because we're about to normalize these counts."}, {"start": 1559.0, "end": 1569.0, "text": " So to create a probability distribution here, we want to divide, we basically want to the peak, p divide p dot sum."}, {"start": 1569.0, "end": 1574.0, "text": " And now we get a vector of smaller numbers and these are now probabilities."}, {"start": 1574.0, "end": 1579.0, "text": " So of course, because we divided by the sum, the sum of p now is one."}, {"start": 1579.0, "end": 1581.0, "text": " So this is a nice proper probability distribution."}, {"start": 1581.0, "end": 1588.0, "text": " It sums to one and this is giving us the probability for any single character to be the first character of a word."}, {"start": 1588.0, "end": 1592.0, "text": " So now we can try to sample from this distribution to sample from these distributions."}, {"start": 1592.0, "end": 1596.0, "text": " We're going to use torshtut multinomial, which I've pulled up here."}, {"start": 1596.0, "end": 1603.0, "text": " So torshtut multinomial returns samples from the multinomial probability distribution,"}, {"start": 1603.0, "end": 1608.0, "text": " which is a complicated way of saying, you give me probabilities and I will give you integers,"}, {"start": 1608.0, "end": 1611.0, "text": " which are sampled according to the probability distribution."}, {"start": 1611.0, "end": 1613.0, "text": " So this is the signature of the method."}, {"start": 1613.0, "end": 1619.0, "text": " And to make everything deterministic, we're going to use a generator object in PyTorch."}, {"start": 1619.0, "end": 1621.0, "text": " So this makes everything deterministic."}, {"start": 1621.0, "end": 1627.0, "text": " So when you run this on your computer, you're going to get the exact same results that I'm getting here on my computer."}, {"start": 1627.0, "end": 1632.0, "text": " So let me show you how this works."}, {"start": 1632.0, "end": 1638.0, "text": " Here's the deterministic way of creating a torch generator object,"}, {"start": 1638.0, "end": 1641.0, "text": " seeding it with some number that we can agree on."}, {"start": 1641.0, "end": 1645.0, "text": " So that seeds a generator, gets us an object g."}, {"start": 1645.0, "end": 1652.0, "text": " And then we can pass that g to a function that creates here random numbers, torshtut random,"}, {"start": 1652.0, "end": 1655.0, "text": " creates random numbers, three of them."}, {"start": 1655.0, "end": 1660.0, "text": " And it's using this generator object to, as a source of randomness."}, {"start": 1660.0, "end": 1666.0, "text": " So without normalizing it, I can just print."}, {"start": 1666.0, "end": 1671.0, "text": " This is sort of like numbers between zero and one that are random according to this thing."}, {"start": 1671.0, "end": 1677.0, "text": " And whenever I run it again, I'm always going to get the same result because I keep using the same generator object,"}, {"start": 1677.0, "end": 1679.0, "text": " which I'm seeding here."}, {"start": 1679.0, "end": 1687.0, "text": " And then if I divide to normalize, I'm going to get a nice probability distribution of just three elements."}, {"start": 1687.0, "end": 1691.0, "text": " And then we can use torshtut multinomial to draw samples from it."}, {"start": 1691.0, "end": 1694.0, "text": " So this is what that looks like."}, {"start": 1694.0, "end": 1701.0, "text": " So torshtut multinomial will take the torshtensor of probability distributions."}, {"start": 1701.0, "end": 1705.0, "text": " Then we can ask for a number of samples like C20."}, {"start": 1705.0, "end": 1711.0, "text": " Replacement equals true means that when we draw an element, we can draw it,"}, {"start": 1711.0, "end": 1716.0, "text": " and then we can put it back into the list of eligible indices to draw again."}, {"start": 1716.0, "end": 1722.0, "text": " And we have to specify replacement as true because by default, for some reason, it's false."}, {"start": 1722.0, "end": 1726.0, "text": " So I think it's just something to be careful with."}, {"start": 1726.0, "end": 1728.0, "text": " And the generator is passed in here."}, {"start": 1728.0, "end": 1732.0, "text": " So we are going to always get deterministic results, the same results."}, {"start": 1732.0, "end": 1738.0, "text": " So if I run these two, we're going to get a bunch of samples from this distribution."}, {"start": 1738.0, "end": 1745.0, "text": " Now you'll notice here that the probability for the first element in this tensor is 60%."}, {"start": 1745.0, "end": 1751.0, "text": " So in these 20 samples, we'd expect 60% of them to be zero."}, {"start": 1751.0, "end": 1754.0, "text": " We'd expect 30% of them to be one."}, {"start": 1754.0, "end": 1762.0, "text": " And because the element index 2 has only 10% probability, very few of these samples should be two."}, {"start": 1762.0, "end": 1765.0, "text": " And indeed, we only have a small number of twos."}, {"start": 1765.0, "end": 1769.0, "text": " And we can sample as many as we would like."}, {"start": 1769.0, "end": 1776.0, "text": " And the more we sample, the more these numbers should, roughly, have the distribution here."}, {"start": 1776.0, "end": 1782.0, "text": " So we should have lots of zeros, half as many ones."}, {"start": 1782.0, "end": 1791.0, "text": " And we should have three times s few, sorry, s few ones, and three times s few twos."}, {"start": 1791.0, "end": 1793.0, "text": " So you see that we have very few twos."}, {"start": 1793.0, "end": 1796.0, "text": " We have some ones and most of them are zero."}, {"start": 1796.0, "end": 1799.0, "text": " So that's what torsion multilimals doing."}, {"start": 1799.0, "end": 1802.0, "text": " For us here, we are interested in this row."}, {"start": 1802.0, "end": 1806.0, "text": " We've created this p here."}, {"start": 1806.0, "end": 1809.0, "text": " And now we can sample from it."}, {"start": 1809.0, "end": 1817.0, "text": " So if we use the same seed, and then we sample from this distribution, let's just get one sample."}, {"start": 1817.0, "end": 1822.0, "text": " Then we see that the sample is, say, 13."}, {"start": 1822.0, "end": 1825.0, "text": " So this will be the index."}, {"start": 1825.0, "end": 1828.0, "text": " And let's, you see how it's a tensor that wraps 13."}, {"start": 1828.0, "end": 1832.0, "text": " We again have to use dot item to pop out that integer."}, {"start": 1832.0, "end": 1837.0, "text": " And now index would be just number 13."}, {"start": 1837.0, "end": 1846.0, "text": " And of course, we can map the I2S of IX to figure out exactly which character we're sampling here."}, {"start": 1846.0, "end": 1848.0, "text": " We're sampling M."}, {"start": 1848.0, "end": 1853.0, "text": " So we're saying that the first character is in our generation."}, {"start": 1853.0, "end": 1855.0, "text": " And just look at the row here."}, {"start": 1855.0, "end": 1860.0, "text": " M was drawn, and we can see that M actually starts a large number of words."}, {"start": 1860.0, "end": 1865.0, "text": " M started 2,500 words out of 32,000 words."}, {"start": 1865.0, "end": 1869.0, "text": " So almost a bit less than 10% of the words start with M."}, {"start": 1869.0, "end": 1875.0, "text": " So this was actually fairly likely character to draw."}, {"start": 1875.0, "end": 1877.0, "text": " So that would be the first character of our word."}, {"start": 1877.0, "end": 1880.0, "text": " And now we can continue to sample more characters."}, {"start": 1880.0, "end": 1883.0, "text": " Because now we know that M started."}, {"start": 1883.0, "end": 1885.0, "text": " Now M is already sampled."}, {"start": 1885.0, "end": 1889.0, "text": " So now to draw the next character, we will come back here."}, {"start": 1889.0, "end": 1893.0, "text": " And we will work for the row that starts with M."}, {"start": 1893.0, "end": 1897.0, "text": " So you see M and we have a row here."}, {"start": 1897.0, "end": 1904.0, "text": " So we see that M dot is 516, M A is this many, M B is this many, etc."}, {"start": 1904.0, "end": 1906.0, "text": " So these are the counts for the next row."}, {"start": 1906.0, "end": 1909.0, "text": " And that's the next character that we are going to now generate."}, {"start": 1909.0, "end": 1912.0, "text": " So I think we are ready to actually just write out a loop."}, {"start": 1912.0, "end": 1915.0, "text": " So we are going to start to get a sense of how this is going to go."}, {"start": 1915.0, "end": 1922.0, "text": " We always begin at index 0, because that's the start token."}, {"start": 1922.0, "end": 1930.0, "text": " And then while true, we are going to grab the row corresponding to index that we are currently on."}, {"start": 1930.0, "end": 1934.0, "text": " So that's n array at ix."}, {"start": 1934.0, "end": 1939.0, "text": " Converted to float is rp."}, {"start": 1939.0, "end": 1945.0, "text": " Then we normalize this p to sum to 1."}, {"start": 1945.0, "end": 1948.0, "text": " I accidentally ran the infinite loop."}, {"start": 1948.0, "end": 1951.0, "text": " We normalize p to sum to 1."}, {"start": 1951.0, "end": 1954.0, "text": " Then we need this generator object."}, {"start": 1954.0, "end": 1956.0, "text": " And we are going to initialize up here."}, {"start": 1956.0, "end": 1961.0, "text": " And we are going to draw a single sample from this distribution."}, {"start": 1961.0, "end": 1966.0, "text": " And then this is going to tell us what index is going to be next."}, {"start": 1966.0, "end": 1972.0, "text": " If the index sampled is 0, then that's now the end token."}, {"start": 1972.0, "end": 1975.0, "text": " So we will break."}, {"start": 1975.0, "end": 1981.0, "text": " Otherwise, we are going to print s2i of ix."}, {"start": 1981.0, "end": 1985.0, "text": " i2s of ix."}, {"start": 1985.0, "end": 1987.0, "text": " And that's pretty much it."}, {"start": 1987.0, "end": 1990.0, "text": " We're just... this should work."}, {"start": 1990.0, "end": 1992.0, "text": " Okay, more."}, {"start": 1992.0, "end": 1994.0, "text": " So that's the name that we've sampled."}, {"start": 1994.0, "end": 1997.0, "text": " We started with m."}, {"start": 1997.0, "end": 2001.0, "text": " The next step was o, then r, and then dot."}, {"start": 2001.0, "end": 2005.0, "text": " And this dot, we printed here as well."}, {"start": 2005.0, "end": 2010.0, "text": " So let's not do this a few times."}, {"start": 2010.0, "end": 2017.0, "text": " So let's actually create an out list here."}, {"start": 2017.0, "end": 2020.0, "text": " And instead of printing, we're going to append."}, {"start": 2020.0, "end": 2024.0, "text": " So out that append this character."}, {"start": 2024.0, "end": 2027.0, "text": " And then here, let's just print it at the end."}, {"start": 2027.0, "end": 2030.0, "text": " So let's just join up all the outs."}, {"start": 2030.0, "end": 2032.0, "text": " And we're just going to print more."}, {"start": 2032.0, "end": 2035.0, "text": " Now we're always getting the same result because of the generator."}, {"start": 2035.0, "end": 2037.0, "text": " So who want to do this a few times?"}, {"start": 2037.0, "end": 2041.0, "text": " We can go for i and range 10."}, {"start": 2041.0, "end": 2043.0, "text": " We can sample 10 names."}, {"start": 2043.0, "end": 2046.0, "text": " And we can just do that 10 times."}, {"start": 2046.0, "end": 2049.0, "text": " And these are the names that we're getting out."}, {"start": 2049.0, "end": 2054.0, "text": " 10, 20."}, {"start": 2054.0, "end": 2057.0, "text": " I'll be honest with you, this doesn't look right."}, {"start": 2057.0, "end": 2061.0, "text": " So I've started a few minutes to convince myself that it actually is right."}, {"start": 2061.0, "end": 2068.0, "text": " The reason these samples are so terrible is that by-gram language model is actually just like really terrible."}, {"start": 2068.0, "end": 2070.0, "text": " We can generate a few more here."}, {"start": 2070.0, "end": 2075.0, "text": " And you can see that they're kind of like their name, like a little bit, like yanu, irailly, etc."}, {"start": 2075.0, "end": 2079.0, "text": " But they're just like totally messed up."}, {"start": 2079.0, "end": 2083.0, "text": " And I mean the reason that this is so bad, like we're generating h as a name."}, {"start": 2083.0, "end": 2086.0, "text": " But you have to think through it from the model's eyes."}, {"start": 2086.0, "end": 2089.0, "text": " It doesn't know that this h is the very first h."}, {"start": 2089.0, "end": 2092.0, "text": " All it knows is that h was previously."}, {"start": 2092.0, "end": 2095.0, "text": " And now how likely is h the last character?"}, {"start": 2095.0, "end": 2098.0, "text": " Well, it's somewhat likely."}, {"start": 2098.0, "end": 2100.0, "text": " And so it just makes it last character."}, {"start": 2100.0, "end": 2104.0, "text": " It doesn't know that there were other things before it or there were not other things before it."}, {"start": 2104.0, "end": 2108.0, "text": " And so that's why I'm generating all these like nonsense names."}, {"start": 2108.0, "end": 2121.0, "text": " And the other way to do this is to convince yourself that it is actually doing something reasonable, even though it's so terrible, is these little piece here are 27, right?"}, {"start": 2121.0, "end": 2123.0, "text": " Like 27."}, {"start": 2123.0, "end": 2126.0, "text": " So how about if we did something like this?"}, {"start": 2126.0, "end": 2129.0, "text": " Instead of having any structure whatsoever."}, {"start": 2129.0, "end": 2137.0, "text": " How about if p was just a torch dot ones of 27?"}, {"start": 2137.0, "end": 2139.0, "text": " By default, this is a float 32."}, {"start": 2139.0, "end": 2140.0, "text": " So this is fine."}, {"start": 2140.0, "end": 2143.0, "text": " Divide 27."}, {"start": 2143.0, "end": 2150.0, "text": " So what I'm doing here is this is the uniform distribution, which will make everything equally likely."}, {"start": 2150.0, "end": 2152.0, "text": " And we can sample from that."}, {"start": 2152.0, "end": 2154.0, "text": " So let's see if that doesn't need better."}, {"start": 2154.0, "end": 2155.0, "text": " Okay."}, {"start": 2155.0, "end": 2160.0, "text": " So it's this is what you have from a model that is completely untrained where everything is equally likely."}, {"start": 2160.0, "end": 2162.0, "text": " So it's obviously garbage."}, {"start": 2162.0, "end": 2168.0, "text": " And then if we have a trained model, which is trained on just by grams, this is what we get."}, {"start": 2168.0, "end": 2172.0, "text": " So you can see that it is more name like it is actually working."}, {"start": 2172.0, "end": 2176.0, "text": " It's just by gram is so terrible and we have to do better."}, {"start": 2176.0, "end": 2180.0, "text": " Now next I would like to fix an inefficiency that we have going on here."}, {"start": 2180.0, "end": 2186.0, "text": " Because what we're doing here is we're always fetching a row of n from the counts matrix up ahead."}, {"start": 2186.0, "end": 2188.0, "text": " And we're always doing the same things."}, {"start": 2188.0, "end": 2190.0, "text": " We're converting to float and we're dividing."}, {"start": 2190.0, "end": 2193.0, "text": " And we're doing this every single iteration of the slope."}, {"start": 2193.0, "end": 2195.0, "text": " And we just keep normalizing these rows over and over again."}, {"start": 2195.0, "end": 2197.0, "text": " And it's extremely inefficient and wasteful."}, {"start": 2197.0, "end": 2204.0, "text": " So what I'd like to do is I'd like to actually prepare a matrix capital p that will just have the probabilities in it."}, {"start": 2204.0, "end": 2208.0, "text": " So in other words, it's going to be the same as the capital n matrix here of counts."}, {"start": 2208.0, "end": 2213.0, "text": " But every single row will have the row of probabilities that is normalized to one,"}, {"start": 2213.0, "end": 2218.0, "text": " indicating the probability distribution for the next character given the character before it."}, {"start": 2218.0, "end": 2221.0, "text": " As defined by which row we're in."}, {"start": 2221.0, "end": 2225.0, "text": " So basically what we'd like to do is we'd like to just do it up front here."}, {"start": 2225.0, "end": 2228.0, "text": " And then we would like to just use that row here."}, {"start": 2228.0, "end": 2233.0, "text": " So here we would like to just do p equals p of ix instead."}, {"start": 2233.0, "end": 2234.0, "text": " Okay."}, {"start": 2234.0, "end": 2239.0, "text": " The other reason I want to do this is not just for efficiency, but also I would like us to practice these"}, {"start": 2239.0, "end": 2241.0, "text": " and dimensional tensors."}, {"start": 2241.0, "end": 2244.0, "text": " And I'd like us to practice their manipulation."}, {"start": 2244.0, "end": 2247.0, "text": " And especially something that's called broadcasting that we'll go into in a second."}, {"start": 2247.0, "end": 2253.0, "text": " We're actually going to have to become very good at these tensor manipulations because we're going to build out all the way to transformers."}, {"start": 2253.0, "end": 2257.0, "text": " We're going to be doing some pretty complicated array operations for efficiency."}, {"start": 2257.0, "end": 2262.0, "text": " And we need to really understand that and be very good at it."}, {"start": 2262.0, "end": 2268.0, "text": " So in doing what we want to do is we first want to grab the floating point copy of n."}, {"start": 2268.0, "end": 2271.0, "text": " And I'm mimicking the line here basically."}, {"start": 2271.0, "end": 2276.0, "text": " And then we want to divide all the rows so that they sum to one."}, {"start": 2276.0, "end": 2278.0, "text": " So we'd like to do something like this."}, {"start": 2278.0, "end": 2280.0, "text": " p divide p dot sum."}, {"start": 2280.0, "end": 2288.0, "text": " But now we have to be careful because p dot sum actually produces a sum."}, {"start": 2288.0, "end": 2289.0, "text": " Sorry."}, {"start": 2289.0, "end": 2298.0, "text": " P equals n dot float copy. p dot sum produces a sums up all of the counts of this entire matrix n."}, {"start": 2298.0, "end": 2301.0, "text": " And gives us a single number of just the summation of everything."}, {"start": 2301.0, "end": 2303.0, "text": " So that's not the way we want to divide."}, {"start": 2303.0, "end": 2310.0, "text": " We want to simultaneously and in parallel divide all the rows by their respective sums."}, {"start": 2310.0, "end": 2315.0, "text": " So what we have to do now is we have to go into documentation for tors.sum."}, {"start": 2315.0, "end": 2323.0, "text": " And we can scroll down here to a definition that is relevant to us, which is where we don't only provide an input array that we want to sum."}, {"start": 2323.0, "end": 2327.0, "text": " But we also provide the dimension along which we want to sum."}, {"start": 2327.0, "end": 2332.0, "text": " And in particular, we want to sum up over rows."}, {"start": 2332.0, "end": 2338.0, "text": " Now one more argument that I want you to pay attention to here is the keep them as false."}, {"start": 2338.0, "end": 2342.0, "text": " If keep them is true and the output tensor is of the same size as input,"}, {"start": 2342.0, "end": 2347.0, "text": " except of course the dimension along which you summed, which will become just one."}, {"start": 2347.0, "end": 2354.0, "text": " But if you pass in, keep them as false, then this dimension is squeezed out."}, {"start": 2354.0, "end": 2364.0, "text": " And so tors.sum not only does the sum and collapses dimension to be of size one, but in addition, it does what's called a squeeze where it squeezes out, it squeezes out that dimension."}, {"start": 2364.0, "end": 2370.0, "text": " So basically what we want here is we instead want to do p dot sum of sum axis."}, {"start": 2370.0, "end": 2375.0, "text": " And in particular, notice that p dot shape is 27 by 27."}, {"start": 2375.0, "end": 2382.0, "text": " So when we sum up across axis zero, then we would be taking the zero dimension and we would be summing across it."}, {"start": 2382.0, "end": 2391.0, "text": " So when keep them as true, then this thing will not only give us the counts across along the columns,"}, {"start": 2391.0, "end": 2395.0, "text": " but notice that basically the shape of this is one by 27."}, {"start": 2395.0, "end": 2400.0, "text": " We just get a row vector and the reason we get a row vector here again is because we pass in zero dimension."}, {"start": 2400.0, "end": 2405.0, "text": " So this zero dimension becomes one and we've done a sum and we get a row."}, {"start": 2405.0, "end": 2415.0, "text": " And so basically we've done the sum this way vertically and arrived at just a single one by 27 vector of counts."}, {"start": 2415.0, "end": 2419.0, "text": " What happens when you take out keep them is that we just get 27."}, {"start": 2419.0, "end": 2428.0, "text": " So it squeezes out that dimension and we just get one dimensional vector of size 27."}, {"start": 2428.0, "end": 2439.0, "text": " Now we don't actually want one by 27 row vector because that gives us the counts or the sums across the columns."}, {"start": 2439.0, "end": 2445.0, "text": " We actually want to sum the other way along dimension one and you'll see that the shape of this is 27 by 1."}, {"start": 2445.0, "end": 2452.0, "text": " So it's a column vector. It's a 27 by 1 vector of counts."}, {"start": 2452.0, "end": 2463.0, "text": " And that's because what's happened here is that we're going horizontally and this 27 by 27 matrix becomes a 27 by 1 array."}, {"start": 2463.0, "end": 2470.0, "text": " Now you'll notice by the way that the actual numbers of these counts are identical."}, {"start": 2470.0, "end": 2477.0, "text": " And that's because this special array of counts here comes from by grams to the sticks and actually it just so happens by chance."}, {"start": 2477.0, "end": 2485.0, "text": " Or because of the way this array is constructed that the sums along the columns or along the rows horizontally or vertically is identical."}, {"start": 2485.0, "end": 2492.0, "text": " But actually what we want to do in this case is we want to sum across the rows horizontally."}, {"start": 2492.0, "end": 2499.0, "text": " So what we want here is be that some of one with keep them true 27 by 1 column vector."}, {"start": 2499.0, "end": 2504.0, "text": " And now what we want to do is we want to divide by that."}, {"start": 2504.0, "end": 2512.0, "text": " Now we have to be careful here again. Is it possible to take what's a p. shape you see here is 27 by 27."}, {"start": 2512.0, "end": 2521.0, "text": " Is it possible to take a 27 by 27 array and divide it by what is a 27 by 1 array?"}, {"start": 2521.0, "end": 2528.0, "text": " Is that an operation that you can do? And whether or not you can perform this operation is determined by what's called broadcasting rules."}, {"start": 2528.0, "end": 2543.0, "text": " So if you just search broadcasting semantics in torch, you'll notice that there's a special definition for what's called broadcasting that for whether or not these two arrays can be combined in a binary operation like division."}, {"start": 2543.0, "end": 2548.0, "text": " So the first condition is each tensor has at least one dimension, which is the case for us."}, {"start": 2548.0, "end": 2557.0, "text": " And then when iterating over the dimension sizes starting at the trailing dimension, the dimension sizes must either be equal, one of them is one or one of them does not exist."}, {"start": 2557.0, "end": 2567.0, "text": " So let's do that. We need to align the two arrays and their shapes, which is very easy because both of these shapes have two elements."}, {"start": 2567.0, "end": 2572.0, "text": " So they're aligned. Then we iterate over from the right and going to the left."}, {"start": 2572.0, "end": 2578.0, "text": " Each dimension must be either equal, one of them is a one or one of them does not exist."}, {"start": 2578.0, "end": 2586.0, "text": " So in this case, they're not equal, but one of them is a one. So this is fine. And then this dimension, they're both equal. So this is fine."}, {"start": 2586.0, "end": 2592.0, "text": " So all the dimensions are fine. And therefore the this operation is broadcastable."}, {"start": 2592.0, "end": 2600.0, "text": " So that means that this operation is allowed. And what is it that these arrays do when you divide 27 by 27 by 27 by one?"}, {"start": 2600.0, "end": 2609.0, "text": " What it does is that it takes this dimension one and it stretches it out. It copies it to match 27 here. In this case."}, {"start": 2609.0, "end": 2620.0, "text": " So in our case, it takes this column vector, which is 27 by one. And it copies it 27 times to make these both be 27 by 27 internally."}, {"start": 2620.0, "end": 2628.0, "text": " You can think of it that way. And so it copies those counts. And then it does an element wise division, which is what we want."}, {"start": 2628.0, "end": 2634.0, "text": " Because these counts we want to divide by them on every single one of these columns in this matrix."}, {"start": 2634.0, "end": 2645.0, "text": " So this actually we expect will normalize every single row. And we can check that this is true by taking the first row, for example, and taking it some."}, {"start": 2645.0, "end": 2657.0, "text": " We expect this to be one because it's not normalized. And then we expect this now, because if we actually correctly normalize all the rows, we expect to get the exact same result here."}, {"start": 2657.0, "end": 2665.0, "text": " So let's run this. It's the exact same result. So this is correct. So now I would like to scare you a little bit."}, {"start": 2665.0, "end": 2672.0, "text": " You actually have to like basically encourage you very strongly to read through broadcasting semantics. And I encourage you to treat this with respect."}, {"start": 2672.0, "end": 2683.0, "text": " And it's not something to play fast and smooth. It's something to really respect, really understand and look up maybe some tutorials for broadcasting and practice it and be careful with it because you can very quickly run into bugs."}, {"start": 2683.0, "end": 2690.0, "text": " Let me show you what I mean. You see how here we have p.1. Keep them this true."}, {"start": 2690.0, "end": 2698.0, "text": " The shape of this is 27 by 1. Let me take out this line just so we have the n and then we can see the counts."}, {"start": 2698.0, "end": 2706.0, "text": " We can see that this is all the counts across all the rows. And it's 27 by 1 column vector."}, {"start": 2706.0, "end": 2713.0, "text": " Now suppose that I tried to do the following, but I erased keep them this true here."}, {"start": 2713.0, "end": 2722.0, "text": " What does that do? If keep them is not true, it's false. Then remember according to documentation, it gets rid of this dimension 1. It squeezes it out."}, {"start": 2722.0, "end": 2731.0, "text": " So basically we just get all the same counts, the same result except the shape of it is not 27 by 1. It is just 27 by 1 disappears."}, {"start": 2731.0, "end": 2740.0, "text": " But all the counts are the same. So you'd think that this divide that would work."}, {"start": 2740.0, "end": 2746.0, "text": " First of all, can we even write this? Is it even expected to run? Is it broadcastable?"}, {"start": 2746.0, "end": 2760.0, "text": " Let's determine if this result is broadcastable. p.1 is shape. It's 27. This is 27 by 27. So 27 by 27 broadcasting into 27."}, {"start": 2760.0, "end": 2770.0, "text": " So now suppose broadcasting number 1 align all the dimensions on the right done. Now iteration over all the dimensions started from the right going to the left."}, {"start": 2770.0, "end": 2780.0, "text": " All the dimensions must either be equal. One of them must be 1 or one then does not exist. So here they are or equal. Here the dimension does not exist."}, {"start": 2780.0, "end": 2792.0, "text": " So internally what broadcasting will do is it will create a 1 here and then we see that one of them is a 1 and this will get copied and this will run. This will broadcast."}, {"start": 2792.0, "end": 2808.0, "text": " Okay, so you'd expect this to work because we are this broadcast and this we can divide this. Now if I run this, you'd expect it to work but it doesn't."}, {"start": 2808.0, "end": 2817.0, "text": " Now you actually get garbage. You got a wrong result because this is actually a bug. This keeps them equal to true."}, {"start": 2817.0, "end": 2831.0, "text": " Makes it work. This is a bug. In both cases we are doing the correct counts. We are summing up across the rows but keep them saving us and making it work."}, {"start": 2831.0, "end": 2842.0, "text": " So in this case, I'd like you to encourage you to potentially like pause this video at this point and try to think about why this is buggy and why the keep them was necessary here."}, {"start": 2842.0, "end": 2849.0, "text": " Okay, so the reason to do for this is I'm trying to hint it here when I was giving you a bit of a hint on how this works."}, {"start": 2849.0, "end": 2866.0, "text": " This 27 vector internally inside the broadcasting, this becomes a 1 by 27 and 1 by 27 is a row vector. Right? And now we are dividing 27 by 27 by 1 by 27 and torch will replicate this dimension."}, {"start": 2866.0, "end": 2880.0, "text": " So basically it will take this row vector and it will copy it vertically now 27 times so the 27 by 27 lies exactly and element wise divides."}, {"start": 2880.0, "end": 2889.0, "text": " And so basically what's happening here is we're actually normalizing the columns instead of normalizing the rows."}, {"start": 2889.0, "end": 2903.0, "text": " So you can check what's happening here is that p at 0 which is the first row of p dot sum is not 1 it's 7. It is the first column as an example that sums to 1."}, {"start": 2903.0, "end": 2916.0, "text": " So to summarize where does the issue come from? The issue comes from the silent adding of the dimension here because in broadcasting rules you align on the right and go from right to left and if the dimension doesn't exist you create it."}, {"start": 2916.0, "end": 2926.0, "text": " So that's where the problem happens. We still did the counts correctly. We did the counts across the rows and we got the counts on the right here as a column vector."}, {"start": 2926.0, "end": 2937.0, "text": " But because the key things was true this dimension was discarded and now we just have a vector 27. And because of broadcasting the way it works this vector of 27 suddenly becomes a row vector."}, {"start": 2937.0, "end": 2947.0, "text": " And then this row vector gets replicated vertically and that every single point we are dividing by the count in the opposite direction."}, {"start": 2947.0, "end": 2954.0, "text": " So this thing just doesn't work. This needs to be keep them as true in this case."}, {"start": 2954.0, "end": 2964.0, "text": " So then then we have that p at 0 is normalized. And conversely the first column you'd expect to potentially not be normalized."}, {"start": 2964.0, "end": 2975.0, "text": " And this is what makes it work. So pretty subtle and hopefully this helps to scare you that you should have respect for broadcasting. Be careful."}, {"start": 2975.0, "end": 2985.0, "text": " Check your work and understand how it works under the hood and make sure that it's broadcasting in the direction that you like. Otherwise you're going to introduce very subtle bugs, very hard to find bugs."}, {"start": 2985.0, "end": 2994.0, "text": " And just be careful. One more note on efficiency. We don't want to be doing this here because this creates a completely new tensor that we store into p."}, {"start": 2994.0, "end": 3004.0, "text": " We prefer to use in place operations if possible. So this would be an in place operation has the potential to be faster. It doesn't create new memory under the hood."}, {"start": 3004.0, "end": 3016.0, "text": " And then let's erase this. We don't need it. And let's also just do fewer. Just so I'm not wasting space. Okay. So we're actually in the pre-good spot now."}, {"start": 3016.0, "end": 3027.0, "text": " We trained a bi-gram language model and we trained it really just by counting how frequently any pairing occurs and then normalizing so that we get a nice property distribution."}, {"start": 3027.0, "end": 3036.0, "text": " So really these elements of this array p are really the parameters of our bi-gram language model given us and summarizing the statistics of these diagrams."}, {"start": 3036.0, "end": 3046.0, "text": " So we trained a model and then we know how to sample from a model. We just iteratively sampled the next character and feed it in each time and get a next character."}, {"start": 3046.0, "end": 3058.0, "text": " Now what I'd like to do is I'd like to somehow evaluate the quality of this model. We'd like to somehow summarize the quality of this model into a single number. How good is it at predicting the training set?"}, {"start": 3058.0, "end": 3071.0, "text": " And as an example, so in the training set, we can evaluate now the training loss and this training loss is telling us about sort of the quality of this model in a single number just like we saw in micrograd."}, {"start": 3071.0, "end": 3083.0, "text": " So let's try to think through the quality of the model and how we would evaluate it. Basically what we're going to do is we're going to copy paste this code that we previously used for counting."}, {"start": 3083.0, "end": 3095.0, "text": " Okay. And let me just print these bi-grams first. We're going to use f strings and I'm going to print character one followed by character two. These are the bi-grams. And then I didn't want to do it for all the words. Let's just do first three words."}, {"start": 3095.0, "end": 3108.0, "text": " So here we have Emma, Olivia and Eva bi-grams. Now what we'd like to do is we'd like to basically look at the probability that the model assigns to every one of these bi-grams."}, {"start": 3108.0, "end": 3116.0, "text": " So in other words, we can look at the probability, which is summarized in the matrix B of Ix1, Ix2."}, {"start": 3116.0, "end": 3129.0, "text": " And then we can print it here as probability. And because these probabilities are way too large, let me percent or call on.4F to like truncated a bit."}, {"start": 3129.0, "end": 3135.0, "text": " So what do we have here? We're looking at the probabilities that the model assigns to every one of these bi-grams in the dataset."}, {"start": 3135.0, "end": 3152.0, "text": " And so we can see some of them are 4% 3% etc. Just to have a measuring stick in our mind, by the way, we have 27 possible characters or tokens. And if everything was equally likely, then you'd expect all these probabilities to be 4% roughly."}, {"start": 3152.0, "end": 3170.0, "text": " So anything above 4% means that we've learned something useful from these bi-grams statistics. And you see that roughly some of these are 4%, but some of them are as high as 40%, 35%. And so on. So you see that the model actually assigned a pretty high probability to whatever's in the training set. And so that's a good thing."}, {"start": 3170.0, "end": 3183.0, "text": " Basically, if you have a very good model, you'd expect that these probabilities should be near one, because that means that your model is correctly predicting what's going to come next, especially in the training set where you trained your model."}, {"start": 3183.0, "end": 3192.0, "text": " So now we'd like to think about how can we summarize these probabilities into a single number that measures the quality of this model."}, {"start": 3192.0, "end": 3202.0, "text": " Now when you look at the literature into maximum likelihood estimation and statistical modeling and so on, you'll see that what's typically used here is something called the likelihood."}, {"start": 3202.0, "end": 3218.0, "text": " And the likelihood is the product of all of these probabilities. And so the product of all of these probabilities is the likelihood. And it's really telling us that the probability of the entire dataset assigned by the model that we've trained."}, {"start": 3218.0, "end": 3230.0, "text": " And that is a measure of quality. So the product of these should be as high as possible when you are training the model and when you have a good model, your product of these probabilities should be very high."}, {"start": 3230.0, "end": 3241.0, "text": " Now because the product of these probabilities and is an unwieldy thing to work with, you can see that all of them are between 0 and 1. So your product of these probabilities will be a very tiny number."}, {"start": 3241.0, "end": 3248.0, "text": " So for convenience, what people work with usually is not the likelihood, but they work with what's called the log likelihood."}, {"start": 3248.0, "end": 3255.0, "text": " So the product of these is the likelihood to get the log likelihood. We just have to take the log of the probability."}, {"start": 3255.0, "end": 3264.0, "text": " And so the log of the probability here, I have the log of x from 0 to 1. The log is a, you see here monotonic transformation of the probability."}, {"start": 3264.0, "end": 3281.0, "text": " Where if you pass in 1, you get 0. So probability 1 gets your log probability of 0. And then as you go lower and lower probability, the log will grow more and more negative until all the way to negative infinity at 0."}, {"start": 3281.0, "end": 3290.0, "text": " So here we have a log prob, which is religious, the torshtod log of probability. Let's print it out to get a sense of what that looks like."}, {"start": 3290.0, "end": 3297.0, "text": " Log prob also point for f."}, {"start": 3297.0, "end": 3303.0, "text": " So as you can see, when we plug in numbers that are very close, some of our higher numbers, we get closer and closer to 0."}, {"start": 3303.0, "end": 3310.0, "text": " And then if we plug in very bad probabilities, we get more and more negative number. That's bad."}, {"start": 3310.0, "end": 3325.0, "text": " And the reason we work with this is for large extent convenience, right, because we have mathematically that if you have some product, a times b times c, of all these probabilities, right, the likelihood is the product of all these probabilities."}, {"start": 3325.0, "end": 3340.0, "text": " Then the log of these is just log of a plus log of b plus log of c. If you remember your logs from your high school or undergrad and so on."}, {"start": 3340.0, "end": 3349.0, "text": " So we have that basically the likelihood of the product probabilities, the what likelihood is just the sum of the logs of the individual probabilities."}, {"start": 3349.0, "end": 3360.0, "text": " So log likelihood starts at 0 and then log likelihood here, we can just accumulate simply."}, {"start": 3360.0, "end": 3374.0, "text": " And then the end we can print this print the log likelihood, f strings, maybe you're familiar with this."}, {"start": 3374.0, "end": 3381.0, "text": " So what likelihood is negative 38."}, {"start": 3381.0, "end": 3385.0, "text": " Now we actually want."}, {"start": 3385.0, "end": 3393.0, "text": " So how high can log likelihood get? It can go to zero. So when all the probabilities are one log likelihood to zero."}, {"start": 3393.0, "end": 3397.0, "text": " And then when all the probabilities are lower, this will grow more and more negative."}, {"start": 3397.0, "end": 3408.0, "text": " Now we don't actually like this because what we'd like is a loss function and a loss function has the semantics that low is good because we're trying to minimize the loss."}, {"start": 3408.0, "end": 3416.0, "text": " So we actually need to invert this and that's what gives us something called the negative log likelihood."}, {"start": 3416.0, "end": 3424.0, "text": " Negative log likelihood is just negative of the log likelihood."}, {"start": 3424.0, "end": 3429.0, "text": " These are f strings, by the way, if you'd like to look this up, negative log likelihood equals."}, {"start": 3429.0, "end": 3432.0, "text": " So the negative log likelihood now is just negative of it."}, {"start": 3432.0, "end": 3439.0, "text": " And so the negative log likelihood is a very nice loss function because the lowest it can get is zero."}, {"start": 3439.0, "end": 3444.0, "text": " And the higher it is, the worse off the predictions are that you're making."}, {"start": 3444.0, "end": 3453.0, "text": " And then one more modification to this that sometimes people do is that for convenience, they actually like to normalize by they like to make it an average instead of a sum."}, {"start": 3453.0, "end": 3458.0, "text": " And so here, let's just keep some counts as well."}, {"start": 3458.0, "end": 3461.0, "text": " So n plus equals one starts at zero."}, {"start": 3461.0, "end": 3467.0, "text": " And then here we can have sort of like a normalized log likelihood."}, {"start": 3467.0, "end": 3474.0, "text": " If we just normalize it by the count, then we will sort of get the average log likelihood."}, {"start": 3474.0, "end": 3481.0, "text": " So this would be usually our loss function here is this we would this is what we would use."}, {"start": 3481.0, "end": 3486.0, "text": " So our loss function for the training set assigned by the model is 2.4."}, {"start": 3486.0, "end": 3488.0, "text": " That's the quality of this model."}, {"start": 3488.0, "end": 3491.0, "text": " And the lower it is, the better off we are."}, {"start": 3491.0, "end": 3493.0, "text": " And the higher it is, the worse off we are."}, {"start": 3493.0, "end": 3502.0, "text": " And the job of our training is to find the parameters that minimize the negative log likelihood loss."}, {"start": 3502.0, "end": 3505.0, "text": " And that would be like a high quality model."}, {"start": 3505.0, "end": 3508.0, "text": " Okay, so to summarize, I actually wrote it out here."}, {"start": 3508.0, "end": 3515.0, "text": " So our goal is to maximize likelihood, which is the product of all the probabilities assigned by the model."}, {"start": 3515.0, "end": 3519.0, "text": " And we want to maximize this likelihood with respect to the model parameters."}, {"start": 3519.0, "end": 3523.0, "text": " And in our case, the model parameters here are defined in the table."}, {"start": 3523.0, "end": 3530.0, "text": " These numbers, the probabilities are the model parameters, sort of in our background language model so far."}, {"start": 3530.0, "end": 3534.0, "text": " But you have to keep in mind that here we are storing everything in a table format, the probabilities."}, {"start": 3534.0, "end": 3540.0, "text": " But what's coming up as a brief preview is that these numbers will not be kept explicitly."}, {"start": 3540.0, "end": 3543.0, "text": " But these numbers will be calculated by neural network."}, {"start": 3543.0, "end": 3544.0, "text": " So that's coming up."}, {"start": 3544.0, "end": 3548.0, "text": " And we want to change and tune the parameters of these neural networks."}, {"start": 3548.0, "end": 3553.0, "text": " We want to change these parameters to maximize the likelihood, the product of the probabilities."}, {"start": 3553.0, "end": 3559.0, "text": " Now, maximizing the likelihood is equivalent to maximizing the log likelihood because log is a monotonic function."}, {"start": 3559.0, "end": 3566.0, "text": " Here's the graph of log. And basically, all it is doing is it's just scaling your..."}, {"start": 3566.0, "end": 3569.0, "text": " You can look at it as just a scaling of the loss function."}, {"start": 3569.0, "end": 3575.0, "text": " And so the optimization problem here and here are actually equivalent because this is just a scaling."}, {"start": 3575.0, "end": 3577.0, "text": " You can look at it that way."}, {"start": 3577.0, "end": 3580.0, "text": " And so these are two identical optimization problems."}, {"start": 3580.0, "end": 3585.0, "text": " Maximizing the log likelihood is equivalent to minimizing the negative log likelihood."}, {"start": 3585.0, "end": 3592.0, "text": " And then in practice, people actually minimize the average negative log likelihood to get numbers like 2.4."}, {"start": 3592.0, "end": 3596.0, "text": " And then this summarizes the quality of your model."}, {"start": 3596.0, "end": 3599.0, "text": " And we'd like to minimize it and make it as small as possible."}, {"start": 3599.0, "end": 3602.0, "text": " And the lowest it can get is zero."}, {"start": 3602.0, "end": 3609.0, "text": " And the lower it is, the better off your model is because it's signing high probabilities to your data."}, {"start": 3609.0, "end": 3614.0, "text": " Now let's estimate the probability over the entire training set just to make sure that we get something around 2.4."}, {"start": 3614.0, "end": 3616.0, "text": " Let's run this over the entire..."}, {"start": 3616.0, "end": 3620.0, "text": " Oops! Let's take out the print statement as well."}, {"start": 3620.0, "end": 3624.0, "text": " Okay, 2.45 or the entire training set."}, {"start": 3624.0, "end": 3628.0, "text": " Now what I'd like to show you is that you can actually evaluate the probability for any word that you want."}, {"start": 3628.0, "end": 3635.0, "text": " Like for example, if we just test a single word, Andre, and bring back the print statement,"}, {"start": 3635.0, "end": 3639.0, "text": " then you see that Andre is actually kind of like an unlikely word."}, {"start": 3639.0, "end": 3644.0, "text": " Like on average, we take three log probability to represent it."}, {"start": 3644.0, "end": 3650.0, "text": " And roughly that's because EJ apparently is very uncommon as an example."}, {"start": 3650.0, "end": 3653.0, "text": " Now, think through this."}, {"start": 3653.0, "end": 3660.0, "text": " I'm going to take Andre and I append Q, and I test the probability of it. Andre Q."}, {"start": 3660.0, "end": 3663.0, "text": " We actually get infinity."}, {"start": 3663.0, "end": 3669.0, "text": " And that's because JQ has a 0% probability according to our model. So the log likelihood..."}, {"start": 3669.0, "end": 3674.0, "text": " So the log of 0 will be negative infinity. We get infinite loss."}, {"start": 3674.0, "end": 3679.0, "text": " So this is kind of undesirable, right? Because we plugged in a string that could be like a somewhat reasonable name."}, {"start": 3679.0, "end": 3686.0, "text": " But basically what this is saying is that this model is exactly 0% likely to predict this name."}, {"start": 3686.0, "end": 3690.0, "text": " And our loss is infinity on this example."}, {"start": 3690.0, "end": 3698.0, "text": " And really, the reason for that is that J is followed by Q 0 times, where is Q?"}, {"start": 3698.0, "end": 3702.0, "text": " JQ is 0. And so JQ is 0% likely."}, {"start": 3702.0, "end": 3705.0, "text": " So this is actually kind of gross and people don't like this too much."}, {"start": 3705.0, "end": 3710.0, "text": " To fix this, there's a very simple fix that people like to do to sort of smooth out your model a little bit."}, {"start": 3710.0, "end": 3712.0, "text": " And it's called model smoothing."}, {"start": 3712.0, "end": 3716.0, "text": " And roughly what's happening is that we will add some eight counts."}, {"start": 3716.0, "end": 3721.0, "text": " So imagine adding a count of one to everything."}, {"start": 3721.0, "end": 3727.0, "text": " So we add a count of one like this. And then we recalculate the probabilities."}, {"start": 3727.0, "end": 3730.0, "text": " And that's model smoothing. And you can add as much as you like."}, {"start": 3730.0, "end": 3733.0, "text": " You can add five and that will give you a smoother model."}, {"start": 3733.0, "end": 3737.0, "text": " And the more you add here, the more uniform model you're going to have."}, {"start": 3737.0, "end": 3741.0, "text": " And the less you add, the more peaked model you're going to have."}, {"start": 3741.0, "end": 3746.0, "text": " Of course, so one is like a pretty decent count to add."}, {"start": 3746.0, "end": 3751.0, "text": " And that will ensure that there will be no zeros in our probability matrix P."}, {"start": 3751.0, "end": 3754.0, "text": " And so this will of course change the generations a little bit."}, {"start": 3754.0, "end": 3757.0, "text": " In this case, it didn't but it in principle it could."}, {"start": 3757.0, "end": 3761.0, "text": " But what that's going to do now is that nothing will be infinity unlikely."}, {"start": 3761.0, "end": 3765.0, "text": " So now our model will predict some other probability."}, {"start": 3765.0, "end": 3768.0, "text": " And we see that JQ now has a very small probability."}, {"start": 3768.0, "end": 3772.0, "text": " So the model still finds it's very surprising that this was a word or a by gram."}, {"start": 3772.0, "end": 3774.0, "text": " But we don't get negative infinity."}, {"start": 3774.0, "end": 3777.0, "text": " So it's kind of like a nice fix that people like to apply sometimes and it's called models moving."}, {"start": 3777.0, "end": 3782.0, "text": " Okay, so we've now trained a respectable by gram character level language model."}, {"start": 3782.0, "end": 3788.0, "text": " And we saw that we both sort of trained the model by looking at the counts of all the by grams."}, {"start": 3788.0, "end": 3792.0, "text": " And normalizing the rows to get probability distributions."}, {"start": 3792.0, "end": 3800.0, "text": " So we saw that we can also then use those parameters of this model to perform sampling of new words."}, {"start": 3800.0, "end": 3803.0, "text": " So we sample new names according to those distributions."}, {"start": 3803.0, "end": 3806.0, "text": " And we also saw that we can evaluate the quality of this model."}, {"start": 3806.0, "end": 3810.0, "text": " And the quality of this model is summarized in a single number, which is the negative log likelihood."}, {"start": 3810.0, "end": 3820.0, "text": " And the lower this number is the better the model is because it is giving high probabilities to the actual next characters in all the by grams in our training set."}, {"start": 3820.0, "end": 3822.0, "text": " So that's all well and good."}, {"start": 3822.0, "end": 3826.0, "text": " But we've arrived at this model explicitly by doing something that felt sensible."}, {"start": 3826.0, "end": 3828.0, "text": " We were just performing counts."}, {"start": 3828.0, "end": 3831.0, "text": " And then we were normalizing those counts."}, {"start": 3831.0, "end": 3834.0, "text": " Now what I would like to do is I would like to take an alternative approach."}, {"start": 3834.0, "end": 3838.0, "text": " We will end up in a very, very similar position, but the approach will look very different."}, {"start": 3838.0, "end": 3844.0, "text": " Because I would like to cast the problem of by gram character level language modeling into the neural network framework."}, {"start": 3844.0, "end": 3850.0, "text": " And in neural network framework, we're going to approach things slightly differently, but again, end up in a very similar spot."}, {"start": 3850.0, "end": 3852.0, "text": " I'll go into that later."}, {"start": 3852.0, "end": 3857.0, "text": " Now, our neural network is going to be a still a by gram character level language model."}, {"start": 3857.0, "end": 3860.0, "text": " So it receives a single character as an input."}, {"start": 3860.0, "end": 3864.0, "text": " Then there's neural network with some weights or some parameters w."}, {"start": 3864.0, "end": 3869.0, "text": " And it's going to output the probability distribution over the next character in a sequence."}, {"start": 3869.0, "end": 3875.0, "text": " It's going to make guesses as to what is likely to follow this character that was input to the model."}, {"start": 3875.0, "end": 3881.0, "text": " And then in addition to that, we're going to be able to evaluate any setting of the parameters of the neural net,"}, {"start": 3881.0, "end": 3885.0, "text": " because we have a loss function, the negative lot likelihood."}, {"start": 3885.0, "end": 3889.0, "text": " So we're going to take a look at its probability distributions, and we're going to use the labels,"}, {"start": 3889.0, "end": 3894.0, "text": " which are basically just the identity of the next character in that by gram, the second character."}, {"start": 3894.0, "end": 3903.0, "text": " So knowing what the second character actually comes next in the by gram allows us to then look at what, how high of probability the model assigns to that character."}, {"start": 3903.0, "end": 3906.0, "text": " And then we of course want the probability to be very high."}, {"start": 3906.0, "end": 3910.0, "text": " And that is another way of saying that the loss is low."}, {"start": 3910.0, "end": 3915.0, "text": " So we're going to use gradient based optimization then to tune the parameters of this network,"}, {"start": 3915.0, "end": 3918.0, "text": " because we have the loss function, and we're going to minimize it."}, {"start": 3918.0, "end": 3924.0, "text": " So we're going to tune the weights so that the neural net is correctly predicting the probabilities for the next character."}, {"start": 3924.0, "end": 3926.0, "text": " So let's get started."}, {"start": 3926.0, "end": 3929.0, "text": " The first thing I want to do is I want to compile the training set of this neural network."}, {"start": 3929.0, "end": 3936.0, "text": " So create the training set of all the by grams."}, {"start": 3936.0, "end": 3947.0, "text": " And here I'm going to copy paste this code, because this code iterates over all the by grams."}, {"start": 3947.0, "end": 3950.0, "text": " So here we start with the words, we iterate over all the by grams."}, {"start": 3950.0, "end": 3953.0, "text": " And previously, as you recall, we did the counts."}, {"start": 3953.0, "end": 3956.0, "text": " But now we're not going to do counts. We're just creating a training set."}, {"start": 3956.0, "end": 3962.0, "text": " Now this training set will be made up of two lists."}, {"start": 3962.0, "end": 3969.0, "text": " We have the inputs and the targets, the labels."}, {"start": 3969.0, "end": 3973.0, "text": " And these by grams will denote x, y. Those are the characters, right?"}, {"start": 3973.0, "end": 3977.0, "text": " And so we're given the first character of the by gram, and then we're trying to predict the next one."}, {"start": 3977.0, "end": 3979.0, "text": " Both of these are going to be integers."}, {"start": 3979.0, "end": 3987.0, "text": " So here we'll take x's that append is just x1, y's that append, x2."}, {"start": 3987.0, "end": 3991.0, "text": " And then here we actually don't want lists of integers."}, {"start": 3991.0, "end": 3994.0, "text": " We will create tensors out of these."}, {"start": 3994.0, "end": 4001.0, "text": " So x's is torched up tensor of x's, and y's is torched up tensor of y's."}, {"start": 4001.0, "end": 4004.0, "text": " And then we don't actually want to take all the words just yet,"}, {"start": 4004.0, "end": 4007.0, "text": " because I want everything to be manageable."}, {"start": 4007.0, "end": 4011.0, "text": " So let's just do the first word, which is m, i."}, {"start": 4011.0, "end": 4015.0, "text": " And then it's clear what these x's and y's would be."}, {"start": 4015.0, "end": 4019.0, "text": " Here, let me print character one character two."}, {"start": 4019.0, "end": 4022.0, "text": " Just so you see what's going on here."}, {"start": 4022.0, "end": 4028.0, "text": " So the by grams of these characters is dot e, e m, m, m, a dot."}, {"start": 4028.0, "end": 4034.0, "text": " So this single word, as I mentioned, has one, two, three, four, five examples for our neural network."}, {"start": 4034.0, "end": 4039.0, "text": " There are five separate examples in m, and those examples I show my is here."}, {"start": 4039.0, "end": 4042.0, "text": " When the input to the neural neural network is integer zero,"}, {"start": 4042.0, "end": 4047.0, "text": " the desired label is integer five, which corresponds to e."}, {"start": 4047.0, "end": 4054.0, "text": " When the input to the neural network is five, we want its weights to be arranged so that 13 gets a very high probability."}, {"start": 4054.0, "end": 4058.0, "text": " When 13 is put in, we want 13 to have a high probability."}, {"start": 4058.0, "end": 4062.0, "text": " When 13 is put in, we also want one to have a high probability."}, {"start": 4062.0, "end": 4066.0, "text": " When one is input, we want zero to have a very high probability."}, {"start": 4066.0, "end": 4072.0, "text": " So there are five separate input examples to a neural net in this data set."}, {"start": 4074.0, "end": 4080.0, "text": " I wanted to add a tangent of a note of caution to be careful with a lot of the APIs of some of these frameworks."}, {"start": 4080.0, "end": 4087.0, "text": " You saw me silently use torched dot tensor with a lowercase t, and the output looked right."}, {"start": 4087.0, "end": 4091.0, "text": " But you should be aware that there's actually two ways of constructing a tensor."}, {"start": 4091.0, "end": 4098.0, "text": " There's a torched dot lowercase tensor, and there's also a torched dot capital tensor class, which you can also construct."}, {"start": 4098.0, "end": 4100.0, "text": " So you can actually call both."}, {"start": 4100.0, "end": 4105.0, "text": " You can also do torched dot capital tensor, and you get a nexus and wise as well."}, {"start": 4105.0, "end": 4107.0, "text": " So that's not confusing at all."}, {"start": 4107.0, "end": 4111.0, "text": " There are threads on what is the difference between these two."}, {"start": 4111.0, "end": 4116.0, "text": " And unfortunately, the docs are just like not clear on the difference."}, {"start": 4116.0, "end": 4123.0, "text": " And when you look at the docs of lowercase tensor, construct tensor with no undergrad history by copying data."}, {"start": 4123.0, "end": 4126.0, "text": " It's just like it doesn't, it doesn't make sense."}, {"start": 4126.0, "end": 4131.0, "text": " So the actual difference, as far as I can tell, is explained eventually in this random thread that you can google."}, {"start": 4131.0, "end": 4138.0, "text": " And really, it comes down to, I believe, that, where is this?"}, {"start": 4138.0, "end": 4144.0, "text": " Torched dot tensor refers to the d type, the data type automatically, while torched dot tensor just returns a flow tensor."}, {"start": 4144.0, "end": 4147.0, "text": " I would recommend to stick to torched dot lowercase tensor."}, {"start": 4147.0, "end": 4157.0, "text": " So, indeed, we see that when I construct this with a capital T, the data type here of x is flow 32."}, {"start": 4157.0, "end": 4166.0, "text": " But torched dot lowercase tensor, you see how it's now x dot d type is now integer."}, {"start": 4166.0, "end": 4174.0, "text": " So, it's advised that you use lowercase t, and you can read more about it if you like in some of these threads."}, {"start": 4174.0, "end": 4182.0, "text": " But basically, I'm pointing out some of these things because I want to caution you, and I want you to get used to reading a lot of documentation,"}, {"start": 4182.0, "end": 4187.0, "text": " and reading through a lot of Q and A's and threads like this."}, {"start": 4187.0, "end": 4192.0, "text": " And, you know, some of the stuff is unfortunately not easy and not very well documented, and you have to be careful out there."}, {"start": 4192.0, "end": 4197.0, "text": " What we want here is integers, because that's what makes sense."}, {"start": 4197.0, "end": 4201.0, "text": " And so, lowercase tensor is what we are using."}, {"start": 4201.0, "end": 4206.0, "text": " Okay, now we want to think through how we're going to feed in these examples into a neural network."}, {"start": 4206.0, "end": 4212.0, "text": " Now, it's not quite as straightforward as plugging it in, because these examples right now are integers."}, {"start": 4212.0, "end": 4220.0, "text": " So, there's like a 0, 5, or 13. It gives us the index of the character, and you can't just plug an integer index into a neural net."}, {"start": 4220.0, "end": 4227.0, "text": " These neural nets are sort of made up of these neurons, and these neurons have weights."}, {"start": 4227.0, "end": 4234.0, "text": " And, as you saw in micrograd, these weights act multiplicatively on the inputs, wx, plus b, there's 10HS, and so on."}, {"start": 4234.0, "end": 4241.0, "text": " And so, it doesn't really make sense to make an input neuron take on integer values that you feed in, and then multiply on with weights."}, {"start": 4241.0, "end": 4247.0, "text": " So, instead, a common way of encoding integers is what's called one-hot encoding."}, {"start": 4247.0, "end": 4257.0, "text": " In one-hot encoding, we take an integer like 13, and we create a vector that is all zeros, except for the 13th dimension, which we turn to a 1."}, {"start": 4257.0, "end": 4261.0, "text": " And then that vector can feed into a neural net."}, {"start": 4261.0, "end": 4270.0, "text": " Now, conveniently, PyTorch actually has something called the one-hot function inside Torch and in Functional."}, {"start": 4270.0, "end": 4275.0, "text": " It takes a tensor made up of integers."}, {"start": 4275.0, "end": 4287.0, "text": " Long is an integer, and it also takes a number of classes, which is how large you want your vector to be."}, {"start": 4287.0, "end": 4294.0, "text": " So, here, let's import Torch. and end.functional.sf. This is a common way of importing it."}, {"start": 4294.0, "end": 4300.0, "text": " And then let's do f.1 hot, and we feed in the integers that we want to encode."}, {"start": 4300.0, "end": 4307.0, "text": " So, we can actually feed in the entire array of x's, and we can tell that num class is 27."}, {"start": 4307.0, "end": 4314.0, "text": " So, it doesn't have to try to guess it. It may have guessed that it's only 13, and would give us an incorrect result."}, {"start": 4314.0, "end": 4321.0, "text": " So, this is the one-hot. Let's call this x-inc for x-incoded."}, {"start": 4321.0, "end": 4326.0, "text": " And then we see that x-incoded.shape is 5x27."}, {"start": 4326.0, "end": 4335.0, "text": " And we can also visualize it at plt.imShowofxinc to make it a little bit more clear because this is a little messy."}, {"start": 4335.0, "end": 4340.0, "text": " So, we see that we've encoded all the five examples into vectors."}, {"start": 4340.0, "end": 4346.0, "text": " We have five examples, so we have five rows, and each row here is now an example into a neural mat."}, {"start": 4346.0, "end": 4351.0, "text": " And we see that the appropriate bit is turned on as a one, and everything else is zero."}, {"start": 4351.0, "end": 4358.0, "text": " So, here, for example, the zero-th bit is turned on, the fifth bit is turned on,"}, {"start": 4358.0, "end": 4364.0, "text": " 13th bits are turned on for both of these examples, and then the first bit here is turned on."}, {"start": 4364.0, "end": 4372.0, "text": " So, that's how we can encode integers into vectors, and then these vectors can feed in to neural mats."}, {"start": 4372.0, "end": 4377.0, "text": " One more issue to be careful with here, by the way, is let's look at the data type of echoing."}, {"start": 4377.0, "end": 4383.0, "text": " We always want to be careful with data types. What would you expect x-incodings data type to be?"}, {"start": 4383.0, "end": 4386.0, "text": " When we're plugging numbers into neural mats, we don't want them to be integers."}, {"start": 4386.0, "end": 4390.0, "text": " We want them to be floating point numbers that can take on various values."}, {"start": 4390.0, "end": 4394.0, "text": " But the D-type here is actually 64-bit integer."}, {"start": 4394.0, "end": 4399.0, "text": " And the reason for that, I suspect, is that one hot received a 64-bit integer here,"}, {"start": 4399.0, "end": 4402.0, "text": " and it returned to the same data type."}, {"start": 4402.0, "end": 4408.0, "text": " And when you look at the signature of one hot, it doesn't even take a desired data type of the output tensor."}, {"start": 4408.0, "end": 4414.0, "text": " And so, we can't, in a lot of functions in tortuary, be able to do something like D-type equals torshtotflot32,"}, {"start": 4414.0, "end": 4418.0, "text": " which is what we want, but one hot does not support that."}, {"start": 4418.0, "end": 4423.0, "text": " So instead, we're going to want to cast this to float like this."}, {"start": 4423.0, "end": 4426.0, "text": " So that these, everything is the same."}, {"start": 4426.0, "end": 4430.0, "text": " Everything looks the same, but the D-type is float32."}, {"start": 4430.0, "end": 4433.0, "text": " And floats can feed into neural mats."}, {"start": 4433.0, "end": 4436.0, "text": " So now let's construct our first neuron."}, {"start": 4436.0, "end": 4440.0, "text": " This neuron will look at these input vectors."}, {"start": 4440.0, "end": 4446.0, "text": " And as you remember from micrograd, these neurons basically perform a very simple function, Wx plus B,"}, {"start": 4446.0, "end": 4449.0, "text": " where Wx is a dot product."}, {"start": 4449.0, "end": 4452.0, "text": " So we can achieve the same thing here."}, {"start": 4452.0, "end": 4455.0, "text": " Let's first define the weights of this neuron, basically."}, {"start": 4455.0, "end": 4459.0, "text": " We're the initial weights at initialization for this neuron."}, {"start": 4459.0, "end": 4461.0, "text": " Let's initialize them with torshtotrandin."}, {"start": 4461.0, "end": 4469.0, "text": " Torshtotrandin is fills a tensor with random numbers, drawn from a normal distribution."}, {"start": 4469.0, "end": 4474.0, "text": " And a normal distribution has a probability density function like this."}, {"start": 4474.0, "end": 4479.0, "text": " And so most of the numbers drawn from this distribution will be around zero,"}, {"start": 4479.0, "end": 4482.0, "text": " but some of them will be as high as almost three and so on."}, {"start": 4482.0, "end": 4486.0, "text": " And very few numbers will be above three in magnitude."}, {"start": 4486.0, "end": 4490.0, "text": " So we need to take a size as an input here."}, {"start": 4490.0, "end": 4494.0, "text": " And I'm going to use size as to be 27 by one."}, {"start": 4494.0, "end": 4498.0, "text": " So 27 by one, and then let's visualize W."}, {"start": 4498.0, "end": 4503.0, "text": " So W is a column vector of 27 numbers."}, {"start": 4503.0, "end": 4508.0, "text": " And these weights are then multiplied by the inputs."}, {"start": 4508.0, "end": 4512.0, "text": " So now to perform this multiplication, we can take x encoding,"}, {"start": 4512.0, "end": 4515.0, "text": " and we can multiply it with W."}, {"start": 4515.0, "end": 4520.0, "text": " This is a matrix multiplication operator in PyTorch."}, {"start": 4520.0, "end": 4523.0, "text": " And the output of this operation is 5 by 1."}, {"start": 4523.0, "end": 4526.0, "text": " The reason it's 5 by 5 is the following."}, {"start": 4526.0, "end": 4529.0, "text": " We took x encoding, which is 5 by 27,"}, {"start": 4529.0, "end": 4533.0, "text": " and we multiplied it by 27 by 1."}, {"start": 4533.0, "end": 4540.0, "text": " And in matrix multiplication, you see that the output will become 5 by 1,"}, {"start": 4540.0, "end": 4545.0, "text": " because these 27 will multiply and add."}, {"start": 4545.0, "end": 4549.0, "text": " So basically what we're seeing here out of this operation"}, {"start": 4549.0, "end": 4558.0, "text": " is we are seeing the five activations of this neuron on these five inputs."}, {"start": 4558.0, "end": 4561.0, "text": " And we've evaluated all of them in parallel."}, {"start": 4561.0, "end": 4564.0, "text": " We didn't feed in just a single input to the single neuron."}, {"start": 4564.0, "end": 4568.0, "text": " We fed in simultaneously all the five inputs into the same neuron."}, {"start": 4568.0, "end": 4573.0, "text": " And in parallel, PyTorch has evaluated the Wx plus B,"}, {"start": 4573.0, "end": 4576.0, "text": " but here is just Wx. There's no bias."}, {"start": 4576.0, "end": 4581.0, "text": " It is value W times x for all of them, independently."}, {"start": 4581.0, "end": 4584.0, "text": " Now instead of a single neuron, though, I would like to have 27 neurons."}, {"start": 4584.0, "end": 4587.0, "text": " And I'll show you in a second why I've gone 27 neurons."}, {"start": 4587.0, "end": 4592.0, "text": " So instead of having just a one here, which is indicating this presence of one single neuron,"}, {"start": 4592.0, "end": 4594.0, "text": " we can use 27."}, {"start": 4594.0, "end": 4597.0, "text": " And then when W is 27 by 27,"}, {"start": 4597.0, "end": 4606.0, "text": " this will in parallel evaluate all the 27 neurons on all five inputs,"}, {"start": 4606.0, "end": 4609.0, "text": " giving us a much better, much, much bigger result."}, {"start": 4609.0, "end": 4613.0, "text": " So now what we've done is 5 by 27 multiplied, 27 by 27,"}, {"start": 4613.0, "end": 4617.0, "text": " and the output of this is now 5 by 27."}, {"start": 4617.0, "end": 4623.0, "text": " So we can see that the shape of this is 5 by 27."}, {"start": 4623.0, "end": 4626.0, "text": " So what is every element here telling us?"}, {"start": 4626.0, "end": 4632.0, "text": " It's telling us for every one of 27 neurons that we created."}, {"start": 4632.0, "end": 4639.0, "text": " What is the firing rate of those neurons on every one of those five examples?"}, {"start": 4639.0, "end": 4651.0, "text": " So the element, for example, 3 comma 13 is giving us the firing rate of the 13th neuron looking at the third input."}, {"start": 4651.0, "end": 4658.0, "text": " And the way this was achieved is by a dot product between the third input"}, {"start": 4658.0, "end": 4664.0, "text": " and the 13th column of this W matrix here."}, {"start": 4664.0, "end": 4671.0, "text": " So using a major multiplication, we can very efficiently evaluate the dot product"}, {"start": 4671.0, "end": 4675.0, "text": " between lots of input examples in a batch."}, {"start": 4675.0, "end": 4681.0, "text": " And lots of neurons where all of those neurons have weights in the columns of those W's."}, {"start": 4681.0, "end": 4685.0, "text": " And in major multiplication, we're just doing those dot products in parallel."}, {"start": 4685.0, "end": 4692.0, "text": " Just to show you that this is the case, we can take X and we can take the third row."}, {"start": 4692.0, "end": 4697.0, "text": " And we can take the W and take its 13th column."}, {"start": 4697.0, "end": 4708.0, "text": " And then we can do X and get 3, element wise multiply with W at 13 and sum that up."}, {"start": 4708.0, "end": 4713.0, "text": " This WX plus B. Well, there's no plus B. It's just WX dot product."}, {"start": 4713.0, "end": 4715.0, "text": " And that's this number."}, {"start": 4715.0, "end": 4720.0, "text": " So you see that this is just being done efficiently by the matrix multiplication operation"}, {"start": 4720.0, "end": 4726.0, "text": " for all the input examples and for all the output neurons of this first layer."}, {"start": 4726.0, "end": 4732.0, "text": " Okay, so we fed our 27 dimensional inputs into a first layer of a neural net that has 27 neurons."}, {"start": 4732.0, "end": 4736.0, "text": " Right? So we have 27 inputs and now we have 27 neurons."}, {"start": 4736.0, "end": 4743.0, "text": " These neurons perform W times X. They don't have a bias and they don't have a non-linearity like 10H."}, {"start": 4743.0, "end": 4746.0, "text": " We're going to leave them to be a linear layer."}, {"start": 4746.0, "end": 4750.0, "text": " In addition to that, we're not going to have any other layers. This is going to be it."}, {"start": 4750.0, "end": 4756.0, "text": " It's just going to be the dumbest, smallest, simplest neural net, which is just a single linear layer."}, {"start": 4756.0, "end": 4761.0, "text": " And now I'd like to explain what I want those 27 outputs to be."}, {"start": 4761.0, "end": 4767.0, "text": " Intuitively, what we're trying to produce here for every single input example is we're trying to produce some kind of a probability distribution"}, {"start": 4767.0, "end": 4771.0, "text": " for the next character in a sequence. And there's 27 of them."}, {"start": 4771.0, "end": 4779.0, "text": " But we have to come up with like precise semantics for exactly how we're going to interpret these 27 numbers that these neural state come on."}, {"start": 4779.0, "end": 4785.0, "text": " Now intuitively, you see here that these numbers are negative and some of them are positive, etc."}, {"start": 4785.0, "end": 4794.0, "text": " And that's because these are coming out of the neural net layer initialized with these normal distribution parameters."}, {"start": 4794.0, "end": 4801.0, "text": " But what we want is we want something like we had here, like each row here told us the counts."}, {"start": 4801.0, "end": 4806.0, "text": " And then we normalize the counts to get probabilities. And we want something similar to come out of the neural net."}, {"start": 4806.0, "end": 4810.0, "text": " But what we just have right now is just some negative and positive numbers."}, {"start": 4810.0, "end": 4815.0, "text": " Now, we want those numbers to somehow represent the probabilities for the next character."}, {"start": 4815.0, "end": 4819.0, "text": " But you see that probabilities, they have a special structure."}, {"start": 4819.0, "end": 4823.0, "text": " They're positive numbers and they sum to 1."}, {"start": 4823.0, "end": 4826.0, "text": " And so that doesn't just come out of a neural net."}, {"start": 4826.0, "end": 4833.0, "text": " And then they can't be counts because these counts are positive and counts are integers."}, {"start": 4833.0, "end": 4837.0, "text": " So counts are also not really a good thing to output from a neural net."}, {"start": 4837.0, "end": 4850.0, "text": " So instead what the neural net is going to output and how we are going to interpret the 27 numbers is that these 27 numbers are giving us log counts, basically."}, {"start": 4850.0, "end": 4856.0, "text": " So instead of giving us counts directly, lock in this table, they're giving us log counts."}, {"start": 4856.0, "end": 4861.0, "text": " And to get the counts, we're going to take the log counts and we're going to exponentially eat them."}, {"start": 4861.0, "end": 4867.0, "text": " Now, exponentiation takes the following form."}, {"start": 4867.0, "end": 4873.0, "text": " It takes numbers that are negative or they are positive. It takes the entire real line."}, {"start": 4873.0, "end": 4880.0, "text": " And then if you plug in negative numbers, you're going to get e to the x, which is always below 1."}, {"start": 4880.0, "end": 4883.0, "text": " So you're getting numbers lower than 1."}, {"start": 4883.0, "end": 4891.0, "text": " And if you plug in numbers greater than 0, you're getting numbers greater than 1 all the way growing to the infinity."}, {"start": 4891.0, "end": 4893.0, "text": " And this here grows to 0."}, {"start": 4893.0, "end": 4909.0, "text": " So basically we're going to take these numbers here. And instead of them being positive and negative in all of the place, we're going to interpret them as log counts."}, {"start": 4909.0, "end": 4913.0, "text": " And then we're going to element wise, exponentiate these numbers."}, {"start": 4913.0, "end": 4916.0, "text": " Exponentiating them now gives us something like this."}, {"start": 4916.0, "end": 4924.0, "text": " And you see that these numbers now, because they went through an exponent, all the negative numbers turned into numbers below 1, like 0,338."}, {"start": 4924.0, "end": 4930.0, "text": " And all the positive numbers originally turned into even more positive numbers, sort of greater than 1."}, {"start": 4930.0, "end": 4940.0, "text": " So like for example, 7 is some positive number over here. That is greater than 0."}, {"start": 4940.0, "end": 4951.0, "text": " But exponentiated outputs here basically give us something that we can use and interpret as the equivalent of counts originally."}, {"start": 4951.0, "end": 4956.0, "text": " So you see these counts here, 1, 12, 7, 51, 1, etc."}, {"start": 4956.0, "end": 4961.0, "text": " The neural net is kind of now predicting counts."}, {"start": 4961.0, "end": 4967.0, "text": " And these counts are positive numbers. They can never be below 0. So that makes sense."}, {"start": 4967.0, "end": 4974.0, "text": " And they can now take on various values depending on the settings of W."}, {"start": 4974.0, "end": 4981.0, "text": " So let me break this down. We're going to interpret these to be the log counts."}, {"start": 4981.0, "end": 4988.0, "text": " In other words, for this, that is often used is so called logits. These are logits log counts."}, {"start": 4988.0, "end": 5000.0, "text": " And these will be sort of the counts, logits exponentiated. And this is equivalent to the n matrix, sort of the n array that we used previously."}, {"start": 5000.0, "end": 5012.0, "text": " Remember this was the n. This is the array of counts. And each row here are the counts for the next character, sort of."}, {"start": 5012.0, "end": 5019.0, "text": " So those are the counts. And now the probabilities are just the counts normalized."}, {"start": 5019.0, "end": 5026.0, "text": " And so I'm not going to find the same. But basically I'm not going to scroll all the place."}, {"start": 5026.0, "end": 5034.0, "text": " We've already done this. We want to count that sum along the first dimension. And we want to keep them as true."}, {"start": 5034.0, "end": 5043.0, "text": " We've went over this. And this is how we normalize the rows of our counts matrix to get our probabilities."}, {"start": 5043.0, "end": 5061.0, "text": " So now these are the probabilities. And these are the counts that we have currently. And now when I show the probabilities, you see that every row here, of course, will sum to 1."}, {"start": 5061.0, "end": 5075.0, "text": " Because they're normalized. And the shape of this is 5 by 27. And so really what we've achieved is for every one of our five examples, we now have a row that came out of a neural net."}, {"start": 5075.0, "end": 5084.0, "text": " And because of the transformations here, we made sure that this output of this neural net now are probabilities, or we can interpret to be probabilities."}, {"start": 5084.0, "end": 5094.0, "text": " So our WX here gave us logits. And then we interpret those to be log counts. We exponentiate to get something that looks like counts."}, {"start": 5094.0, "end": 5100.0, "text": " And then we normalize those counts to get a probability distribution. And all of these are differentiable operations."}, {"start": 5100.0, "end": 5110.0, "text": " So what we've done now is we are taking inputs. We have differentiable operations that we can back propagate through. And we're getting out probability distributions."}, {"start": 5110.0, "end": 5128.0, "text": " So, for example, for the zero example that fed in, right, which was the zero example here was a one-half vector of zero. And it basically corresponded to feeding in this example here."}, {"start": 5128.0, "end": 5143.0, "text": " So we're feeding an adot into a neural net. And the way we fed the dot into a neural net is that we first got its index. Then we one hot encoded it. Then it went into the neural net. And out came this distribution of probabilities."}, {"start": 5143.0, "end": 5159.0, "text": " And its shape is 27, there's 27 numbers. And we're going to interpret this as the neural net's assignment for how likely every one of these characters, 27 characters are to come next."}, {"start": 5159.0, "end": 5167.0, "text": " And as we tune the weights W, we're going to be, of course, getting different probabilities out for any character that you input."}, {"start": 5167.0, "end": 5177.0, "text": " And so now the question is just, can we optimize and find a good W such that the probabilities coming out are pretty good. And the way we measure pretty good is by the loss function."}, {"start": 5177.0, "end": 5182.0, "text": " Okay, so I organized everything into a single summary so that hopefully it's a bit more clear. So it starts here."}, {"start": 5182.0, "end": 5192.0, "text": " With an input dataset, we have some inputs through the neural net. And we have some labels for the correct next character in a sequence. These are integers."}, {"start": 5192.0, "end": 5207.0, "text": " Here, I'm using torsion generators now so that you see the same numbers that I see. And I'm generating 27 neurons weights. And each neuron here receives 27 inputs."}, {"start": 5207.0, "end": 5215.0, "text": " Then here we're going to plug in all the input examples x's into a neural net. So here, this is a forward pass."}, {"start": 5215.0, "end": 5229.0, "text": " First, we have to encode all of the inputs into one hot representations. So we have 27 classes. We pass in these integers. And x-nc becomes a array that is 5 by 27."}, {"start": 5229.0, "end": 5236.0, "text": " Zeroes, except for a few ones. We then multiply this in the first layer of a neural net to get low-gets."}, {"start": 5236.0, "end": 5251.0, "text": " Expand and shade the low-gets to get fake counts, sort of, and normalize these counts to get probabilities. So these last two lines, by the way, here, are called the softmax, which I pulled up here."}, {"start": 5251.0, "end": 5263.0, "text": " Softmax is a very often used layer in a neural net that takes these z's, which are low-gets. Expand and shade them. And the rise in normalizes."}, {"start": 5263.0, "end": 5278.0, "text": " It's a way of taking outputs of a neural net layer. And these outputs can be positive or negative. And it outputs probability distributions. It outputs something that is always sums to one in our positive numbers, just like probabilities."}, {"start": 5278.0, "end": 5288.0, "text": " So it's going to look like a normalization function if you want to think of it that way. And you can put it on top of any other linear layer inside a neural net. And it basically makes a neural net output probabilities."}, {"start": 5288.0, "end": 5297.0, "text": " That's very often used. And we used it as well here. So this is the forward pass. And that's how we made a neural net output probability."}, {"start": 5297.0, "end": 5313.0, "text": " Now, you'll notice that all of these, this entire forward pass is made up of differentiable layers. Everything here, we can back propagate through. And we saw some of the back propagation in micrograd."}, {"start": 5313.0, "end": 5324.0, "text": " This is just multiplication and addition. All that's happening here is just multiply and add. And we know how to back propagate through them. Expandentiation, we know how to back propagate through."}, {"start": 5324.0, "end": 5337.0, "text": " And then here we are summing. And sum is easily back propagated as well. And division as well. So everything here is the differential operation. And we can back propagate through."}, {"start": 5337.0, "end": 5346.0, "text": " Now, we achieve these probabilities, which are five by 27 for every single example. We have a vector of probabilities that sum to one."}, {"start": 5346.0, "end": 5356.0, "text": " And then here I wrote a bunch of stuff to sort of like break down the examples. So we have five examples making up Emma, right?"}, {"start": 5356.0, "end": 5368.0, "text": " And there are five by grams inside Emma. So by gram example, a by gram example one is that E is the beginning character right after dot."}, {"start": 5368.0, "end": 5381.0, "text": " And the indexes for these are zero and five. So then we feed in a zero. That's the input as a neural net. We get probabilities from the neural net that are 27 numbers."}, {"start": 5381.0, "end": 5394.0, "text": " And then the label is five because he actually comes after dot. So that's the label. And then we use this label five to index into the probability distribution here."}, {"start": 5394.0, "end": 5404.0, "text": " So this index five here is zero one two three four five. It's this number here, which is here."}, {"start": 5404.0, "end": 5417.0, "text": " So that's basically the probability assigned by the neural net to the actual correct character. You see that the net work currently thinks that this next character that E following dot is only one percent likely, which is of course not very good, right?"}, {"start": 5417.0, "end": 5427.0, "text": " Because this actually is a training example. And the network thinks that this is currently very, very unlikely. But that's just because we didn't get very lucky in generating a good setting of W."}, {"start": 5427.0, "end": 5436.0, "text": " So right now this network thinks it's unlikely and zero point zero one is not a good outcome. So the log likelihood then is very negative."}, {"start": 5436.0, "end": 5445.0, "text": " And the negative log likelihood is very positive. And so four is a very high negative log likelihood. And that means we're going to have a high loss."}, {"start": 5445.0, "end": 5452.0, "text": " Because what is the loss? The loss is just the average negative log likelihood."}, {"start": 5452.0, "end": 5461.0, "text": " And the second character is E and you see here that also the network thought that M following E is very unlikely one percent."}, {"start": 5461.0, "end": 5472.0, "text": " Of the for M following M it thought it was 2% and for a following M it actually thought it was 7% likely. So just by chance, this one actually has a pretty good probability."}, {"start": 5472.0, "end": 5478.0, "text": " And therefore pretty low negative log likelihood. And finally here it thought this was 1% likely."}, {"start": 5478.0, "end": 5487.0, "text": " So overall our average negative log likelihood, which is the loss, the total loss that summarizes basically the how well this network currently works."}, {"start": 5487.0, "end": 5492.0, "text": " At least on this one word, not on the full data suggest the one word is 3.76."}, {"start": 5492.0, "end": 5496.0, "text": " Which is actually very fairly high loss. This is not a very good setting of W's."}, {"start": 5496.0, "end": 5501.0, "text": " Now here's what we can do. We're currently getting 3.76."}, {"start": 5501.0, "end": 5508.0, "text": " We can actually come here and we can change our W. We can resample it. So let me just add one to have a different seed."}, {"start": 5508.0, "end": 5518.0, "text": " And then we get a different W. And then we can rerun this. And with this different seed, with this different setting of W's, we now get 3.37."}, {"start": 5518.0, "end": 5528.0, "text": " So this is a much better W. And it's better because the probabilities just happen to come out higher for the characters that actually are next."}, {"start": 5528.0, "end": 5534.0, "text": " And so you can imagine actually just resampling this, you know, we can try two."}, {"start": 5534.0, "end": 5540.0, "text": " So, okay, this was not very good. Let's try one more. We can try three."}, {"start": 5540.0, "end": 5549.0, "text": " Okay, this was terrible setting because we have a very high loss. So anyway, I'm going to erase this."}, {"start": 5549.0, "end": 5559.0, "text": " What I'm doing here, which is just guess and check of randomly assigning parameters and seeing if the network is good. That is amateur hour. That's not how you optimize in your alert."}, {"start": 5559.0, "end": 5565.0, "text": " The way you optimize in your alert is you start with some random guess. And we're going to commit to this one, even though it's not very good."}, {"start": 5565.0, "end": 5568.0, "text": " But not the big deal is we have a loss function."}, {"start": 5568.0, "end": 5585.0, "text": " So this loss is made up only of the furniture operations. And we can minimize the loss by tuning W's by computing the gradients of the loss with respect to these W matrices."}, {"start": 5585.0, "end": 5593.0, "text": " And so then we can tune W to minimize the loss and find a good setting of W using gradient based optimization. So let's see how that will work."}, {"start": 5593.0, "end": 5602.0, "text": " Now, things are actually going to look almost identical to what we had with micrograd. So here I pulled up the lecture from micrograd, the notebook."}, {"start": 5602.0, "end": 5608.0, "text": " It's from this repository. And when I scroll all the way to the end where we left off with micrograd, we had something very, very similar."}, {"start": 5608.0, "end": 5617.0, "text": " We had a number of input examples. In this case, we had four input examples inside X's. And we had their targets. These are targets."}, {"start": 5617.0, "end": 5624.0, "text": " Just like here we have our X's now, but we have five of them. And they're now integers instead of vectors."}, {"start": 5624.0, "end": 5631.0, "text": " But we're going to convert our integers to vectors, except our vectors will be 27 large instead of three large."}, {"start": 5631.0, "end": 5640.0, "text": " And then here what we did is first we did a forward pass where we ran a neural net on all the inputs to get predictions."}, {"start": 5640.0, "end": 5650.0, "text": " Our neural net at the time this n effects was a net of multilayer perceptron. Our neural net is going to look different because our neural net is just a single layer."}, {"start": 5650.0, "end": 5655.0, "text": " Single linear layer followed by a softmax. So that's our neural net."}, {"start": 5655.0, "end": 5663.0, "text": " And the loss here was the mean squared error. So we simply subtracted the prediction from the ground truth and squared it and some they roll up."}, {"start": 5663.0, "end": 5676.0, "text": " And that was the loss. And loss was the single number that summarized the quality of the neural net. And when loss is low, like almost zero, that means the neural net is predicting correctly."}, {"start": 5676.0, "end": 5682.0, "text": " So we had a single number that that summarized the performance of the neural net."}, {"start": 5682.0, "end": 5687.0, "text": " And everything here was differentiable and was stored in massive compute graph."}, {"start": 5687.0, "end": 5699.0, "text": " And then we iterated over all the parameters we made sure that the gradients are set to zero. And we called lost a backward and lost a backward initiated back propagation at the final output node of loss."}, {"start": 5699.0, "end": 5706.0, "text": " Right. So yeah, remember these expressions. We had lost all the way at the end. We start that propagation and we went all the way back."}, {"start": 5706.0, "end": 5714.0, "text": " And we made sure that we populated all the parameters dot grad. So that grad started at zero, but back propagation filled it in."}, {"start": 5714.0, "end": 5727.0, "text": " And then in the update, we iterated over the parameters and we simply did a parameter update where every single element of our parameters was nudged in the opposite direction of the gradient."}, {"start": 5727.0, "end": 5738.0, "text": " And so we're going to do the exact same thing here. So I'm going to pull this up on the side here."}, {"start": 5738.0, "end": 5746.0, "text": " So that we have it available and we're actually going to do the exact same thing. So this was the forward pass. So where we did this."}, {"start": 5746.0, "end": 5752.0, "text": " And props is our white bread. So now we have to evaluate the loss, but we're not using the mean square there."}, {"start": 5752.0, "end": 5758.0, "text": " We're using the negative log likelihood because we are doing classification. We're not doing regression as it's called."}, {"start": 5758.0, "end": 5761.0, "text": " So here we want to calculate loss."}, {"start": 5761.0, "end": 5770.0, "text": " Now the way we calculated is is just this average negative log likelihood. Now this probs here."}, {"start": 5770.0, "end": 5780.0, "text": " Has a shape of five by 27. And so to get all the we basically want to pluck out the probabilities at the correct indices here."}, {"start": 5780.0, "end": 5789.0, "text": " So in particular, because the labels are stored here in the array wise, basically what we're after is for the first example, we're looking at probability of five."}, {"start": 5789.0, "end": 5800.0, "text": " Right, at the index five. For the second example at the second row or row index one, we are interested in the probability of science to index 13."}, {"start": 5800.0, "end": 5807.0, "text": " At the second example, we also have 13. At the third row, we want one."}, {"start": 5807.0, "end": 5814.0, "text": " And at the last row, which is four, we want zero. So these are the probabilities we're interested in. Right."}, {"start": 5814.0, "end": 5819.0, "text": " And you can see that they're not amazing as we saw above."}, {"start": 5819.0, "end": 5827.0, "text": " So these are the probabilities we want, but we want like a more efficient way to access these probabilities, not just listing them out in a tuple like this."}, {"start": 5827.0, "end": 5842.0, "text": " So it turns out that the way to this in PyTorch, one of the ways at least, is we can basically pass in all of these, sorry about that, all of these integers and vectors."}, {"start": 5842.0, "end": 5847.0, "text": " So the these ones, you see how they're just zero, one, two, three, four."}, {"start": 5847.0, "end": 5854.0, "text": " We can actually create that using MP, not MP, sorry, torch dot arrange of five zero, one, two, three, four."}, {"start": 5854.0, "end": 5861.0, "text": " So we can index here with torch dot arrange of five. And here we index with wise."}, {"start": 5861.0, "end": 5869.0, "text": " And you see that that gives us exactly these numbers."}, {"start": 5869.0, "end": 5876.0, "text": " So that plucks out the probabilities of that the neural network assigns to the correct next character."}, {"start": 5876.0, "end": 5883.0, "text": " Now we take those probabilities and we don't we actually look at the log probability. So we want to dot log."}, {"start": 5883.0, "end": 5894.0, "text": " And then we want to just average that up. So take the mean of all that. And then it's the negative average log likelihood that is the loss."}, {"start": 5894.0, "end": 5903.0, "text": " So the loss here is 3.7 something and you see that this loss 3.76, 3.76 is exactly as we've obtained before."}, {"start": 5903.0, "end": 5909.0, "text": " But this is a vectorized form of that expression. So we get the same loss."}, {"start": 5909.0, "end": 5916.0, "text": " And the same loss we can consider sort of as part of this forward pass and we've achieved here now loss."}, {"start": 5916.0, "end": 5924.0, "text": " Okay, so we made our way all the way to loss. We define the forward pass. We forwarded the network and the loss. Now we're ready to do backward pass."}, {"start": 5924.0, "end": 5928.0, "text": " So backward pass."}, {"start": 5928.0, "end": 5936.0, "text": " We want to first make sure that all the gradients are reset. So they're at zero. Now in pie torch, you can set the gradients to be zero."}, {"start": 5936.0, "end": 5946.0, "text": " But you can also just set it to none and setting it to none is more efficient and pie torch will interpret none as like a lack of a gradient and is the same as zeros."}, {"start": 5946.0, "end": 5950.0, "text": " So this is a way to set to zero the gradient."}, {"start": 5950.0, "end": 5954.0, "text": " And now we do lost the backward."}, {"start": 5954.0, "end": 5966.0, "text": " Before we do lost that backward, we need one more thing. If you remember from micro-grad, pie torch actually requires that we pass in requires grad is true."}, {"start": 5966.0, "end": 5974.0, "text": " So that we tell pie torch that we are interested in calculating gradient for this lead tensor by default. This is false."}, {"start": 5974.0, "end": 5981.0, "text": " So let me recalculate with that and then setting none and lost that backward."}, {"start": 5981.0, "end": 5993.0, "text": " Now something magical happened when lost the backward was run because pie torch just like micro-grad, when we did the forward pass here, it keeps track of all the operations under the hood."}, {"start": 5993.0, "end": 6001.0, "text": " It builds a full computational graph. Just like the graphs we produced in micro-grad, those graphs exist inside pie torch."}, {"start": 6001.0, "end": 6010.0, "text": " And so it knows all the dependencies and all the mathematical operations of everything. And when you then calculate the loss, we can call a dot backward on it."}, {"start": 6010.0, "end": 6020.0, "text": " And that backward then fills in the gradients of all the intermediates all the way back to W's, which are the parameters of our neural net."}, {"start": 6020.0, "end": 6029.0, "text": " So now we can do WL grad and we see that it has structure. There's stuff inside it."}, {"start": 6029.0, "end": 6040.0, "text": " And these gradients, every single element here, so W.shape is 27 by 27, W grads shape is the same, 27 by 27."}, {"start": 6040.0, "end": 6048.0, "text": " And every element of W.grad is telling us the influence of that weight on the loss function."}, {"start": 6048.0, "end": 6061.0, "text": " So for example, this number all the way here, if this element, the zero zero element of W, because the gradient is positive, it's telling us that this has a positive influence on the loss."}, {"start": 6061.0, "end": 6078.0, "text": " Slightly nudging, W slightly taking W zero zero and adding a small h to it would increase the loss, mildly, because this gradient is positive. Some of these gradients are also negative."}, {"start": 6078.0, "end": 6086.0, "text": " So that's telling us about the gradient information. And we can use this gradient information to update the weights of this neural network."}, {"start": 6086.0, "end": 6096.0, "text": " So let's not do the update. It's going to be very similar to what we had in micrograd. We need no loop over all the parameters because we only have one parameter tensor and that is W."}, {"start": 6096.0, "end": 6109.0, "text": " So we simply do W dot data plus equals. We can actually copy this almost exactly negative 0.1 times W dot grad."}, {"start": 6109.0, "end": 6119.0, "text": " And that would be the update to the tensor. So that updates the tensor."}, {"start": 6119.0, "end": 6130.0, "text": " And because the tensor is updated, we would expect that now the loss should decrease. So here, if I print loss."}, {"start": 6130.0, "end": 6145.0, "text": " It was 3.76, right? So we've updated the W here. So if I recalculate forward pass, loss now should be slightly lower. So 3.76 goes to 3.74."}, {"start": 6145.0, "end": 6155.0, "text": " And then we can again set to set grad to none and backward update. And now the parameters changed again."}, {"start": 6155.0, "end": 6162.0, "text": " So if we recalculate the forward pass, we expect a lower loss again 3.72."}, {"start": 6162.0, "end": 6175.0, "text": " And this is again doing the we're now doing reading the set. And when we achieve a low loss, that will mean that the network is assigning high probabilities to the correct next characters."}, {"start": 6175.0, "end": 6186.0, "text": " Okay, so I rearranged everything and I put it all together from scratch. So here is where we construct our data set of diagrams. You see that we are still iterating only over the first word, Emma."}, {"start": 6186.0, "end": 6196.0, "text": " I'm going to change that in a second. I added a number that counts the number of elements in axis so that we explicitly see that number of examples is 5."}, {"start": 6196.0, "end": 6200.0, "text": " Because currently we're just working with Emma. There's five bygones there."}, {"start": 6200.0, "end": 6209.0, "text": " And here I added a loop of exactly what we had before. So we had 10 iterations of very decent of forward pass, backward pass, and an update."}, {"start": 6209.0, "end": 6218.0, "text": " And so running these two cells, initialization and creating the scent, gives us some improvement on the last function."}, {"start": 6218.0, "end": 6226.0, "text": " But now I want to use all the words. And there's not five, but 228,000 bygones now."}, {"start": 6226.0, "end": 6236.0, "text": " However, this should require no modification whatsoever. Everything should just run because all the code we wrote doesn't carry their five bygones or 228,000 bygones."}, {"start": 6236.0, "end": 6240.0, "text": " And with everything we should just work. So you see that this will just run."}, {"start": 6240.0, "end": 6244.0, "text": " But now we are optimizing over the entire training set of all the bygones."}, {"start": 6244.0, "end": 6250.0, "text": " And you see now that we are decreasing very slightly. So actually we can probably afford a larger learning rate."}, {"start": 6250.0, "end": 6257.0, "text": " And probably for even larger learning rate."}, {"start": 6257.0, "end": 6263.0, "text": " Even 50 seems to work on this very, very simple example."}, {"start": 6263.0, "end": 6269.0, "text": " So let me re-enact the lies and let's run 100 iterations."}, {"start": 6269.0, "end": 6273.0, "text": " See what happens."}, {"start": 6273.0, "end": 6284.0, "text": " Okay. We seem to be coming up to some pretty good losses here. 2.47. Let me run 100 more."}, {"start": 6284.0, "end": 6287.0, "text": " What is the number that we expect by the way in the loss?"}, {"start": 6287.0, "end": 6292.0, "text": " We expect to get something around what we had originally actually."}, {"start": 6292.0, "end": 6299.0, "text": " So all the way back, if you remember, in the beginning of this video, when we optimized just by counting,"}, {"start": 6299.0, "end": 6304.0, "text": " our loss was roughly 2.47 after we had its moving."}, {"start": 6304.0, "end": 6310.0, "text": " But before its moving, we had roughly 2.45, likely it. Sorry, loss."}, {"start": 6310.0, "end": 6314.0, "text": " And so that's actually roughly the vicinity of what we expect to achieve."}, {"start": 6314.0, "end": 6319.0, "text": " But before we achieved it by counting, and here we are achieving roughly the same result,"}, {"start": 6319.0, "end": 6326.0, "text": " but with gradient-based optimization. So we come to about 2.46, 2.45, etc."}, {"start": 6326.0, "end": 6330.0, "text": " And that makes sense because fundamentally we're not taking any additional information."}, {"start": 6330.0, "end": 6333.0, "text": " We're still just taking in the previous character and trying to predict the next one,"}, {"start": 6333.0, "end": 6338.0, "text": " but instead of doing it explicitly by counting and normalizing,"}, {"start": 6338.0, "end": 6340.0, "text": " we are doing it with gradient-based learning."}, {"start": 6340.0, "end": 6345.0, "text": " And it just so happens that the explicit approach happens to very well optimize the loss function."}, {"start": 6345.0, "end": 6348.0, "text": " Without any need for a gradient-based optimization,"}, {"start": 6348.0, "end": 6353.0, "text": " because the setup for bi-gram language models is so straightforward, so simple,"}, {"start": 6353.0, "end": 6359.0, "text": " we can just afford to estimate as probably is directly and maintain them in a table."}, {"start": 6359.0, "end": 6363.0, "text": " But the gradient-based approach is significantly more flexible."}, {"start": 6363.0, "end": 6371.0, "text": " So we've actually gained a lot because what we can do now is we can expand this approach"}, {"start": 6371.0, "end": 6373.0, "text": " and complexify the neural net."}, {"start": 6373.0, "end": 6376.0, "text": " So currently we're just taking a single character and feeding into a neural net,"}, {"start": 6376.0, "end": 6378.0, "text": " and the neural is extremely simple."}, {"start": 6378.0, "end": 6380.0, "text": " But we're about to iterate on this substantially."}, {"start": 6380.0, "end": 6383.0, "text": " We're going to be taking multiple previous characters,"}, {"start": 6383.0, "end": 6387.0, "text": " and we're going to be feeding them into increasingly more complex neural nets."}, {"start": 6387.0, "end": 6392.0, "text": " But fundamentally, the output of the neural net will always just be low jits."}, {"start": 6392.0, "end": 6395.0, "text": " And those low jits will go through the exact same transformation."}, {"start": 6395.0, "end": 6399.0, "text": " We are going to take them through a softmax, calculate the loss function,"}, {"start": 6399.0, "end": 6403.0, "text": " and the negative log likelihood, and do gradient-based optimization."}, {"start": 6403.0, "end": 6407.0, "text": " And so actually, as we complexify the neural nets,"}, {"start": 6407.0, "end": 6412.0, "text": " and work all the way up to transformers, none of this will really fundamentally change."}, {"start": 6412.0, "end": 6414.0, "text": " None of this will fundamentally change."}, {"start": 6414.0, "end": 6417.0, "text": " The only thing that will change is the way we do the forward pass,"}, {"start": 6417.0, "end": 6423.0, "text": " or we've taken some previous characters and calculated logits for the next character in a sequence."}, {"start": 6423.0, "end": 6429.0, "text": " That will become more complex, and I will use the same machinery to optimize it."}, {"start": 6429.0, "end": 6435.0, "text": " And it's not obvious how we would have extended this bi-gram approach"}, {"start": 6435.0, "end": 6439.0, "text": " into the case where there are many more characters at the input."}, {"start": 6439.0, "end": 6442.0, "text": " Because eventually these tables would get way too large,"}, {"start": 6442.0, "end": 6447.0, "text": " because there's way too many combinations of what previous characters could be."}, {"start": 6447.0, "end": 6452.0, "text": " If you only have one previous character, we can just keep everything in a table, the counts."}, {"start": 6452.0, "end": 6455.0, "text": " But if you have the last 10 characters that are input,"}, {"start": 6455.0, "end": 6457.0, "text": " we can't actually keep everything in a table anymore."}, {"start": 6457.0, "end": 6460.0, "text": " So this is fundamentally an unscatable approach,"}, {"start": 6460.0, "end": 6463.0, "text": " and the neural network approach is significantly more scalable,"}, {"start": 6463.0, "end": 6467.0, "text": " and it's something that actually we can improve on over time."}, {"start": 6467.0, "end": 6469.0, "text": " So that's where we will be digging next."}, {"start": 6469.0, "end": 6471.0, "text": " I wanted to point out two more things."}, {"start": 6471.0, "end": 6476.0, "text": " Number one, I want you to notice that this x and k here,"}, {"start": 6476.0, "end": 6479.0, "text": " this is made up of one hot vectors,"}, {"start": 6479.0, "end": 6483.0, "text": " and then those one hot vectors are multiplied by this w matrix."}, {"start": 6483.0, "end": 6488.0, "text": " And we think of this as a multiple neurons being forwarded in a fully connected manner."}, {"start": 6488.0, "end": 6492.0, "text": " But actually what's happening here is that, for example,"}, {"start": 6492.0, "end": 6497.0, "text": " if you have a one hot vector here that has a one at say the fifth dimension,"}, {"start": 6497.0, "end": 6501.0, "text": " then because of the way the matrix multiplication works,"}, {"start": 6501.0, "end": 6507.0, "text": " multiplying that one hot vector with w actually ends up plucking out the fifth row of w."}, {"start": 6507.0, "end": 6511.0, "text": " A lot of logits would become just the fifth row of w,"}, {"start": 6511.0, "end": 6515.0, "text": " and that's because of the way the matrix multiplication works."}, {"start": 6515.0, "end": 6519.0, "text": " So that's actually what ends up happening."}, {"start": 6519.0, "end": 6523.0, "text": " So, but that's actually exactly what happened before."}, {"start": 6523.0, "end": 6526.0, "text": " Because remember, all the way up here,"}, {"start": 6526.0, "end": 6529.0, "text": " we have a by-gram, we took the first character,"}, {"start": 6529.0, "end": 6534.0, "text": " and then that first character indexed into a row of this array here."}, {"start": 6534.0, "end": 6538.0, "text": " And that row gave us the probability distribution for the next character."}, {"start": 6538.0, "end": 6546.0, "text": " So the first character was used as a lookup into a matrix here to get the probability distribution."}, {"start": 6546.0, "end": 6548.0, "text": " Well, that's actually exactly what's happening here."}, {"start": 6548.0, "end": 6550.0, "text": " Because we're taking the index."}, {"start": 6550.0, "end": 6553.0, "text": " We're encoding it as one hot and multiplying it by w."}, {"start": 6553.0, "end": 6560.0, "text": " So, logits literally becomes the appropriate row of w."}, {"start": 6560.0, "end": 6562.0, "text": " And that gets just as before,"}, {"start": 6562.0, "end": 6564.0, "text": " expedited to create the counts,"}, {"start": 6564.0, "end": 6567.0, "text": " and then normalized and becomes probability."}, {"start": 6567.0, "end": 6574.0, "text": " So, this w here is literally the same as this array here."}, {"start": 6574.0, "end": 6579.0, "text": " But, w, remember, is the log counts, not the counts."}, {"start": 6579.0, "end": 6582.0, "text": " So it's more precise to say that w, expedentially,"}, {"start": 6582.0, "end": 6586.0, "text": " w.x is this array."}, {"start": 6586.0, "end": 6589.0, "text": " But this array was filled in by counting,"}, {"start": 6589.0, "end": 6592.0, "text": " and by basically,"}, {"start": 6592.0, "end": 6594.0, "text": " popularly in the counts of bi-grams,"}, {"start": 6594.0, "end": 6596.0, "text": " whereas in the gradient base framework,"}, {"start": 6596.0, "end": 6598.0, "text": " we initialize it randomly,"}, {"start": 6598.0, "end": 6603.0, "text": " and then we let the loss guide us to arrive at the exact same array."}, {"start": 6603.0, "end": 6606.0, "text": " So, this array, exactly here,"}, {"start": 6606.0, "end": 6610.0, "text": " is basically the array w at the end of optimization,"}, {"start": 6610.0, "end": 6612.0, "text": " except we arrive at it,"}, {"start": 6612.0, "end": 6615.0, "text": " piece by piece by following the loss."}, {"start": 6615.0, "end": 6618.0, "text": " And that's why we also obtain the same loss function at the end."}, {"start": 6618.0, "end": 6620.0, "text": " And the second note is, if I come here,"}, {"start": 6620.0, "end": 6624.0, "text": " remember the smoothing where we added fake counts to our counts"}, {"start": 6624.0, "end": 6628.0, "text": " in order to smooth out and make more uniform"}, {"start": 6628.0, "end": 6631.0, "text": " the distributions of these probabilities."}, {"start": 6631.0, "end": 6634.0, "text": " And that prevented us from assigning zero probability"}, {"start": 6634.0, "end": 6637.0, "text": " to any one bi-gram."}, {"start": 6637.0, "end": 6640.0, "text": " Now, if I increase the count here,"}, {"start": 6640.0, "end": 6643.0, "text": " what's happening to the probability?"}, {"start": 6643.0, "end": 6645.0, "text": " As I increase the count,"}, {"start": 6645.0, "end": 6648.0, "text": " probability becomes more and more uniform, right?"}, {"start": 6648.0, "end": 6652.0, "text": " Because these counts go only up to like 900 or whatever."}, {"start": 6652.0, "end": 6655.0, "text": " So if I'm adding plus a million to every single number here,"}, {"start": 6655.0, "end": 6657.0, "text": " you can see how the row,"}, {"start": 6657.0, "end": 6659.0, "text": " and its probability, then, when we divide,"}, {"start": 6659.0, "end": 6662.0, "text": " it's just going to become more and more close to exactly even"}, {"start": 6662.0, "end": 6665.0, "text": " probability in a form distribution."}, {"start": 6665.0, "end": 6667.0, "text": " It turns out that the gradient base framework"}, {"start": 6667.0, "end": 6670.0, "text": " has an equivalent to smoothing."}, {"start": 6670.0, "end": 6673.0, "text": " In particular,"}, {"start": 6673.0, "end": 6676.0, "text": " think through these w's here,"}, {"start": 6676.0, "end": 6678.0, "text": " which we initialize randomly."}, {"start": 6678.0, "end": 6682.0, "text": " We could also think about initializing w's to be zero."}, {"start": 6682.0, "end": 6685.0, "text": " If all the entries of w are zero,"}, {"start": 6685.0, "end": 6688.0, "text": " then you'll see that logits will become all zero."}, {"start": 6688.0, "end": 6690.0, "text": " And then, expd."}, {"start": 6690.0, "end": 6692.0, "text": " And those logits becomes all one."}, {"start": 6692.0, "end": 6695.0, "text": " And then, the probabilities turn out to be exactly uniform."}, {"start": 6695.0, "end": 6698.0, "text": " So basically, when w's are all equal to each other,"}, {"start": 6698.0, "end": 6701.0, "text": " or say, especially zero,"}, {"start": 6701.0, "end": 6704.0, "text": " then the probabilities come out completely uniform."}, {"start": 6704.0, "end": 6709.0, "text": " So, trying to incentivize w to be near zero"}, {"start": 6709.0, "end": 6713.0, "text": " is basically equivalent to label smoothing."}, {"start": 6713.0, "end": 6715.0, "text": " And the more you incentivize that in a loss function,"}, {"start": 6715.0, "end": 6718.0, "text": " the more smooth distribution you're going to achieve."}, {"start": 6718.0, "end": 6721.0, "text": " So this brings us to something that's called regularization,"}, {"start": 6721.0, "end": 6724.0, "text": " where we can actually augment the loss function"}, {"start": 6724.0, "end": 6728.0, "text": " to have a small component that we call a regularization loss."}, {"start": 6728.0, "end": 6731.0, "text": " In particular, what we're going to do is we can take w."}, {"start": 6731.0, "end": 6732.0, "text": " And we can, for example,"}, {"start": 6732.0, "end": 6734.0, "text": " square all of its entries."}, {"start": 6734.0, "end": 6738.0, "text": " And then we can, oops, sorry about that."}, {"start": 6738.0, "end": 6743.0, "text": " We can take all the entries of w and we can sum them."}, {"start": 6743.0, "end": 6745.0, "text": " And because we're squaring,"}, {"start": 6745.0, "end": 6748.0, "text": " there will be no signs anymore."}, {"start": 6748.0, "end": 6751.0, "text": " Natives and positives all get squashed to be positive numbers."}, {"start": 6751.0, "end": 6755.0, "text": " And then, the way this works is you achieve zero loss"}, {"start": 6755.0, "end": 6757.0, "text": " if w is exactly or zero."}, {"start": 6757.0, "end": 6761.0, "text": " But if w has non-zero numbers, you accumulate loss."}, {"start": 6761.0, "end": 6764.0, "text": " And so we can actually take this and we can add it on here."}, {"start": 6764.0, "end": 6771.0, "text": " So we can do something like loss plus w square dot sum."}, {"start": 6771.0, "end": 6774.0, "text": " Or let's actually, instead of sum, let's take a mean."}, {"start": 6774.0, "end": 6777.0, "text": " Because otherwise the sum gets too large."}, {"start": 6777.0, "end": 6781.0, "text": " So mean is like a little bit more manageable."}, {"start": 6781.0, "end": 6783.0, "text": " And then we have a regularization loss here."}, {"start": 6783.0, "end": 6785.0, "text": " I'll say 0.01 times."}, {"start": 6785.0, "end": 6786.0, "text": " Or something like that."}, {"start": 6786.0, "end": 6789.0, "text": " You can choose the regularization strength."}, {"start": 6789.0, "end": 6792.0, "text": " And then we can just optimize this."}, {"start": 6792.0, "end": 6795.0, "text": " And now this optimization actually has two components."}, {"start": 6795.0, "end": 6798.0, "text": " Not only is it trying to make all the probabilities work out."}, {"start": 6798.0, "end": 6799.0, "text": " But in addition to that,"}, {"start": 6799.0, "end": 6804.0, "text": " there's an additional component that simultaneously tries to make all w's b zero."}, {"start": 6804.0, "end": 6806.0, "text": " Because if w's are non-zero, you feel a loss."}, {"start": 6806.0, "end": 6810.0, "text": " And so minimizing this, the only way to achieve that is for w to b zero."}, {"start": 6810.0, "end": 6813.0, "text": " And so you can think of this as adding like a spring force,"}, {"start": 6813.0, "end": 6817.0, "text": " or like a gravity force, that pushes w to b zero."}, {"start": 6817.0, "end": 6819.0, "text": " So w wants to b zero."}, {"start": 6819.0, "end": 6821.0, "text": " And the probabilities want to be uniform."}, {"start": 6821.0, "end": 6825.0, "text": " But they also simultaneously want to match up your probabilities"}, {"start": 6825.0, "end": 6827.0, "text": " as indicated by the data."}, {"start": 6827.0, "end": 6830.0, "text": " And so the strength of this regularization"}, {"start": 6830.0, "end": 6836.0, "text": " is exactly controlling the amount of counts that you add here."}, {"start": 6836.0, "end": 6844.0, "text": " Adding a lot more counts here corresponds to increasing this number."}, {"start": 6844.0, "end": 6846.0, "text": " Because the more you increase it,"}, {"start": 6846.0, "end": 6849.0, "text": " the more this part of the loss function dominates this part."}, {"start": 6849.0, "end": 6853.0, "text": " And the more these weights will be unable to grow."}, {"start": 6853.0, "end": 6858.0, "text": " Because as they grow, they accumulate way too much loss."}, {"start": 6858.0, "end": 6861.0, "text": " And so if this is strong enough,"}, {"start": 6861.0, "end": 6864.0, "text": " then we are not able to overcome the force of this loss."}, {"start": 6864.0, "end": 6868.0, "text": " And we will never, and basically everything will be uniform predictions."}, {"start": 6868.0, "end": 6870.0, "text": " So I thought that's kind of cool."}, {"start": 6870.0, "end": 6872.0, "text": " Okay, and lastly, before we wrap up,"}, {"start": 6872.0, "end": 6876.0, "text": " I wanted to show you how you would sample from this neural net model."}, {"start": 6876.0, "end": 6880.0, "text": " And I copy-pasted the sampling code from before,"}, {"start": 6880.0, "end": 6884.0, "text": " where remember that we sampled five times."}, {"start": 6884.0, "end": 6886.0, "text": " And all we did was start at zero,"}, {"start": 6886.0, "end": 6890.0, "text": " we grabbed the current ix row of p."}, {"start": 6890.0, "end": 6892.0, "text": " And that was our probability row,"}, {"start": 6892.0, "end": 6894.0, "text": " from which we sampled the next index,"}, {"start": 6894.0, "end": 6898.0, "text": " and just accumulated that and break when zero."}, {"start": 6898.0, "end": 6903.0, "text": " And running this gave us these results."}, {"start": 6903.0, "end": 6907.0, "text": " I still have the p in memory, so this is fine."}, {"start": 6907.0, "end": 6912.0, "text": " Now, this p doesn't come from the row of p."}, {"start": 6912.0, "end": 6915.0, "text": " Instead it comes from this neural net."}, {"start": 6915.0, "end": 6917.0, "text": " First, we take ix,"}, {"start": 6917.0, "end": 6922.0, "text": " and we encode it into a one-hot row of x-ank."}, {"start": 6922.0, "end": 6925.0, "text": " This x-ank multiplies rw,"}, {"start": 6925.0, "end": 6929.0, "text": " which really just plugs out the row of w corresponding to ix."}, {"start": 6929.0, "end": 6930.0, "text": " Really, that's what's happening."}, {"start": 6930.0, "end": 6932.0, "text": " And that gets our logits,"}, {"start": 6932.0, "end": 6935.0, "text": " and then we normalize those logits,"}, {"start": 6935.0, "end": 6938.0, "text": " and we actually exponential to get counts,"}, {"start": 6938.0, "end": 6941.0, "text": " and then we normalize to get the distribution,"}, {"start": 6941.0, "end": 6944.0, "text": " and then we can sample from the distribution."}, {"start": 6944.0, "end": 6947.0, "text": " So if I run this,"}, {"start": 6947.0, "end": 6950.0, "text": " kind of anti-climatic or climatic,"}, {"start": 6950.0, "end": 6952.0, "text": " depending on how you look at it,"}, {"start": 6952.0, "end": 6955.0, "text": " but we get the exact same result."}, {"start": 6955.0, "end": 6958.0, "text": " And that's because this is the identical model."}, {"start": 6958.0, "end": 6961.0, "text": " Not only does it achieve the same loss,"}, {"start": 6961.0, "end": 6962.0, "text": " but as I mentioned,"}, {"start": 6962.0, "end": 6964.0, "text": " these are identical models,"}, {"start": 6964.0, "end": 6967.0, "text": " but we came to this answer in a very different way,"}, {"start": 6967.0, "end": 6969.0, "text": " and it's got a very different interpretation."}, {"start": 6969.0, "end": 6970.0, "text": " But fundamentally,"}, {"start": 6970.0, "end": 6971.0, "text": " this is basically the same model,"}, {"start": 6971.0, "end": 6973.0, "text": " and gives the same samples here."}, {"start": 6973.0, "end": 6975.0, "text": " And so, that's kind of cool."}, {"start": 6975.0, "end": 6978.0, "text": " Okay, so we've actually covered a lot of ground."}, {"start": 6978.0, "end": 6982.0, "text": " We introduced the bi-gram character level language model."}, {"start": 6982.0, "end": 6984.0, "text": " We saw how we can train the model,"}, {"start": 6984.0, "end": 6985.0, "text": " how we can sample from the model,"}, {"start": 6985.0, "end": 6988.0, "text": " and how we can evaluate the quality of the model,"}, {"start": 6988.0, "end": 6990.0, "text": " using the negative log likelihood loss."}, {"start": 6990.0, "end": 6993.0, "text": " And then we actually train the model in two completely different ways,"}, {"start": 6993.0, "end": 6996.0, "text": " that actually get the same result and the same model."}, {"start": 6996.0, "end": 6998.0, "text": " In the first way, we just count it up,"}, {"start": 6998.0, "end": 7000.0, "text": " the frequency of all the bi-grams,"}, {"start": 7000.0, "end": 7001.0, "text": " and normalized."}, {"start": 7001.0, "end": 7002.0, "text": " In the second way,"}, {"start": 7002.0, "end": 7006.0, "text": " we used the negative log likelihood loss,"}, {"start": 7006.0, "end": 7007.0, "text": " as a guide,"}, {"start": 7007.0, "end": 7010.0, "text": " to optimizing the counts matrix,"}, {"start": 7010.0, "end": 7011.0, "text": " or the counts array,"}, {"start": 7011.0, "end": 7013.0, "text": " so that the loss is minimized,"}, {"start": 7013.0, "end": 7015.0, "text": " in the gradient-based framework."}, {"start": 7015.0, "end": 7018.0, "text": " And we saw that both of them give the same result,"}, {"start": 7018.0, "end": 7021.0, "text": " and that's it."}, {"start": 7021.0, "end": 7023.0, "text": " Now, the second one of these,"}, {"start": 7023.0, "end": 7025.0, "text": " the gradient-based framework, is much more flexible."}, {"start": 7025.0, "end": 7028.0, "text": " And right now, our neural network is super simple."}, {"start": 7028.0, "end": 7030.0, "text": " We're taking a single previous character,"}, {"start": 7030.0, "end": 7033.0, "text": " and we're taking it through a single linear layer,"}, {"start": 7033.0, "end": 7034.0, "text": " to calculate the logits."}, {"start": 7034.0, "end": 7036.0, "text": " This is about to complexify."}, {"start": 7036.0, "end": 7037.0, "text": " So, in the follow-up videos,"}, {"start": 7037.0, "end": 7040.0, "text": " we're going to be taking more and more of these characters,"}, {"start": 7040.0, "end": 7043.0, "text": " and we're going to be feeding them into a neural net."}, {"start": 7043.0, "end": 7045.0, "text": " But this neural net will still output the exact same thing."}, {"start": 7045.0, "end": 7048.0, "text": " The neural net will output logits."}, {"start": 7048.0, "end": 7050.0, "text": " And these logits will still be normalized in the exact same way,"}, {"start": 7050.0, "end": 7052.0, "text": " and all the loss, and everything else,"}, {"start": 7052.0, "end": 7054.0, "text": " and the gradient-based framework,"}, {"start": 7054.0, "end": 7055.0, "text": " everything stays identical."}, {"start": 7055.0, "end": 7058.0, "text": " It's just that this neural net will now complexify"}, {"start": 7058.0, "end": 7060.0, "text": " all the way to transformers."}, {"start": 7060.0, "end": 7062.0, "text": " So, that's going to be pretty awesome,"}, {"start": 7062.0, "end": 7064.0, "text": " and I'm looking forward to it for now."}, {"start": 7064.0, "end": 7080.0, "text": " Bye."}]