[User] Please transcribe this text

[Assistant] Well, there is at least a curious coincidence of dates. A ventilator is made, a cord is hung, and a lady who sleeps in the bed dies. Does not that strike you? I cannot as yet see any connection.


[User] Please transcribe this text

[Assistant] Did you observe anything very peculiar about that bed? No. It was clamped to the floor. Did you ever see a bed fastened like that before? I can't say that I have.


[User] Please transcribe this text

[Assistant] We are only just in time to prevent some subtle and horrible crime. Subtle enough and horrible enough.


[User] Please transcribe this text

[Assistant] When a doctor does go wrong he is the first of criminals. He has nerve and he has knowledge.


[User] Please transcribe this text

[Assistant] Palmer and Pritchard were among the heads of their profession.


[User] Please transcribe this text

[Assistant] This man strikes even deeper, but I think, Watson, that we shall be able to strike deeper still.


[User] Please transcribe this text

[Assistant] But we shall have horrors enough before the night is over. For goodness sake, let us have a quiet pipe and turn our minds for a few hours to something more cheerful.


[User] Please transcribe this text

[Assistant] About nine o'clock the light among the trees was extinguished, and all was dark in the direction of the manor house.


[User] Please transcribe this text

[Assistant] Two hours passed slowly away, and then suddenly, just at the stroke of eleven, a single bright light shone out right in front of us.


[User] Please transcribe this text

[Assistant] That is our signal," said Holmes, springing to his feet. It comes from the middle window.


[User] Please transcribe this text

[Assistant] As we passed out he exchanged a few words with the landlord, explaining that we were going on a late visit to an acquaintance, and that it was possible that we might spend the night there.


[User] Please transcribe this text

[Assistant] A moment later, we were out on the dark road.


[User] Please transcribe this text

[Assistant] A chill wind blowing in our faces and one yellow light twinkling in front of us through the gloom to guide us on our sombre errand.


[User] Please transcribe this text

[Assistant] There was little difficulty in entering the grounds for unrepaired breaches gaped in the old park wall.


[User] Please transcribe this text

[Assistant] Making our way among the trees, we reached the lawn, crossed it, and were about to enter through the window, when out from a clump of laurel bushes there darted what seemed to be a hideous and distorted child, who threw itself upon the grass with writhing limbs, and then ran swiftly across the lawn into the darkness.


[User] Please transcribe this text

[Assistant] Holmes was for the moment as startled as I. His hand closed like a vice upon my wrist in his agitation, and he broke into a low laugh and put his lips to my ear. It is a nice household, he murmured. That is the baboon.


[User] Please transcribe this text

[Assistant] I had forgotten the strange pets which the doctor affected. It was a cheetah, too.


[User] Please transcribe this text

[Assistant] Perhaps we might find it upon our shoulders at any moment.


[User] Please transcribe this text

[Assistant] I confess that I felt easier in my mind when, after following Holmes's example and slipping off my shoes, I found myself inside the bedroom.


[User] Please transcribe this text

[Assistant] My companion noiselessly closed the shutters, moved the lamp onto the table, and cast his eyes round the room. All was as we have seen it in the daytime.


[User] Please transcribe this text

[Assistant] Then creeping up to me and making a trumpet of his hand, he whispered into my ear again so gently that it was all that I could do to distinguish the words.


[User] Please transcribe this text

[Assistant] The least sound would be fatal to our plans. I nodded to show that I had heard. We must sit without light. He would see it through the ventilator. I nodded again. Do not go asleep. Your very life may depend upon it. Have your pistol ready in case we should need it. I will sit on the side of the bed, and you in that chair. I took out my revolver and laid it on the corner of the table.


[User] Please transcribe this text

[Assistant] Holmes had brought up a long thin cane, and this he placed upon the bed beside him.


[User] Please transcribe this text

[Assistant] By it he laid the box of matches and the stump of a candle.


[User] Please transcribe this text

[Assistant] Then he turned down the lamp and we were left in darkness. How shall I ever forget that dreadful vigil


[User] Please transcribe this text

[Assistant] I could not hear a sound, not even the drawing of a breath, and yet I knew that my companion sat open-eyed within a few feet of me, in the same state of nervous tension in which I was myself.


[User] Please transcribe this text

[Assistant] The shutters cut off the least ray of light and we waited in absolute darkness.


[User] Please transcribe this text

[Assistant] From outside came the occasional cry of a night-bird, and once at our very window a long-drawn cat-like whine, which told us that the cheetah was indeed at liberty.


[User] Please transcribe this text

[Assistant] Far away we could hear the deep tones of the parish clock, which boomed out every quarter of an hour. How long they seemed, those quarters!


[User] Please transcribe this text

[Assistant] Suddenly there was the momentary gleam of a light up in the direction of the ventilator, which vanished immediately, but was succeeded by a strong smell of burning oil and heated metal.


[User] Please transcribe this text

[Assistant] Someone in the next room had lit a dark lantern.


[User] Please transcribe this text

[Assistant] I heard a gentle sound of movement, and then all was silent once more, though the smell grew stronger. For half an hour I sat with straining ears, then suddenly another sound became audible.


[User] Please transcribe this text

[Assistant] A very gentle, soothing sound, like that of a small jet of steam escaping continually from a kettle.


[User] Please transcribe this text

[Assistant] The instant that we heard it, Holmes sprang from the bed, struck a match, and lashed furiously with his cane at the bell-pole. You see it, Watson? he yelled. You see it? But I saw nothing.


[User] Please transcribe this text

[Assistant] At the moment when Holmes struck the light I heard a low, clear whistle.


[User] Please transcribe this text

[Assistant] But the sudden glare flashing into my weary eyes made it impossible for me to tell what it was at which my friend lashed so savagely.


[User] Please transcribe this text

[Assistant] I could, however, see that his face was deadly pale and filled with horror and loathing.


[User] Please transcribe this text

[Assistant] He had ceased to strike and was gazing up at the ventilator, when suddenly there broke from the silence of the night the most horrible cry to which I have ever listened.


[User] Please transcribe this text

[Assistant] It swelled up louder and louder. A hoarse yell of pain and fear and anger, all mingled in the one dreadful shriek.


[User] Please transcribe this text

[Assistant] They say that a way down in the village, and even in the distant parsonage, that cry raised the sleepers from their beds.


[User] Please transcribe this text

[Assistant] It means that it is all over," Holmes answered, and perhaps, after all, it is for the best. Take your pistol, and we will enter Dr. Roylott's room.


[User] Please transcribe this text

[Assistant] With a grave face, he lit the lamp and led the way down the corridor.


[User] Please transcribe this text

[Assistant] Twice he struck at the chamber door without any reply from within, and he turned the handle and entered, I at his heels with the cocked pistol in my hand. It was a singular sight which met our eyes.


[User] Please transcribe this text

[Assistant] On the tables stood a dark lantern with the shutter half open, throwing a brilliant beam of light upon the iron safe, the door of which was a jar.


[User] Please transcribe this text

[Assistant] Beside this table, on the wooden chair, sat Dr. Grimesby Roylott, clad in a long grey dressing-gown, his bare ankles protruding beneath, and his feet thrust into red heelless Turkish slippers.


[User] Please transcribe this text

[Assistant] Across his lap lay the short stock with the long lash which we had noticed during the day. His chin was cocked upward, and his eyes were fixed in a dreadful, rigid stare at the corner of the ceiling.


[User] Please transcribe this text

[Assistant] Round his brow he had a peculiar yellow band with brownish speckles, which seemed to be bound tightly around his head.


[User] Please transcribe this text

[Assistant] As we entered, he made neither sound nor motion.


[User] Please transcribe this text

[Assistant] Band, the speckled band, whispered Holmes. I took a step forward.


[User] Please transcribe this text

[Assistant] In an instant his strange headgear began to move, and there reared itself from among his hair the squat diamond-shaped head and puffed neck of a loathsome servant.


[User] Please transcribe this text

[Assistant] It is a swamp adder, cried Holmes, the deadliest snake in India. He has died within ten seconds of being bitten.


[User] Please transcribe this text

[Assistant] Violence does, in truth, recoil upon the violent, and the schemer falls into the pit which he digs for another.


[User] Please transcribe this text

[Assistant] Let us thrust this creature back into its den, and we can then remove Miss Stoner to some place of shelter, and let the county police know what has happened.


[User] Please transcribe this text

[Assistant] As he spoke he drew the dog-whip swiftly from the dead man's lap.


[User] Please transcribe this text

[Assistant] And throwing the noose round the reptile's neck, he drew it from its horrid perch, and carrying it at arm's length, threw it into the iron safe, which he closed upon it.


[User] Please transcribe this text

[Assistant] Such are the true facts of the death of Dr. Grimesby Roylott of Stoke Moran.


[User] Please transcribe this text

[Assistant] It is not necessary that I should prolong a narrative which has already run to too great a length by telling how we broke the sad news to the terrified girl.


[User] Please transcribe this text

[Assistant] How we conveyed her by the morning train to the care of her good aunt at Harrow, of how the slow process of official inquiry came to the conclusion that the doctor met his fate while indiscreetly playing with a dangerous pet.


[User] Please transcribe this text

[Assistant] The little which I had yet to learn of the case was told me by Sherlock Holmes as we travelled back next day.


[User] Please transcribe this text

[Assistant] Come to an entirely erroneous conclusion, which shows, my dear Watson, how dangerous it is always to reason from insufficient data.


[User] Please transcribe this text

[Assistant] The presence of the gypsies, and the use of the word band, which was used by the poor girl no doubt to explain the appearance which she had caught a hurried glimpse of by the light of her match, were sufficient to put me upon an entirely wrong


[User] Please transcribe this text

[Assistant] I can only claim the merit that I instantly reconsidered my position when, however it became clear to me, that whatever danger threatened an occupant of the room could not come either from the window or the door.


[User] Please transcribe this text

[Assistant] My attention was speedily drawn, as I have already remarked to you, to this ventilator, and to the bell-rope which hung down to the bed.


[User] Please transcribe this text

[Assistant] The discovery that this was a dummy, and that the bed was clamped to the floor, instantly gave rise to the suspicion that the rope was there as a bridge for something passing through the hole and coming to the bed.


[User] Please transcribe this text

[Assistant] The idea of a snake instantly occurred to me, and when I coupled it with my knowledge that the doctor was furnished with a supply of creatures from India, I felt that I was probably on the right track.


[User] Please transcribe this text

[Assistant] The idea of using a form of poison which could not possibly be discovered by any chemical test was just such a one as would occur to a clever and ruthless man who had had an Eastern training. The rapidity with which such a poison would take effect would also, from his point of view, be an advantage.


[User] Please transcribe this text

[Assistant] It would be a sharp-eyed coroner, indeed, who could distinguish the two little dark punctures which would show where the poison fangs had done their work. Then I thought of the whistle.


[User] Please transcribe this text

[Assistant] Of course, he must recall the snake before the morning light revealed it to the victim.


[User] Please transcribe this text

[Assistant] He had trained it probably by the use of the milk which we saw to return to him when summoned.


[User] Please transcribe this text

[Assistant] He would put it through this ventilator at the hour that he thought best, with the certainty that it would crawl down the rope and land on the bed.


[User] Please transcribe this text

[Assistant] It might or might not bite the occupant. Perhaps she might escape every night for a week, but sooner or later she must fall a victim.


[User] Please transcribe this text

[Assistant] I had come to these conclusions before ever I had entered his room. An inspection of his chair showed me that he had been in the habit of standing on it, which of course would be necessary in order that he should reach the ventilator. The sight of the safe, the saucer of milk, and the loop of whip-cord were enough to finally dispel any doubts which may have remained.


[User] Please transcribe this text

[Assistant] The metallic clang heard by Miss Stoner was obviously caused by her stepfather hastily closing the door of his safe upon its terrible occupant.


[User] Please transcribe this text

[Assistant] Having once made up my mind, you know the steps which I took in order to put the matter to the proof.


[User] Please transcribe this text

[Assistant] I heard the creature hiss as I have no doubt that you did also, and I instantly lit the light and attacked it, with the result of driving it through the ventilator.


[User] Please transcribe this text

[Assistant] And also with the result of causing it to turn upon its master at the other side.


[User] Please transcribe this text

[Assistant] Some of the blows of my cane came home and roused its snakeish temper, so that it flew upon the first person it saw.


[User] Please transcribe this text

[Assistant] Alright, hello everybody. Welcome back. Glad you came back today.


[User] Please transcribe this text

[Assistant] We will unveil the first tutorial, the first project.


[User] Please transcribe this text

[Assistant] This is Deep Traffic, codenamed Deep Traffic, where your task is to solve the traffic problem using deep reinforcement learning. And I'll talk about what's involved in designing a network there, how you submit your own network, and how you participate in the competition.


[User] Please transcribe this text

[Assistant] As I said, the winner gets a very special prize. It'll be announced later. What is machine learning?


[User] Please transcribe this text

[Assistant] Several types. There's supervised learning. As I mentioned yesterday, that's what's meant usually when you discuss about, you talk about machine learning and talk about its successes. Supervised learning requires a data set.


[User] Please transcribe this text

[Assistant] Well you know the ground truth. You know the inputs and the outputs.


[User] Please transcribe this text

[Assistant] And you provide that to a machine learning algorithm in order to


[User] Please transcribe this text

[Assistant] Learn the mapping between the inputs and the outputs in such a way that you can generalize to further examples in the future. On supervised learning is the other side.


[User] Please transcribe this text

[Assistant] When you know absolutely nothing about the outputs, about the truth.


[User] Please transcribe this text

[Assistant] of the data that you're working with. All you get is data.


[User] Please transcribe this text

[Assistant] And you have to find underlying structure.


[User] Please transcribe this text

[Assistant] Underlying representation of the data that's meaningful for you to accomplish a certain task, whatever that is. There's semi-supervised data.


[User] Please transcribe this text

[Assistant] Or only parts, usually a very small amount.


[User] Please transcribe this text

[Assistant] Is labeled. There's ground truth available for just a small fraction of it. If you think of images that are out there on the internet,


[User] Please transcribe this text

[Assistant] And then you think about ImageNet, a data set where every image is labeled.


[User] Please transcribe this text

[Assistant] The size of that, uh, ImageNet data set is a tiny subset of all the images.


[User] Please transcribe this text

[Assistant] On the, uh, available online.


[User] Please transcribe this text

[Assistant] But that's the task we're dealing with. As human beings, as people interested in doing machine learning, is


[User] Please transcribe this text

[Assistant] How to expand the size of that.


[User] Please transcribe this text

[Assistant] There's an agent that has to exist in the world. And that agent...


[User] Please transcribe this text

[Assistant] Knows the inputs that the world provides.


[User] Please transcribe this text

[Assistant] But knows very little about that world, except through occasional.


[User] Please transcribe this text

[Assistant] Time delayed rewards. This is what it's like to be human. This is what life is about.


[User] Please transcribe this text

[Assistant] You don't know what's good and bad. You kind of have to just live it. And every once in a while...


[User] Please transcribe this text

[Assistant] You find out that all that stuff you did last week was a pretty bad idea.


[User] Please transcribe this text

[Assistant] That's reinforcement learning. That's semi-supervised.


[User] Please transcribe this text

[Assistant] In the sense that only a small subset of the data comes with some ground truth, some certainty that you have to then extract knowledge from.


[User] Please transcribe this text

[Assistant] So first, at the core of anything that's, that works currently, in terms of in a practical sense, there has to be some ground truth. There has to be some truth.


[User] Please transcribe this text

[Assistant] That we can hold on to as we try to generalize.


[User] Please transcribe this text

[Assistant] And that's supervised learning. Even as in reinforcement learning, the only thing we can count on is that truth that comes in the form of a reward.


[User] Please transcribe this text

[Assistant] So the standard supervised learning pipeline is you have some raw data, the inputs you have,


[User] Please transcribe this text

[Assistant] Ground truth, the labels, the outputs, that matches to the inputs.


[User] Please transcribe this text

[Assistant] Then, you run a certain, any kind of algorithm, whether that's a neural network.


[User] Please transcribe this text

[Assistant] Or another preprocessing processing algorithm that extracts the features from that data set.


[User] Please transcribe this text

[Assistant] You can think of a picture of a face. That algorithm could...


[User] Please transcribe this text

[Assistant] Extract the nose, the eyes, the corners of the eyes, the pupil.


[User] Please transcribe this text

[Assistant] Or even lower level features in that image.


[User] Please transcribe this text

[Assistant] After that, we insert those features into a, into a model, a machine learning model, and we train that model.


[User] Please transcribe this text

[Assistant] Then we, as we, whatever that algorithm is, as we pass it through that training process, we then evaluate.


[User] Please transcribe this text

[Assistant] After we've seen the exa- this one particular example.


[User] Please transcribe this text

[Assistant] How much better are we at other tasks?


[User] Please transcribe this text

[Assistant] And as we repeat this loop, the model learns to perform better and better at generalizing from the raw data to the labels that we have.


[User] Please transcribe this text

[Assistant] And finally, you get to release that model into the wild to actually do prediction on data it's never seen before that you don't know about.


[User] Please transcribe this text

[Assistant] Okay, so, neural networks is what this class is about.


[User] Please transcribe this text

[Assistant] It's one of the machine learning algorithms that has proven to be very successful.


[User] Please transcribe this text

[Assistant] And the building block, the computational building block of a neural network is a neuron. A perceptron is a type of neuron. It's the original old school neuron. Where the output is binary, a zero or a one. It's not real valued. And the process that a perceptron goes through is its, has multiple inputs,


[User] Please transcribe this text

[Assistant] The inputs, each of the inputs have weights on them. Shown here on the left as 0.7, 0.6, 1.4.


[User] Please transcribe this text

[Assistant] Those weights are applied to the inputs, and a perceptron the inputs are ones or zeros. Binary. And those weights are applied


[User] Please transcribe this text

[Assistant] A bias on each neuron is then added on top


[User] Please transcribe this text

[Assistant] There's a test whether that summed value plus the bias is below or above a threshold. If it's above a threshold, it produces a 1. If it's below a threshold, it produces a 0. Simple. It's one of the only things we understand about neural networks.


[User] Please transcribe this text

[Assistant] Confidently. We can prove a lot of things about this neuron.


[User] Please transcribe this text

[Assistant] Is that a neuron can approximate, uh, uh, approximate a neon gate. A neon gate is...


[User] Please transcribe this text

[Assistant] A logical operation, a logical function that takes its input.


[User] Please transcribe this text

[Assistant] It has two inputs, A and B, here on the di- on the diagram on the left.


[User] Please transcribe this text

[Assistant] And the table shows what that function is. When the inputs are zeros, zero one, in any order, the output is a one. Otherwise, it's a zero.


[User] Please transcribe this text

[Assistant] The cool thing about an AND gate is that it's a universal gate.


[User] Please transcribe this text

[Assistant] That you can build up any computer you have, where your phone in your pocket today can be built out of just NAND gates. So it's a, it's a functionally complete. You can build any logical function out of them if you stack them together in arbitrary ways. The problem with NAND gates and computers is they're built from the bottom up. You have to design these circuits of NAND gates. So the cool thing here is with the perceptron,


[User] Please transcribe this text

[Assistant] We can learn this magical nan gate. We can learn this function.


[User] Please transcribe this text

[Assistant] So let's go through what, how we can do that.


[User] Please transcribe this text

[Assistant] How a perceptron can perform the NAND operation. Here's the four examples. If we put the weights of negative two on each of the inputs, and a bias of three on the neuron, and if we perform that same operation,


[User] Please transcribe this text

[Assistant] Of summing the weights times the inputs, plus the bias. In the top left, we get


[User] Please transcribe this text

[Assistant] When the inputs are zeros, and there's sum to the bias, we get a three.


[User] Please transcribe this text

[Assistant] That's a, that's a positive number, which means the output of a perceptron will be a one. In the top right, when the input is a zero and a one,


[User] Please transcribe this text

[Assistant] That sum is still a positive number. Again, produces a 1.


[User] Please transcribe this text

[Assistant] And so on. When the inputs are both ones, then,


[User] Please transcribe this text

[Assistant] Uh, the output is a negative one, less than zero.


[User] Please transcribe this text

[Assistant] Basic computational truth you can hold on to as we talk about some of the magical things neural network can do.


[User] Please transcribe this text

[Assistant] Because if you compare a circuit of neon gates and a circuit of neurons,


[User] Please transcribe this text

[Assistant] The difference while a circuit of neurons, which is what we think of as a neural network.


[User] Please transcribe this text

[Assistant] Can perform the same thing as the circuit of, of NAND gates. What it can also do is it can learn.


[User] Please transcribe this text

[Assistant] It can learn the arbitrary logical functions that a arbitrary, uh, circuit of NAND gates can represent. But it doesn't require the human designer. We can evolve, if you will.


[User] Please transcribe this text

[Assistant] So, one of the key aspects here, one of the key drawbacks of Perceptron, is it's not very smooth in it's output.


[User] Please transcribe this text

[Assistant] As we change the weights on the inputs and we change the bias and we tweak it a little bit, it's very likely that when you get


[User] Please transcribe this text

[Assistant] It's very easy to make the neuron output a zero instead of a one, a one instead of a zero.


[User] Please transcribe this text

[Assistant] So when we start stacking many of these together, it's hard to control the output of the thing as a whole. Now, the essential step that makes a neural network work, that a circuit of perceptrons doesn't, is if the output is made smooth, is made continuous with an activation function.


[User] Please transcribe this text

[Assistant] And so instead of using a step function, like a perceptron does, shown there on the left, we use


[User] Please transcribe this text

[Assistant] Any kind of smooth function, sigmoid.


[User] Please transcribe this text

[Assistant] Where the output can change gradually as you change the weights and the bias.


[User] Please transcribe this text

[Assistant] The process of adjusting those weights gradually and seeing how it has an effect on the rest of the network.


[User] Please transcribe this text

[Assistant] You just keep tweaking weights here and there and seeing how much closer you get to the ground truth. And if you get farther away, you just adjust the weights in the opposite direction.


[User] Please transcribe this text

[Assistant] There is, what we'll mostly talk about today is feed forward neural networks. On the left, going from inputs to outputs. With no loops.


[User] Please transcribe this text

[Assistant] There is also these amazing things called recurrent neural networks. They're amazing because they have memory.


[User] Please transcribe this text

[Assistant] They have a memory of state. They remember...


[User] Please transcribe this text

[Assistant] The temporal dynamics of the data they went through.


[User] Please transcribe this text

[Assistant] But the painful thing is that they're really hard to train.


[User] Please transcribe this text

[Assistant] Today we'll talk about feed for neural networks. So let's look at this example.


[User] Please transcribe this text

[Assistant] An example of stacking a few, a few of these neurons together. Let's think of the task


[User] Please transcribe this text

[Assistant] The basic task now famous using the classification of numbers. You have an image of a number, handwritten number, and your task is given that image to say what number is in that image.


[User] Please transcribe this text

[Assistant] Now what is an image? An image is a collection of pixels. In this case, 28 by 28 pixels. That's a total of 784 numbers. Those numbers are from 0 to 255.


[User] Please transcribe this text

[Assistant] And on the left of the network, the size of that input, despite the diagram, is 784 neurons. That's the input. Then comes the hidden layer. It's called the hidden layer because it has


[User] Please transcribe this text

[Assistant] It has no interaction with the input or the output.


[User] Please transcribe this text

[Assistant] It is simply a block use the, it's at the core of the computational power of neural networks is the hidden layer. It's tasked with forming a representation


[User] Please transcribe this text

[Assistant] of the data in such a way that it maps from the inputs to the outputs.


[User] Please transcribe this text

[Assistant] In this case, there is 15 neurons in a hidden layer.


[User] Please transcribe this text

[Assistant] There is ten values on the output, uh, corresponding to each of the numbers.


[User] Please transcribe this text

[Assistant] There's several ways you could, you can build this kind of network, and this is what the magic of neural networks is, you can do it in a lot of ways. You only really need four outputs to represent values zero through nine.


[User] Please transcribe this text

[Assistant] But in practice, it seems that having ten outputs works better. And how do these work? Whenever the input is a five,


[User] Please transcribe this text

[Assistant] The output neuron in charge of the five gets really excited. And outputs a value that's close to one. From zero to one. Close to one. And then the other ones...


[User] Please transcribe this text

[Assistant] Get a output of value, hopefully, that is close to zero. And when they don't,


[User] Please transcribe this text

[Assistant] We adjust the weights in such a way that they get closer to zero and closer to one depending on whether it's the correct neuron associated with the picture.


[User] Please transcribe this text

[Assistant] What we've discussed just now is the forward pass through the network.


[User] Please transcribe this text

[Assistant] It's the pass when you take the inputs, apply the weights, sum them together, add the bias, produce the output.


[User] Please transcribe this text

[Assistant] And check which of the outputs produces the highest confidence of the number.


[User] Please transcribe this text

[Assistant] Then once those probabilities for each of the numbers is, is provided, we determine the gradient.


[User] Please transcribe this text

[Assistant] That's used to punish or reward the weights that resulted in either the correct or the incorrect decisions. And that's called backpropagation. We step backwards through the network applying those punishments or rewards.


[User] Please transcribe this text

[Assistant] Because of the smoothness of the activation functions, that is a mathematically efficient operation. That's where the GPUs step in. So for our example of numbers, the ground truth for number six looks like the following in the slides. Y of X equals to a ten-dimensional vector.


[User] Please transcribe this text

[Assistant] The ground truth that comes with the image. The loss function here


[User] Please transcribe this text

[Assistant] The basic loss function is the squared error. Y of X is the ground truth and A is the output of the neural network resulting from the forward pass.


[User] Please transcribe this text

[Assistant] So when you input that number of a fi- of a 6, and it outputs, whatever it outputs, that's A, a 10 dimensional vector. And it's summed over the inputs to produce the squared error.


[User] Please transcribe this text

[Assistant] That's our loss function. The loss function, the objective function, that's what's used to determine


[User] Please transcribe this text

[Assistant] How much to reward or punish the back propagated weights throughout the network.


[User] Please transcribe this text

[Assistant] And the basic operation of optimizing that loss function, of minimizing that loss function,


[User] Please transcribe this text

[Assistant] Is done with various variants of gradient descent. It's hopefully a somewhat smooth function. But it's a highly non-linear function.


[User] Please transcribe this text

[Assistant] This is why we can't prove much about neural networks.


[User] Please transcribe this text

[Assistant] It's a highly high dimensional, highly nonlinear function that's hopefully smooth enough where the gradient descent can find its way to at least a good solution. And there has to be some stochastic element there that


[User] Please transcribe this text

[Assistant] Uh, doesn't get, uh, that jumps around to ensure that it doesn't get stuck in a local minima of this very complex function.


[User] Please transcribe this text

[Assistant] Okay, that's supervised learning. There's inputs, there's outputs. Ground truth. That's our comfort zone. Because we're pretty confident we know what's going on.


[User] Please transcribe this text

[Assistant] All you have to do is just, you have this data set, you train, and...


[User] Please transcribe this text

[Assistant] You train a network on that data set and you can evaluate it. You can write a paper and try to beat a previous paper. It's great.


[User] Please transcribe this text

[Assistant] The problem is when you then use that neural network to create


[User] Please transcribe this text

[Assistant] An intelligent system that you put out there in the world.


[User] Please transcribe this text

[Assistant] And now that system no longer is working with your data set.


[User] Please transcribe this text

[Assistant] It has to exist in this world that doesn't, that's maybe very different from the ground truth. So the takeaway from supervised learning is that neural network's a great memorization.


[User] Please transcribe this text

[Assistant] But in a sort of philosophical way, they might not be great at generalizing, at reasoning beyond


[User] Please transcribe this text

[Assistant] The specific flavor of data set that they were trained on. The hope for reinforcement learning is that


[User] Please transcribe this text

[Assistant] We can extend the knowledge we gain in a supervised way.


[User] Please transcribe this text

[Assistant] Of what does, how good a certain state is or how bad a certain state is. This is a kind of brute force reasoning and I'll talk about


[User] Please transcribe this text

[Assistant] Kind of what I mean there, but it feels like it's closer to reasoning as opposed to memorization. That's a good way to think of supervised learning is memorization. You're just studying for an exam. And as many of you know,


[User] Please transcribe this text

[Assistant] That doesn't mean you're going to be successful in life just because you get an A. And so, uh, a reinforcement learning agent, or just any agent, a human being, or any machine existing in this world,


[User] Please transcribe this text

[Assistant] From the perspective of the agent, it can execute an action, it can receive an observation resulting from that action in a form of a new state, and it can receive a reward or a punishment.


[User] Please transcribe this text

[Assistant] You can break down every, our, you can break down our existence in this way. Simplistic view.


[User] Please transcribe this text

[Assistant] But it's a convenient one on the computational side. And from the environment side, the environment receives the action,


[User] Please transcribe this text

[Assistant] Emits the observation, so your action changes the world, therefore that world has to change.


[User] Please transcribe this text

[Assistant] And then tell you about it and give you a reward or a punishment for it.


[User] Please transcribe this text

[Assistant] Again, one of the most fascinating things, uh, I'll try to convey why this is fascinating a little bit later on, is the work of DeepMind on Atari. This is Atari Breakout, a game where a paddle has to move around,


[User] Please transcribe this text

[Assistant] That's the world it's existing in. It's a paddle. The agent is a paddle. And there's a bouncing ball.


[User] Please transcribe this text

[Assistant] And you're trying to move your actions to the right, move right, move left.


[User] Please transcribe this text

[Assistant] You're trying to move in such a way that the ball doesn't get past you.


[User] Please transcribe this text

[Assistant] And so here is a human level performance of that agent.


[User] Please transcribe this text

[Assistant] And so what does this paddle have to do? It has to operate in this environment. It has to act. Move left, move right.


[User] Please transcribe this text

[Assistant] Each action changes the state of the world.


[User] Please transcribe this text

[Assistant] This may seem obvious, but moving right changes visually.


[User] Please transcribe this text

[Assistant] The state of the world. In fact, what we're watching now on the slides is the world changing before your eyes for this little guy.


[User] Please transcribe this text

[Assistant] And it gets rewards or punishments.


[User] Please transcribe this text

[Assistant] Rewards it gets in the form of points. They're racking up points in the top left of the, uh, of the video. And then when the ball gets past the paddle,


[User] Please transcribe this text

[Assistant] It gets punished by dying, quote unquote. And that's the number of lives there's left. Going from five to four to three, down to zero. And so the goal is to select at any one moment,


[User] Please transcribe this text

[Assistant] The action that maximizes future award.


[User] Please transcribe this text

[Assistant] Without any knowledge of what a reward is.


[User] Please transcribe this text

[Assistant] In a greater sense of the word, all you have is an instantaneous reward or punishment. Instantaneous response of the world to your actions.


[User] Please transcribe this text

[Assistant] And this can be modeled as a Markov decision process. Markov decision process.


[User] Please transcribe this text

[Assistant] Is a mathematically convenient construct. It has no memory. All you get is you have a state


[User] Please transcribe this text

[Assistant] That you're currently in, you perform an action, you get a reward, and you find yourself in a new state. And that repeats over and over.


[User] Please transcribe this text

[Assistant] You start, you, you, from state zero, you go to state one, you once again repeat an action, get a reward, go to the next state. Okay, that's the formulation that we're operating in.


[User] Please transcribe this text

[Assistant] When you're in a certain state, you have no memory of what happened two states ago. Everything is operating on the instantaneous.


[User] Please transcribe this text

[Assistant] And so what are the major components of a reinforcement learning agent? There's a policy. That's...


[User] Please transcribe this text

[Assistant] That's the agent, the function, uh, broadly defined of an agent's behavior. That means...


[User] Please transcribe this text

[Assistant] That includes the knowledge of how, for any given state, what is an action that I will take. With some probability.


[User] Please transcribe this text

[Assistant] Value function is how good each state and action are in any particular state.


[User] Please transcribe this text

[Assistant] A subtle thing that is actually the biggest problem with everything you'll see today is the model is how we represent the environment.


[User] Please transcribe this text

[Assistant] And what you'll see today is some amazing things that neural networks can achieve.


[User] Please transcribe this text

[Assistant] On a relatively simplistic model of the world. And the question whether that model can extend to the real world, where human lives are at stake in the case of driving.


[User] Please transcribe this text

[Assistant] So let's look at the simplistic world. A robot in a room. You start at the bottom left. Your goal is to get to the top right.


[User] Please transcribe this text

[Assistant] Your possible actions are going up, down, left and right.


[User] Please transcribe this text

[Assistant] Now, this world could be deterministic, which means when you take, when you go up, you actually go up. Or it could be non-deterministic, as human life is. Because when you go up, sometimes you go right. So in this case, if you choose to go up,


[User] Please transcribe this text

[Assistant] You move up 80% of the time, you move left 10% of the time, and you move right 10% of the time. And when you get to the,


[User] Please transcribe this text

[Assistant] Top right, you get a reward of plus one, and you get to the second block from that, four two, you get negative one, you get punished. And every time you take a step, you get a slight punishment of point zero four.


[User] Please transcribe this text

[Assistant] Okay, so the question is, if you start at the bottom left,


[User] Please transcribe this text

[Assistant] Is this a good solution? Is this a good policy by which you exist in the world? And it is, if the world is deterministic. If, whenever you choose to go up, you go up. Whenever you choose to go right, you go right.


[User] Please transcribe this text

[Assistant] But if the actions are stochastic, that's not the case. In what I described previously with 0.8 up and probability of 0.1 going left and right, this is the optimal policy.


[User] Please transcribe this text

[Assistant] Now, if we punish every single step with a negative 2, as opposed to a negative 0.04.


[User] Please transcribe this text

[Assistant] So every time you take a step, it hurts. You're going to try to get


[User] Please transcribe this text

[Assistant] To a, a positive block as quickly as possible.


[User] Please transcribe this text

[Assistant] And that's what this policy says. I'll walk through a negative one if I have to, as long as I get, stop getting a negative two.


[User] Please transcribe this text

[Assistant] Now, is there a word for each step as a negative point one?


[User] Please transcribe this text

[Assistant] You might choose to go around that negative one block. A slight detour to avoid the pain.


[User] Please transcribe this text

[Assistant] And then you might take an even longer detour as the reward for each step goes up, or the punishment goes down, I guess.


[User] Please transcribe this text

[Assistant] And then if the, there's an actual positive reward for every step you take, then you'll never, you'll avoid going to your, the finish line. You'll just wander the world. We saw that.


[User] Please transcribe this text

[Assistant] With the, uh, coast racer yesterday, the, uh, the boat.


[User] Please transcribe this text

[Assistant] That chose not to finish the race because it was having too much fun getting points in the middle.


[User] Please transcribe this text

[Assistant] Let's look at the world that this agent is operating in. It's a value function. That value function depends on a reward. The reward that comes in the future.


[User] Please transcribe this text

[Assistant] And that reward is discounted because the world is stochastic. We don't, we can't expect the reward to come along to us in the way that


[User] Please transcribe this text

[Assistant] We hope it does based on the policy. Based on the way we choose to act.


[User] Please transcribe this text

[Assistant] And so there's a gamma there that over time,


[User] Please transcribe this text

[Assistant] As the reward is farther and farther into the future, this counts that reward.


[User] Please transcribe this text

[Assistant] To develop a strategy that maximizes the discounted future award. The sum. This discounted sum. And reinforcement learning.


[User] Please transcribe this text

[Assistant] There's a lot of approaches for coming up with a good policy. A near optimal, an optimal policy.


[User] Please transcribe this text

[Assistant] There's a lot of fun math there. You can try to construct a model.


[User] Please transcribe this text

[Assistant] That optimizes some estimate of this world.


[User] Please transcribe this text

[Assistant] You can try in a Monte Carlo way to just simulate that world and see how it unrolls.


[User] Please transcribe this text

[Assistant] And as it unrolls, you try to compute the optimal policy. Or what we'll talk about today is Q-learning. It's an off-policy approach. Where the policy is estimated as we go along. The policy is represented as a Q-function. The Q-function


[User] Please transcribe this text

[Assistant] Shown there on the left is, I apologize for the equations. I lied. There'll be some equations.


[User] Please transcribe this text

[Assistant] The input to the Q function is a state at time t, s, t, and an action that you choose to take in that state, a, t. And your goal is,


[User] Please transcribe this text

[Assistant] In that state, to choose an action which maximizes the reward in the next step.


[User] Please transcribe this text

[Assistant] And what Q learning does, and I'll describe the process, is it's able to approximate through experience the optimal Q function.


[User] Please transcribe this text

[Assistant] The optimal function that tells you how to act in any state of the world.


[User] Please transcribe this text

[Assistant] You just have to live it. You have to simulate this world. You have to move about it. You have to explore.


[User] Please transcribe this text

[Assistant] In order to see every possible state, try every different action. Get rewarded, get punished.


[User] Please transcribe this text

[Assistant] And figure out what is the optimal thing to do. That's done using this Bellman equation. On the left, the output is the new state, the estimate


[User] Please transcribe this text

[Assistant] The Q function estimate of the new state for new action.


[User] Please transcribe this text

[Assistant] And that's, this is the update rule at the core of Q-learning. You take the estimate, the old estimate, and add, based on the learning rate, alpha from 0 to 1,


[User] Please transcribe this text

[Assistant] Based on your new reward that you received at that time. So you've arrived in a certain state as T. You try to do an action.


[User] Please transcribe this text

[Assistant] And then you got a certain reward and you update your estimate of that state and action pair based on this rule. When the learning rate is zero,


[User] Please transcribe this text

[Assistant] You don't learn. When alpha is zero, you never change.


[User] Please transcribe this text

[Assistant] Your world view based on the new, the new incoming evidence. When alpha is one, you


[User] Please transcribe this text

[Assistant] Every time, change your evaluation, your world evaluation based on the new evidence.


[User] Please transcribe this text

[Assistant] And that's the key ingredient to reinforcement learning. First you explore, then you exploit.


[User] Please transcribe this text

[Assistant] First you explore in a non-greedy way and then you get greedy. You figure out what's good for you and you keep doing it.


[User] Please transcribe this text

[Assistant] Against the greatest world, uh, the greatest human players in the world in a game of Go, for example. As we'll talk about.


[User] Please transcribe this text

[Assistant] And the way you do that is you have an epsilon greedy policy that over time with a probability


[User] Please transcribe this text

[Assistant] Of 1 minus epsilon, you perform an optimal greedy action. With a probability of epsilon, you perform a random action. Random action being explore.


[User] Please transcribe this text

[Assistant] And so, as epsilon goes down from 1 to 0, you explore less and less.


[User] Please transcribe this text

[Assistant] So the algorithm here is really simple. On the bottom of the slide there. It's the algorithm version, the pseudocode version of the equation. The Bellman equation update.


[User] Please transcribe this text

[Assistant] You initialize your estimate of state action pairs.


[User] Please transcribe this text

[Assistant] Arbitrarily. A random number. Uh, this is an important point.


[User] Please transcribe this text

[Assistant] When you start playing or living or doing whatever you're doing, whatever you're doing with reinforcement learning or driving, you have no preconceived notion of what's good and bad. It's random.


[User] Please transcribe this text

[Assistant] Or however you choose to initialize it. And the fact that it learns anything is amazing.


[User] Please transcribe this text

[Assistant] The algorithm repeats the following step. You step into the world, observe an initial state. You select an action A.


[User] Please transcribe this text

[Assistant] So that action, if you're exploring, will be a random action. If you're greedily pursuing the best action you can, it will be the action that maximizes the Q function. You observe or reward after you take the action.


[User] Please transcribe this text

[Assistant] And a new state that you find yourself in. And then you update your estimate.


[User] Please transcribe this text

[Assistant] Of the previous state you were in, having given, taking that action, using that Bellman equation update.


[User] Please transcribe this text

[Assistant] Yes. Yes. Yeah. It's a single. The question was, is the Q function a single value? And yes, it's, it's just a single, uh, continuous value.


[User] Please transcribe this text

[Assistant] The way you model, so let's start this very simplistic world of Atari, uh, paddle. You think, you model it as a paddle that can move left and right, and there's some blocks, and you model the physics of the, uh, the ball.


[User] Please transcribe this text

[Assistant] Uh, that requires a lot of expert knowledge in that particular game. So you sit there hand crafting this model. That's hard to do even for a simplistic game. The other model you could take.


[User] Please transcribe this text

[Assistant] Is looking at this world in the way that humans do, visually. So take the model in as a set of pixels.


[User] Please transcribe this text

[Assistant] Just the model is all the pixels of the world. You know nothing about paddles or balls or physics or colors and points. They're just pixels coming in.


[User] Please transcribe this text

[Assistant] That seems like a ridiculous model of the world, but it seems to work for Atari. It seems to work for human beings. When you're born, you see there's light coming into your eyes, and you don't have any


[User] Please transcribe this text

[Assistant] As far as we know, you don't come with an instruction when you're born.


[User] Please transcribe this text

[Assistant] You know, there's people in the world and there is...


[User] Please transcribe this text

[Assistant] There's good guys and bad guys and there's, this is how you walk. No, all you get is light. Sound.


[User] Please transcribe this text

[Assistant] And you get to learn about the every single thing you think of as...


[User] Please transcribe this text

[Assistant] The way you model the world is a learned representation. And we'll talk about how a neural network does that. It learns to represent the world. But if we have to hand model the world, it's an impossible task.


[User] Please transcribe this text

[Assistant] It's, that's, that's the question. If we have to hand model the world, then that world better be a simplistic one.


[User] Please transcribe this text

[Assistant] That's a great question. So the question was, what is the robustness of this model? If, if the way you represent the world is at all even slightly different from the way you thought that world is, uh, that's not that well


[User] Please transcribe this text

[Assistant] It's already amazing that if you construct, if you have a certain input of the world, if you have a certain model of the world that you can learn anything, it's already amazing.


[User] Please transcribe this text

[Assistant] The question is, and it's an important one, is, uh, we'll talk a little bit about it, not about the world model, but the reward function. If the reward function is slightly different,


[User] Please transcribe this text

[Assistant] Yes, you can change it over. So the question was, uh, do you change the alpha value over time? And you certainly should change the alpha value over time. Yeah.


[User] Please transcribe this text

[Assistant] So the question was, what is the, the complex interplay of the epsilon function with the Q learning update? Um, that's a hundred percent fine tuned, hand tuned to the particular learning problem. So, uh, you certainly want to,


[User] Please transcribe this text

[Assistant] The more complex, the larger the number of states in the world and the larger the number of actions, the longer you have to


[User] Please transcribe this text

[Assistant] Wait before you decrease the epsilon to zero. But you have to play with it.


[User] Please transcribe this text

[Assistant] It's one of the parameters you have to play with, unfortunately, and there's quite a few of them.


[User] Please transcribe this text

[Assistant] Which is why you can't just drop a reinforcement learning agent into the world.


[User] Please transcribe this text

[Assistant] Oh, the effect in that sense. No, no, it's just a coin flip. And if, if that epsilon is 0.5,


[User] Please transcribe this text

[Assistant] Half the time you're gonna take a random action. So it's no, there's no specific, it's not like, uh, you'll take the best action.


[User] Please transcribe this text

[Assistant] And then with some probability, take the second bass and so on. I mean, you could certainly do that, but in the simple formulation that works is you just take a random action. Cause you don't want to have a preconceived notion of what's a good action to try when you're exploring. The whole point is you try crazy stuff. If it's a simulation.


[User] Please transcribe this text

[Assistant] Okay, so, uh, good question. So representation matters.


[User] Please transcribe this text

[Assistant] This is the question about how we represent the world. So we can think of this world of Breakout, for example, of this Atari game.


[User] Please transcribe this text

[Assistant] As a paddle that moves left and right. And the exact position of the different things it can hit. Construct this complex model. Uh, this expert driven model that has to fine tune it to this particular problem.


[User] Please transcribe this text

[Assistant] In practice, the more complex this model gets,


[User] Please transcribe this text

[Assistant] The worst that Bellman equation update that val- that trying to construct a Q function for every single combination of state and actions becomes too difficult because that function is too sparse and huge. So if you think of,


[User] Please transcribe this text

[Assistant] Of looking at this world in a general way, in the way human beings would, is as a collection of pixels, visually.


[User] Please transcribe this text

[Assistant] If you just take in a pixel, this game is a collection of 84 by 84 pixels, an image, an RGB image.


[User] Please transcribe this text

[Assistant] And then you look at not just the current image, but look at the temporal.


[User] Please transcribe this text

[Assistant] trajectory of those images so like if there's a ball moving you want to know about that movement so you look at


[User] Please transcribe this text

[Assistant] Four images. So current image and three images back.


[User] Please transcribe this text

[Assistant] And say they're gray scale with 256 gray levels. That size of the Q table that the Q value function


[User] Please transcribe this text

[Assistant] Whatever that number is, but it's certainly larger than the number of atoms in the universe.


[User] Please transcribe this text

[Assistant] You have to run the simulation long enough to touch at least a few times.


[User] Please transcribe this text

[Assistant] As Elon Musk says, you may need to, uh, run, you know, we live in a simulation. You may run, have to run another, a universe just to, to, uh, to compute the, the cube function in this case.


[User] Please transcribe this text

[Assistant] So that's where deep learning steps in. As, instead of modeling the world as a Q table, you estimate, you try to learn that function.


[User] Please transcribe this text

[Assistant] And so the takeaway from supervised learning, if you remember, that it's good at memorizing, we're good at memorizing data. The hope for reinforcement learning with a Q, Q learning is that we can extend


[User] Please transcribe this text

[Assistant] The occasional rewards we get to generalize over the operation, the actions you take in that world leading up to the rewards.


[User] Please transcribe this text

[Assistant] And the hope for deep learning is that we can move this reinforcement learning system


[User] Please transcribe this text

[Assistant] Into a world that doesn't need to be, uh, that can be defined arbitrarily can include


[User] Please transcribe this text

[Assistant] All the pixels in an Atari game can include all the pixels sensed by a drone or a robot or a car.


[User] Please transcribe this text

[Assistant] But still it needs a formalized definition of that world, which is much easier to do when you're able to take in sensors like an image. So deep Q learning, deep version.


[User] Please transcribe this text

[Assistant] So instead of learning a Q table, a Q function, we try and estimating that Q prime. We try to learn it using machine learning. So it tries to learn some parameters. This huge complex function, we try to learn it. And the way we do that,


[User] Please transcribe this text

[Assistant] As we have a neural network, the same kind that I showed that learned the numbers to map from an image to a, uh, classification of that image into a number. The same kind of network is used to take in a state and an action and produce a Q value.


[User] Please transcribe this text

[Assistant] As I said, with the Q table, it's initialized randomly. The Q function, this deep network, knows nothing in the beginning. All it knows is, in the simulated world, the rewards you get for a particular game. So you have to play time and time again and see the rewards you get for every single iteration of the game. But in the beginning, it knows nothing.


[User] Please transcribe this text

[Assistant] And it's able to learn to play better than human beings. This is a deep mind paper, playing Atari with deep reinforcement learning from 2013.


[User] Please transcribe this text

[Assistant] That's one of the key things that got everybody excited about the role of deep learning and artificial intelligence.


[User] Please transcribe this text

[Assistant] Using a convolutional neural network, which I'll talk about tomorrow. But it's a, it's a vanilla network, like any other. Like I talked about earlier today. Just a regular network. That takes the raw pixels, as I said.


[User] Please transcribe this text

[Assistant] And estimates that Q function from the raw pixels is able to play on many of those games better than a human being.


[User] Please transcribe this text

[Assistant] And the loss function that I mentioned previously. So,


[User] Please transcribe this text

[Assistant] Again, very vanilla loss function, very simple objective function.


[User] Please transcribe this text

[Assistant] The first one you'll probably implement, we have a tutorial on tensorflow. Squared error. So we take this Bellman equation, where the estimate is q,


[User] Please transcribe this text

[Assistant] The Q function estimate of state and action is the maximum.


[User] Please transcribe this text

[Assistant] Reward you get for taking any of the actions, uh, that takes you to any of the future states. And you try to take that action, observe the result of that action, and if the target is different,


[User] Please transcribe this text

[Assistant] That your learned target, the learned, what the function has learned is the expected reward in that case, is different than what you actually got. You adjust it. You adjust the weights in the network.


[User] Please transcribe this text

[Assistant] And this is exactly the process by which we learn how to exist in this pixel world. So you're mapping states and actions to a Q-value.


[User] Please transcribe this text

[Assistant] The algorithm is as follows. This is how we train it. We're given a transition.


[User] Please transcribe this text

[Assistant] S, current state, action taken in that state. R, the reward you get. And S' is what you, the state you find yourself in.


[User] Please transcribe this text

[Assistant] And so we replace the basic update rule in the previous pseudocode by taking a forward pass through the network, given that S, state. We look at what the predicted Q value is of that action.


[User] Please transcribe this text

[Assistant] We then do another forward pass through that network and see what we actually get. And then if we're totally off, we,


[User] Please transcribe this text

[Assistant] Punish, we, uh, back propagate the weights in a way that, uh, next time we'll make less of that mistake. And you repeat this process. And this is a, your,


[User] Please transcribe this text

[Assistant] Playing your s- this is a simulation. You're learning against yourself.


[User] Please transcribe this text

[Assistant] And again, the same rule applies here. Exploration versus exploitation.


[User] Please transcribe this text

[Assistant] Mostly exploring, and then you move towards an epsilon of zero.


[User] Please transcribe this text

[Assistant] And with Atari Breakout, this is the DeepMind paper result.


[User] Please transcribe this text

[Assistant] Is training epics on the x-axis. On the y-axis is the average action value and the average reward per episode.


[User] Please transcribe this text

[Assistant] I'll show why it's kind of an amazing result, but it's messy because there's a lot of tricks involved.


[User] Please transcribe this text

[Assistant] So it's not just putting in a bunch of pixels of a game and getting, uh, an agent that knows how to win at that game. There's a lot of pre-processing and playing with the data required.


[User] Please transcribe this text

[Assistant] So which is unfortunate, because the truth is messier than the hope.


[User] Please transcribe this text

[Assistant] But one of the critical tricks needed is called experience replay. So,


[User] Please transcribe this text

[Assistant] As opposed to letting an agent, so you're learning this big network that learn, that tries to build a model of what's good to do in the world and what's not. And you're learning as you go.


[User] Please transcribe this text

[Assistant] So with experience replay, you're keeping a track of all the things you did. And every once in a while, you look back into your memory and pull out some of those old experiences, the good old times, and train on those again. As opposed to letting the agent


[User] Please transcribe this text

[Assistant] Run itself into some local optima where it tries to learn a very subtle aspect of the game that actually in the global sense doesn't get you farther to winning the game. Very much like life. So here's the algorithm. Deep Q learning algorithm. Ceto code.


[User] Please transcribe this text

[Assistant] We initialize the action value function Q with random weights.


[User] Please transcribe this text

[Assistant] And observe initial state. Again, same thing. Select an action with a probability epsilon. Explore.


[User] Please transcribe this text

[Assistant] Otherwise, choose the best one based on the estimate provided by the neural network and then carry out the action. Observe the reward and store that experience in the replay memory.


[User] Please transcribe this text

[Assistant] And then sample random transition from replay memory.


[User] Please transcribe this text

[Assistant] So, uh, with a certain probability you bring those old times back to get yourself out of the local minima.


[User] Please transcribe this text

[Assistant] And then you train the cure, the Q network using the, the difference between your, what you actually got and your estimate. You repeat this process over and over.


[User] Please transcribe this text

[Assistant] Learns hardly anything, and it just keeps dying. If you look at, it goes from 5 to 4 to 3 to 2 to 1, those are the number of lives left.


[User] Please transcribe this text

[Assistant] Then after two hours of training on a single GPU, it learns to win a, win the,


[User] Please transcribe this text

[Assistant] You know, not die. Uh, rack up points and uh, learns to avoid the ball from passing.


[User] Please transcribe this text

[Assistant] Passing the paddle. Which is great, that's human level performance really.


[User] Please transcribe this text

[Assistant] Better than some humans, you know, but, uh, it still dies sometimes, so it's very human level. And then after four hours, it does something really amazing. It figures out...


[User] Please transcribe this text

[Assistant] How to win at the game in a very lazy way, which is drill.


[User] Please transcribe this text

[Assistant] A hole through the, the blocks up to the top and get the ball stuck up there. And then it does all the hard work for you. That minimizes the probability of the ball getting past your paddle because it's just stuck in the, in the, uhm, in the blocks up top.


[User] Please transcribe this text

[Assistant] So that might be something that you wouldn't even figure out to do yourself.


[User] Please transcribe this text

[Assistant] I need to sort of pause here to clearly explain what's happening. The input to this algorithm is just the pixels of the game. It's the same thing that human beings take in when they take the visual perception. And it's able to learn under this constrained


[User] Please transcribe this text

[Assistant] Definition of what is a reward and a punishment. It's able to learn to get a high reward.


[User] Please transcribe this text

[Assistant] A very small example of it, but it's general. It's general purpose. It knows nothing about games. It knows nothing about paddles or physics. It's just taking sensory input of the game. And they've, uh, did the same thing for a bunch of different games in Atari. And,


[User] Please transcribe this text

[Assistant] What's shown here in this plot on the X axis is a bunch of different games from Atari and on the Y axis is a percentile.


[User] Please transcribe this text

[Assistant] Where 100% is about the best that human beings can do. Meaning it's the score that human beings will get. So everything about there in the middle, everything to the left of that, is far exceeding human level performance.


[User] Please transcribe this text

[Assistant] And below that is on par or worse than a human level performance. So you can learn all, so many boxing, pinball.


[User] Please transcribe this text

[Assistant] All of these games, and it doesn't know anything about any of the individual games. It's just taking in pixels. It's just as if you put a human being beside, behind any of these, uh, games and ask them to learn to be


[User] Please transcribe this text

[Assistant] And there's been a lot of improvements in this algorithm recently. Yes, question?


[User] Please transcribe this text

[Assistant] Nope. There's no, so the question was do they customize the model for game, for a particular game? And no. The point, you could of course, but the point is it doesn't need to be customized for the game. But,


[User] Please transcribe this text

[Assistant] That it's still only on Atari games, right? So the question whether this is transferable to driving, perhaps not.


[User] Please transcribe this text

[Assistant] Right, you play the game, well you do. But now you don't have the, well yeaah, you play one step of the game. So you take action in a state.


[User] Please transcribe this text

[Assistant] And then you observe that. So you have the, the simulation. I mean that's really, uh,


[User] Please transcribe this text

[Assistant] That's one of the biggest problems here is you require the simulation to, uh, in order to get the ground truth.


[User] Please transcribe this text

[Assistant] So, that's a great question or comment. The comment was that for a lot of these situations,


[User] Please transcribe this text

[Assistant] The reward function might not change at all depending on your actions. The rewards are really...


[User] Please transcribe this text

[Assistant] Most of the time delayed 10, 20, 30 steps down the line. Which is why it is amazing that this works at all. That it's learning, it's learning locally.


[User] Please transcribe this text

[Assistant] And through that process of simulation of hundreds to thousands of times it runs through the game, it's able to learn.


[User] Please transcribe this text

[Assistant] If you just pause, look at the math of it, it's very simple math, and look at the result, it's incredible.


[User] Please transcribe this text

[Assistant] So there's a lot of improvements. This one called the General Reinforcement Learning Architecture or GRILLA.


[User] Please transcribe this text

[Assistant] The cool thing about this, in the simulated world at least, is that you can...


[User] Please transcribe this text

[Assistant] You could do both the simulation and the distributed way. You could do the learning and the distributed way.


[User] Please transcribe this text

[Assistant] You can, you can generate experiences, which is what this kind of diagram shows. You can either from human beings or from simulation. So for example, the way that


[User] Please transcribe this text

[Assistant] The way that AlphaGo, uh, the DeepMind team has beat the game of Go, is they, they learn from both expert games, and by playing itself.


[User] Please transcribe this text

[Assistant] So you can do this in a distributed way, and you can do the learning in a distributed way, so you can scale. And in this particular case, the, uh, Gorilla, uh, has achieved a better result than the, uh, DQN network. Eh, that's part of their nature paper.


[User] Please transcribe this text

[Assistant] Driving for a second here. Where does reinforcement learning


[User] Please transcribe this text

[Assistant] Where reinforcement learning can step in and help.


[User] Please transcribe this text

[Assistant] So this is back to the open question that I asked yesterday. Is driving closer to chess or to everyday conversation? Chess meaning it can be formalized in a simplistic way.


[User] Please transcribe this text

[Assistant] You just navigate that constrained space. You choose to move left, you choose to move right in a lane, you choose to speed up or slow down.


[User] Please transcribe this text

[Assistant] Well, if it's a game like chess, which we'll assume for today, as opposed to for tomorrow, for today, we're going to go with the one on the left. And we're going to look at deep traffic. Here's this game, a simulation,


[User] Please transcribe this text

[Assistant] Where the goal is to achieve the highest average speed you can on this seven lane highway full of cars.


[User] Please transcribe this text

[Assistant] And so, as a side note for students, the requirement is they have to follow the tutorial that I'll present a link for at the end of this presentation.


[User] Please transcribe this text

[Assistant] And what they have to do is achieve a speed, build a network that achieves a speed of 65 miles an hour or higher. There is a leaderboard.


[User] Please transcribe this text

[Assistant] And you get to submit the model you come up with, with a simple click of a button. So all of this runs in the browser, which is also another amazing thing. And then you immediately, or relatively so, make your way up the leaderboard.


[User] Please transcribe this text

[Assistant] So let's look, let's zoom in. What is this, uh, what is this world, two dimensional world of traffic is? What is it?


[User] Please transcribe this text

[Assistant] Look like for the, uh, intelligence system. We discretize that world into a grid.


[User] Please transcribe this text

[Assistant] Shown here on the left. That's the representation of the state. There's seven lanes, and every single lane is broken up into blocks, spatially.


[User] Please transcribe this text

[Assistant] And if there's a car in that block, the length of a car is about three blocks. Three of those grid blocks.


[User] Please transcribe this text

[Assistant] And then the red car is you. That's the thing that's running in the intelligent agent.


[User] Please transcribe this text

[Assistant] There is on the left, is the current speed of the red car. It actually says MIT on top.


[User] Please transcribe this text

[Assistant] And if your network sucks, then that number is gonna get to be negative.


[User] Please transcribe this text

[Assistant] Uh, you can also change with a drop down the simulation speed. From normal on the left to fast on the right. So normal is...


[User] Please transcribe this text

[Assistant] The, there's a drop down for different display options. Uh, the default is none in terms of stuff you show on the road. Then there is the learning input.


[User] Please transcribe this text

[Assistant] Which is the, while the whole space is discretized.


[User] Please transcribe this text

[Assistant] You can choose what your car sees.


[User] Please transcribe this text

[Assistant] And that's, you can choose how far ahead it sees, behind, how far to the left and right it sees.


[User] Please transcribe this text

[Assistant] And so by choosing the learning input, to visualize the learning input, you get to see what you set that input to be. Then there is the safety system. This is a system that protects you from yourself. The way we've made this game is that it operates under something similar


[User] Please transcribe this text

[Assistant] If you have some intelligence, if you drive and you have, uhm, adaptive cruise control in your car, it operates in the same way. When it gets close to the car in front, it slows down for you.


[User] Please transcribe this text

[Assistant] And it doesn't let you run the car to the left of you, to the right of you, off the road.


[User] Please transcribe this text

[Assistant] So it constrains the movement.


[User] Please transcribe this text

[Assistant] Capabilities of your car in such a way that you don't hit anybody. Because then it would have to simulate collisions and that would just be a mess. So it protects you from that.


[User] Please transcribe this text

[Assistant] And so you can choose to visualize that quote unquote safety system.


[User] Please transcribe this text

[Assistant] With the visualization box. And then you can also choose to visualize the full map.


[User] Please transcribe this text

[Assistant] This is the full occupancy map that you get if you would like to provide as input to the network.


[User] Please transcribe this text

[Assistant] Now that input for every single grid, that, it's a number. It's not just a zero one whether there's a car in there. It's...


[User] Please transcribe this text

[Assistant] The maximum speed limit which is 80 miles per hour.


[User] Please transcribe this text

[Assistant] Don't, don't go, don't get crazy. 80 miles an hour is the speed limit.


[User] Please transcribe this text

[Assistant] That block when it's empty is set to the 85 miles, uh, 80 miles an hour.


[User] Please transcribe this text

[Assistant] And when it's occupied, it's set to the number that's uh, the speed of the car.


[User] Please transcribe this text

[Assistant] And then the blocks that you, the red car is occupying is set to the number, to a very large number, much higher than the speed limit.


[User] Please transcribe this text

[Assistant] So safety system, here shown in red.


[User] Please transcribe this text

[Assistant] Are the parts of the grid that are not, that your car can't move into? Question.


[User] Please transcribe this text

[Assistant] Yes, uh, the question was, what was the third option I just mentioned, and uh, it's you, the red car itself.


[User] Please transcribe this text

[Assistant] You yourself, the blocks underneath that car are set to a really high number. It's a way for the algorithm to know, for the learning algorithm to know that this, these blocks are special.


[User] Please transcribe this text

[Assistant] Shows red here if the car can't move into those blocks.


[User] Please transcribe this text

[Assistant] In terms of, uh, when it lights up red, it means the car can't speed up anymore in front of it. And when the blocks to the left or to the right light up as red, that means you can't change lanes to the left or right. On the right of the slide,


[User] Please transcribe this text

[Assistant] You're free to go, free to do whatever you want. That's what that indicates. Cause all the blocks are yellow.


[User] Please transcribe this text

[Assistant] Safety system says you're free to choose any of the five actions. And the five actions are move left, move right, stay in place, accelerate, or slow down. And those actions are given as input.


[User] Please transcribe this text

[Assistant] That action is, is, uh, that's what's produced by the, what's called here the brain.


[User] Please transcribe this text

[Assistant] The brain takes in the current state as input, the last reward and produces and through, and learns, uses that reward to train the network through backward function there, back propagation and then ask the brain given the current state to give it the next action.


[User] Please transcribe this text

[Assistant] With a forward pass, the forward function. You don't need to know the operation of this function in particular. This is not something you need to worry about, but you can if you want. You can customize this learning step.


[User] Please transcribe this text

[Assistant] There is, by the way, what I'm describing now, there's just a few lines of code right there in the browser.


[User] Please transcribe this text

[Assistant] That you can change it immediately, uh, well, with the press of a button.


[User] Please transcribe this text

[Assistant] You don't need to have any special hardware. You don't need to do anything special. And the tutorial cleanly outlines exactly all of these steps. But it's kind of amazing that you can design a deep neural network that's part of the reinforcement learning agent. So it's a deep Q learning agent right there in the browser.


[User] Please transcribe this text

[Assistant] So you can choose the side, the lane side variable which controls how many


[User] Please transcribe this text

[Assistant] Lanes to the side you see, so when that value is zero, you only look forward. When that value is one, you have one lane to the left, one valley to the right. It's really the lane, the radius of your perception system. Patches ahead is how far ahead you look. Patches behind is how far behind you look.


[User] Please transcribe this text

[Assistant] And so for example here, the lane side equals two, that means it looks two to the left, two to the right.


[User] Please transcribe this text

[Assistant] Obviously, if to the right is off road.


[User] Please transcribe this text

[Assistant] It provides a value of zero in those blocks.


[User] Please transcribe this text

[Assistant] If we set the patches behind to be 10, it looks 10 patches back. Behind starting at the one patch back is starting from the front of the car.


[User] Please transcribe this text

[Assistant] The scoring for the evaluation for the competition is your average speed over a predefined period of time.


[User] Please transcribe this text

[Assistant] And so the method we do, we use to collect that speed is we, uh, we run the agent 10 runs about 30 simulated minutes of game each. And take the median speed of the 10 runs. That's the score.


[User] Please transcribe this text

[Assistant] So, uh, so given that we've gotten some, for this, for this code, recently gotten some publicity online, unfortunately.


[User] Please transcribe this text

[Assistant] This might be a dangerous thing to say. It's no cheating possible, but because it's done server side and we did, uh, this is JavaScript and it runs in the browser. It's hopefully a sandbox. So we can't do anything tricky, but we dare you to try.


[User] Please transcribe this text

[Assistant] You can, uh, try it locally to get an estimate. And I'll...


[User] Please transcribe this text

[Assistant] There's a button that says evaluate and it gives you a score right back of how well you're doing with the current network.


[User] Please transcribe this text

[Assistant] That button is, start evaluation run, you press the button, it does a progress bar, and it gives you the average speed.


[User] Please transcribe this text

[Assistant] Where you modify all the variables I mentioned and the tutorial describes this in detail.


[User] Please transcribe this text

[Assistant] And then once you're ready, you modify a few things. You can press apply code. It restarts it.


[User] Please transcribe this text

[Assistant] Kills all the training that you've done up to this point or resets it. And starts the training again. So save often. And there's a save button. So the training, uh, is done on a separate thread. In web workers.


[User] Please transcribe this text

[Assistant] Which are exciting things that allow you to allow JavaScript to run amazingly in a, in a, in a,


[User] Please transcribe this text

[Assistant] On multiple CPU cores in a parallel way.


[User] Please transcribe this text

[Assistant] So, the simulation that scores this, or the, uh, sorry, the training is done a lot faster than real time.


[User] Please transcribe this text

[Assistant] A thousand frames a second, a thousand movement steps a second. This is all in JavaScript.


[User] Please transcribe this text

[Assistant] And the net state gets shipped to the main simulation from time to time as the training goes on. So all you have to do is press run training.


[User] Please transcribe this text

[Assistant] And it trains, and the car behaves better over time.


[User] Please transcribe this text

[Assistant] Maybe I'll actually show it in the browser to...


[User] Please transcribe this text

[Assistant] Let's see if we're, well, is it gonna mess up? All good?


[User] Please transcribe this text

[Assistant] So here's the game. When it starts, this is running live in the browser.


[User] Please transcribe this text

[Assistant] In the browser, a neural network. So currently it's not very good. It's driving at two miles an hour and watching everybody pass. So,


[User] Please transcribe this text

[Assistant] What's being shown live is the loss function.


[User] Please transcribe this text

[Assistant] Which is pretty poor. So in order to train, like I said, thousand frames a second, you just press the run training button.


[User] Please transcribe this text

[Assistant] And pretty quickly it learns based on the network you specify in the code box. How to...


[User] Please transcribe this text

[Assistant] And based on the input and all the things that I mentioned, training finished.


[User] Please transcribe this text

[Assistant] We on purpose put in a network that's not very good in there. So right now it won't on average be doing that well. But it does better than standing there in place. And then you can do the start evaluation run.


[User] Please transcribe this text

[Assistant] To simulate the network much faster than real time to see how well it does. This is a similar evaluation step that we take.


[User] Please transcribe this text

[Assistant] When determining where you stand on the leaderboard. The current average speed.


[User] Please transcribe this text

[Assistant] I may be logged in, maybe not. If you're logged in...


[User] Please transcribe this text

[Assistant] And then, all you have to do is log in. Ah, this is the most flawless demo of my life.


[User] Please transcribe this text

[Assistant] Thank you for your submission. And so now my submission is entered as Lex in the leaderboard of my 56.56 or whatever it was. So I dare all of you to try to beat that.


[User] Please transcribe this text

[Assistant] You could do so by pressing the save code button. That saves the config, the various JavaScript configurations and that saves the network layout to file.


[User] Please transcribe this text

[Assistant] And then you can load from file as well. Again, danger, it overrides the code for you.


[User] Please transcribe this text

[Assistant] Make sure that you train the network. We don't train it for you. You bring us, you submit a model and you have to press train.


[User] Please transcribe this text

[Assistant] And it gets evaluated in time, it enters a queue to get evaluated. This is public facing so the queue can grow pretty big.


[User] Please transcribe this text

[Assistant] And it goes to that queue, evaluates it, and then depending on where you stand, you get added to the leaderboard, showing the top 10 entries. You can resubmit often.


[User] Please transcribe this text

[Assistant] Okay, we're using, we're using code.


[User] Please transcribe this text

[Assistant] Now implementation of, uh, neural networks done in just JavaScript.


[User] Please transcribe this text

[Assistant] By Andre Karpathy from Stanford Now Open AI. ComNetJS is the library.


[User] Please transcribe this text

[Assistant] And what's being visualized there is also being visualized in the game is the inputs to the network. In this case it's 135 inputs. You can also specify not just the


[User] Please transcribe this text

[Assistant] Not just the, how far ahead behind you see into the left and to the right. You can specify how far back in time you look as well.


[User] Please transcribe this text

[Assistant] And so what's visualized there is the input to the network 135.


[User] Please transcribe this text

[Assistant] Neurons, and then the output, regression, similar to the kind of output we saw with numbers, where there's ten outputs saying if it's a zero, one, there are nine. Here the output is one of the five actions. Left, right, stay in place, speed up or slow down. The CognetJS settings, is you can select the number of inputs.


[User] Please transcribe this text

[Assistant] If you want to mess with this stuff, this is all the stuff you don't need to mess with, because we already give you the variables of lane side and patches ahead and so on. You can select the number of actions, the temporal window, and the network size.


[User] Please transcribe this text

[Assistant] This is the input, the size of the input. Again, all of this is in the tutorial, just to give you a little outline. There is a, the first fully connected layer has ten neurons.


[User] Please transcribe this text

[Assistant] With Rayleigh activation functions, same kind of smooth function that we talked about before, and the regression layer for the output.


[User] Please transcribe this text

[Assistant] And there's a bunch of other messy options that you can play with, if you dare.


[User] Please transcribe this text

[Assistant] But those aren't the ones I mentioned before, it's really the important ones. It's selecting the number of layers, the size of those layers. You get to build your own very neural network that drives.


[User] Please transcribe this text

[Assistant] And the actual learning is done with the backward propagation and then the


[User] Please transcribe this text

[Assistant] That returns the action by doing a forward pass to the network.


[User] Please transcribe this text

[Assistant] In case you're interested in this kind of stuff, there is an amazingly cool code editor, uh, that's a monoco editor.


[User] Please transcribe this text

[Assistant] That's, it just works, it does some auto completion, so you get to play with it, makes everything very convenient in terms of code editing. A lot of this visualization of the game,


[User] Please transcribe this text

[Assistant] And the simulation, and the simulation we'll talk about tomorrow is done in the browser using HTML5 Canvas. So here is a simple specification of a blue box with Canvas. And this is very efficient and easy to work with.


[User] Please transcribe this text

[Assistant] And the thing that a lot of us are excited about, a very subtle one, but that you can not just run


[User] Please transcribe this text

[Assistant] So with the V8 engine, JavaScript has become super fast. You can, you could train neural networks in the browser. That's already amazing. And then with web workers, as long as you have Chrome, a modern browser.


[User] Please transcribe this text

[Assistant] Is, you can run multiple processes in separate threads.


[User] Please transcribe this text

[Assistant] So you can do a lot of stuff, you can do visualization separately, and you can train in separate threads. Very cool.


[User] Please transcribe this text

[Assistant] Okay, so the tutorial is cars.mit.edu and deep traffic. We won't put these links on the website for a little bit.


[User] Please transcribe this text

[Assistant] Which, we don't want those to leak out, especially with the claims that you can't cheat.


[User] Please transcribe this text

[Assistant] And while it's pretty efficient in terms of running time, so everything is running on your machine, client side.


[User] Please transcribe this text

[Assistant] It's still, you have to pull some images here and pull some of the code. So the tutorial is on cars.mit.edu slash deep traffic and the simulation is deep traffic JS. So cars.mit.edu slash deep traffic JS. I encourage you to go there, play with the network, submit your code.


[User] Please transcribe this text

[Assistant] And win the very special prize. It is a pretty cool one, but we're still working on it. There is a prize, I swear. Alright, so let's...


[User] Please transcribe this text

[Assistant] Take a pause and think about what we talked about today.


[User] Please transcribe this text

[Assistant] The most exciting accomplishment, I think...


[User] Please transcribe this text

[Assistant] Is when the game, when I first started...


[User] Please transcribe this text

[Assistant] As a freshman, took Intro to Artificial Intelligence.


[User] Please transcribe this text

[Assistant] It was said that it's a game that's impossible for machines to beat. Because of the combinatorial complexity, there's just a sheer number of options. It's so much more complex than chess. And so,


[User] Please transcribe this text

[Assistant] The most amazing accomplishment of deep reinforcement learning to me is the design of AlphaGo.


[User] Please transcribe this text

[Assistant] When for the first time the world champion in Go was beaten by DeepMind's AlphaGo.


[User] Please transcribe this text

[Assistant] And the way they did it, and this is I think very relevant to driving, is you start


[User] Please transcribe this text

[Assistant] So, you take expert games to construct a network.


[User] Please transcribe this text

[Assistant] First, so you look, you don't play against yourself, the agent doesn't play against itself, but they learn from expert games.


[User] Please transcribe this text

[Assistant] So there's some human ground truth. This human ground truth represents reality.


[User] Please transcribe this text

[Assistant] So for driving this is important. We have a, well we're starting to get a lot of data where video of drivers is being recorded. So we can learn on that data before we then run the agents through simulation where it learns much larger magnitudes of data sets through simulation. And they did just that.


[User] Please transcribe this text

[Assistant] This is probably one of my favorite videos of all time, but I just recently saw it so I could just watch this for hours. But it's a reminder that you can't


[User] Please transcribe this text

[Assistant] You can't trust your first estimates of a reward function.


[User] Please transcribe this text

[Assistant] Again, all the references are available online for these slides. We'll put up the slides.


[User] Please transcribe this text

[Assistant] I imagine you might have, if you want to come down and talk to us for questions for the either Docker or JavaScript question.


[User] Please transcribe this text

[Assistant] It's, so the question was, what is the visualization? You're seeing in deep traffic, you're seeing a car move about. How is it moving? It's moving based on the latest snapshot of the network you trained.


[User] Please transcribe this text

[Assistant] So it's just visualizing for you, just for fun, the network you train, most recently.


[User] Please transcribe this text

[Assistant] Okay, so if you, if people have questions, stick around afterwards. Just details on Docker and, and uh, yeaah.


[User] Please transcribe this text

[Assistant] Wait, is that grass? Tell me the truth, that's an order.


[User] Please transcribe this text

[Assistant] Oooh, hoho, and what a marvelous big leaf, little milk-brown hair.


[User] Please transcribe this text

[Assistant] Hold this under here. I'm going to show you a trick. Really?


[User] Please transcribe this text

[Assistant] I've seen many seasons come and go. Each time I learn a bit more.


[User] Please transcribe this text

[Assistant] Come on, back to the oak tree for some summer deliciousness.


[User] Please transcribe this text

[Assistant] Oh, isn't it beautiful? Very. But, what is it?


[User] Please transcribe this text

[Assistant] This is where the tracks stop. And this is where the holly came from. But there's no one here.


[User] Please transcribe this text

[Assistant] It looks like someone is surprising us with these beautiful gifts. But who? Hmm, look at these tracks.


[User] Please transcribe this text

[Assistant] Like I told the police. She was a good egg. And she gave me those photos.


[User] Please transcribe this text

[Assistant] Pete was cruising home off duty and told me he saw you near that garbage dump. Now I've ordered... Told you to stay away from that place. It's dangerous.


[User] Please transcribe this text

[Assistant] You got no idea what kind of scum are living out there. Stay away. I'm serious.


[User] Please transcribe this text

[Assistant] I'm sick of your disrespect. Tell me the truth. That's an order. Who's is it?


[User] Please transcribe this text

[Assistant] I don't like strangers in my home. Especially dopers. So you're bringing drugs into my home. How about if I call the police? That'd screw up your spotless Blackwell record.


[User] Please transcribe this text

[Assistant] You do seem to get around, Max. I'm sick of you losers dragging Chloe down. Missy, you sure do like to pop up and start trouble. Like this afternoon. You don't have anything smart to say now, do you?


[User] Please transcribe this text

[Assistant] And Max, if I see you here again, you'll learn all about real trouble.


[User] Please transcribe this text

[Assistant] During my long and intimate acquaintance with Mr. Sherlock Holmes, I had never heard him refer to his relations, and hardly ever to his own early life.


[User] Please transcribe this text

[Assistant] This reticence upon his part had increased the somewhat inhuman effect which he produced upon me, until sometimes I found myself regarding him as an isolated phenomenon, a brain without a heart, as deficient in human sympathy as he was preeminent in intelligence.


[User] Please transcribe this text

[Assistant] His aversion to women and his disinclination to form new friendships were both typical of his unemotional character.


[User] Please transcribe this text

[Assistant] But not more so than his complete suppression of every reference to his own people.


[User] Please transcribe this text

[Assistant] I'd come to believe that he was an orphan with no relatives living.


[User] Please transcribe this text

[Assistant] But one day, to my very great surprise, he began to talk to me about his brother.


[User] Please transcribe this text

[Assistant] It was after tea on a summer evening, and the conversation which had roamed in a desultory, spasmodic fashion, from golf clubs to the causes of the change in the obliquity of the ecliptic, came round at last to the question of atavism and hereditary aptitudes.


[User] Please transcribe this text

[Assistant] The point under discussion was how far any singular gift in an individual was due to his ancestry, and how far to his own early training.


[User] Please transcribe this text

[Assistant] In your own case," said I, from all that you have told me, it seems obvious that your faculty of observation and your peculiar facility for deduction are due to your own systematic training." To some extent, he answered thoughtfully.


[User] Please transcribe this text

[Assistant] My ancestors were country squires, who appear to have led much the same life as is natural to their class. But, nonetheless, my turn that way is in my veins, and may have come with my grandmother, who was a sister of Vernet, the French artist. Art in the blood is liable to take the strangest forms. But how do you know that it is hereditary?


[User] Please transcribe this text

[Assistant] Because my brother Mycroft possesses it in a larger degree than I do. This was news to me indeed.


[User] Please transcribe this text

[Assistant] If there were another man with such singular powers in England...


[User] Please transcribe this text

[Assistant] How was it that neither police nor public had heard of him?


[User] Please transcribe this text

[Assistant] I put the question, with a hint that it was my companion's modesty which made him acknowledge his brother as his superior. Holmes laughed at my suggestion.


[User] Please transcribe this text

[Assistant] My dear Watson," said he, I cannot agree with those who rank modesty among the virtues.


[User] Please transcribe this text

[Assistant] To the logician all things should be seen exactly as they are, and to underestimate oneself is as much a departure from truth as to exaggerate one's own powers.


[User] Please transcribe this text

[Assistant] When I say, therefore, that my croft has better powers of observation than I, you may take it that I am speaking the exact and literal truth. Is he your junior? Seven years my senior. How comes it that he's unknown? Oh, he's very well known in his own circle. Where, then? Well, in the Diogenes Club, for example.