[User] Please transcribe this text

[Assistant] As usual, the underlying networks from AlexNet to VGGnet and to now ResNet have been one of the big reasons for the improvements of these networks to be able to perform the segmentation.


[User] Please transcribe this text

[Assistant] So naturally they mirrored the ImageNet challenge performance in adapting these networks. So the state of the art uses ResNet or similar networks. Conditional random fields.


[User] Please transcribe this text

[Assistant] For smoothing based on the input image intensities and the dilated


[User] Please transcribe this text

[Assistant] Convolution that maintains the computational cost but increases the resolution of the upsampling throughout the intermediate feature maps.


[User] Please transcribe this text

[Assistant] And that takes us to the state of the art that we used to produce the images, uh, to produce the images for the competition.


[User] Please transcribe this text

[Assistant] ResNet-DUC for dance upsampling convolution. Instead of bilinear upsampling, you make the upsampling learnable.


[User] Please transcribe this text

[Assistant] You learn the upscaling filters. That's on the bottom. That's really the key part that made it work.


[User] Please transcribe this text

[Assistant] There should be a theme here. Sometimes the biggest addition that can be done is parameterizing one of the aspects of the network that you've taken for granted. Letting the network learn that aspect. And the other...


[User] Please transcribe this text

[Assistant] I'm not sure how important it is to the success, but it's a cool little addition. It's a hybrid dilated convolution.


[User] Please transcribe this text

[Assistant] As I showed that visualization where the convolution is spread apart a little bit in the input from the input to the output, the steps of that dilated convolution filter


[User] Please transcribe this text

[Assistant] When they're changed, it produces a smoother result. Because when it's kept the same,


[User] Please transcribe this text

[Assistant] There, certain input pixels get a lot more attention than others. So losing that favoritism is what's achieved by using a variable different dilation rate.


[User] Please transcribe this text

[Assistant] Those are the two tricks, but really the biggest one is the parametrization of the upscaling filters.


[User] Please transcribe this text

[Assistant] Okay, so that's what we're, that's what we use to generate that data and that's what we provide you the code with, uh, if you're interested in competing in PsychFuse.


[User] Please transcribe this text

[Assistant] The other aspect here, that everything we talked about from the classification to the, uh, segmentation to making sense of images is, the information about, uh, time, the temporal dynamics of the scene is thrown away.


[User] Please transcribe this text

[Assistant] And for the driving context, for the robotics context, and what we'd like to do with SegFuse for the segmentation, dynamic scene segmentation context of when you try to interpret what's going on in the scene over time and use that information. Time is essential.


[User] Please transcribe this text

[Assistant] Through the 2D projection of an image is fascinating and there's a lot of set of open problems there. So flow is what's very helpful to, as a starting point, to help us understand how these pixels move. Flow, optical flow,


[User] Please transcribe this text

[Assistant] Dense optical flows, the computation, our best...


[User] Please transcribe this text

[Assistant] And moved in a, in a temporally.


[User] Please transcribe this text

[Assistant] There's two images in 30 frames a second. There's one image at time zero. The other is 33.3 milliseconds later. And the dense optical flow is our best estimate of how each pixel in the input image moved to in the output image.


[User] Please transcribe this text

[Assistant] The optical flow for every pixel produces a direction of where we think that pixel moved and the magnitude of how far it moved. That allows us to take information that we detected about the first frame and try to propagate it forward.


[User] Please transcribe this text

[Assistant] This is the competition. It's to try to segment an image and propagate that information forward. For manual annotation


[User] Please transcribe this text

[Assistant] Of a, of a image, so this kind of coloring book annotation where you color every single pixel in a state of the art data set.


[User] Please transcribe this text

[Assistant] For driving cityscapes that it takes 1.5, 1.5 hours, 90 minutes to do that coloring. That's 90 minutes per image.


[User] Please transcribe this text

[Assistant] That's extremely long time. That's why there doesn't exist today a data set. And in this class we're going to create one.


[User] Please transcribe this text

[Assistant] Of segmentation of these images. Through time. Through video.


[User] Please transcribe this text

[Assistant] So long videos where every single frame is fully segmented. That's still an open problem that we need to solve.


[User] Please transcribe this text

[Assistant] Slow is a piece of that. And we also provide you


[User] Please transcribe this text

[Assistant] The, this computed state of the art flow using flow net 2.0.


[User] Please transcribe this text

[Assistant] Use neural networks to learn the optical flow, the dense optical flow.


[User] Please transcribe this text

[Assistant] And it did so with two kinds of architectures. Flow net S, flow net simple, and flow net core, flow net C.


[User] Please transcribe this text

[Assistant] The, uh, simple one is simply taking the two images, so what's, what's the task here? There's two images, and you, you want to produce from those two images, they follow each other in time, 33.3 milliseconds apart, and, uh, your task is, is the output to produce the dense optical flow.


[User] Please transcribe this text

[Assistant] So for the simple architecture, you just stack them together. Each are RGB, so it produces a six channel input to the network. There's a lot of convolution, and finally it's the same kind of process as the fully convolution neural networks to produce the optical flow. Then there is flow net correlation.


[User] Please transcribe this text

[Assistant] architecture where you perform some convolutions separately before using a correlation layer to combine the feature maps.


[User] Please transcribe this text

[Assistant] And different data sets and different applications. So FlowNet 2.0 in December.


[User] Please transcribe this text

[Assistant] 2016 is one of the state of the art frameworks, code bases that we used to generate the Daydoll show.


[User] Please transcribe this text

[Assistant] Combines the FlowNet S and FlowNet C, and improves over the initial FlowNet, producing a smoother flow field, preserves the fine motion detail along the edges of the objects,


[User] Please transcribe this text

[Assistant] And it runs extremely efficiently. Depending on the architecture, there's a few variants, either 8 to 140 frames a second.


[User] Please transcribe this text

[Assistant] And the process there is essentially one that's common across various applications of deep learning is stacking these networks together.


[User] Please transcribe this text

[Assistant] The very interesting aspect here that


[User] Please transcribe this text

[Assistant] We're still exploring and again applicable in all of deep learning. In this case, it seemed that there was a strong effect in taking sparse, small, multiple data set and doing the training. The order of which those data sets were used for the training process mattered a lot. That's very interesting.


[User] Please transcribe this text

[Assistant] So, using FlowNet 2.0, here's the data set we're making available for PsychFuse the competition.


[User] Please transcribe this text

[Assistant] Original video, driving around, uh, Cambridge. Then we're providing the ground truth for a training set.


[User] Please transcribe this text

[Assistant] So that training set for every single frame, 30 frames a second, we're providing the segmentation, frame to frame to frame. Segmented on Mechanical Turk. We're also providing the output.


[User] Please transcribe this text

[Assistant] Of the network that I mentioned, the state of the art segmentation network. That's pretty damn close to the ground truth. But still not.


[User] Please transcribe this text

[Assistant] And our task is, this is the interesting thing is,


[User] Please transcribe this text

[Assistant] Well there's two options. One is to take the output of this network


[User] Please transcribe this text

[Assistant] And use other networks to help you propagate the information better. So with this segmentation,


[User] Please transcribe this text

[Assistant] The output of this network does is it only takes a frame by frame by frame. It's not using the temporal information at all.


[User] Please transcribe this text

[Assistant] So the question is, can we figure out a way, can we figure out tricks to use temporal information to improve this segmentation so it looks more like this segmentation?


[User] Please transcribe this text

[Assistant] And we're also providing the optical flow from frame to frame to frame. So the optical flow based on flow net 2.0 of how each of the pixels moved.


[User] Please transcribe this text

[Assistant] Okay, and that forms the SegFuse competition. 10,000 images, and the task is to submit code. We have starter code in Python and on GitHub.


[User] Please transcribe this text

[Assistant] To take in the original video. Take in for the training set. The ground truth. The segmentation from the state of the art segmentation network. The optical flow from the state of the art optical flow network.


[User] Please transcribe this text

[Assistant] And taking that together to improve the stuff on the bottom left, the segmentation. To try to achieve the ground truth on the top right.


[User] Please transcribe this text

[Assistant] I'd like to thank you. Tomorrow at 1pm is Waymo in Stata.


[User] Please transcribe this text

[Assistant] 32123, the next lecture next week will be on deep learning for sensing the human, understanding the human, and we will release online only lecture on capsule networks and GANs, general adversarial networks. Thank you very much.


[User] Please transcribe this text

[Assistant] Football, Max. Big football. Step away.


[User] Please transcribe this text

[Assistant] Regular neural networks, fully connected neural networks. We've talked about convolutional neural networks that work with images. We've talked about reinforcement, deep reinforcement learning where we plug in a neural network into a reinforcement learning algorithm.


[User] Please transcribe this text

[Assistant] When an agent, when a system has to not only perceive the world but also act in it and collect reward. And today we'll talk about


[User] Please transcribe this text

[Assistant] Perhaps the least understood, but the most exciting neural network out there, flavor of neural network, is recurrent neural networks.


[User] Please transcribe this text

[Assistant] There's a website, I don't know if you heard, cars.mit.edu, where you should create an account if you're a registered student. That's one of the requirements. You need to have an account if you want to get credit for this. Uh, you need to submit code for DeepTrafficJS and DeepTeslaJS. And for DeepTraffic,


[User] Please transcribe this text

[Assistant] You have to have a neural network that drives faster than 65 miles an hour. If you need help to achieve that speed, please email us. We can drop, we can give you some hints.


[User] Please transcribe this text

[Assistant] For those of you who are old school SNL fans, there's a deep thoughts section now in, uh, in the profile page where we encourage you to talk about the kinds of things you tried in deep traffic or any of the other deep Tesla or any of


[User] Please transcribe this text

[Assistant] Of the work you've done as part of this class for deep learning. Okay.


[User] Please transcribe this text

[Assistant] We've talked about the, the vanilla neural networks on the left. The vanilla neural network is the one where it's computing, it's approximating a function that maps from one input


[User] Please transcribe this text

[Assistant] Convolutional neural networks can operate on audio. You can give it a chunk of audio, 5 second audio clip. That still counts as one input, because it's fixed size. As long as the size of the input is fixed, that's one chunk of input.


[User] Please transcribe this text

[Assistant] And as long as you have ground truth that maps that chunk of input to some output ground truth.


[User] Please transcribe this text

[Assistant] That's a vanilla neural network. Whether it's a fully connected neural network or a convolutional neural network. Today we'll talk about the amazing, the mysterious recurrent neural networks.


[User] Please transcribe this text

[Assistant] They compute functions from one to many, from many to one, from many to many.


[User] Please transcribe this text

[Assistant] What does that mean? They take his input, sequences, time series, audio,


[User] Please transcribe this text

[Assistant] Video, whenever there's a sequence of data and that temporal dynamics that connects the data is more important than the spatial.


[User] Please transcribe this text

[Assistant] So when there's a lot of information being conveyed in the sequence,


[User] Please transcribe this text

[Assistant] In the, in the temporal change of whatever that type of data is, that's when you want to use recurrent neural networks. Like speech, natural language.


[User] Please transcribe this text

[Assistant] Audio. And the power of this is that for many of them, uh, for a recurrent neural network where they really shine is when the size of the input


[User] Please transcribe this text

[Assistant] So you don't have a fixed chunk of data that you're putting in, it's variable input. And the same goes for the output. So you can have, you can give it a sequence of speech, several seconds of speech, and then the output is,


[User] Please transcribe this text

[Assistant] A single label of whether the speaker is male or female. That's many to one. You can also do many to many translation. You can have natural language put into the network.


[User] Please transcribe this text

[Assistant] In, uh, Spanish and the output is in English. Machine translation. That's many to many.


[User] Please transcribe this text

[Assistant] And that many to many doesn't have to be mapped directly into same size sequences.


[User] Please transcribe this text

[Assistant] The question was, are there any models where there's feedback from output and input? And that's exactly what recurrent neural networks are. It produces output and it copies that output and loops it back in.


[User] Please transcribe this text

[Assistant] That's exactly, that's exactly the, almost the definition of our current neural network. There's a loop in there that produces the output and also takes that output as input once again.


[User] Please transcribe this text

[Assistant] And so there's also many to many where the sequences don't align. Like machine translation, the size of the output sequence might be totally different than the input sequence. We'll look at a lot of cool applications, like you can, uh,


[User] Please transcribe this text

[Assistant] Start a song, so learn on the audio of a particular song and have the recurrent neural network continue that song after a certain period of time. You can learn to generate sequences of audio of natural language of video. Okay.


[User] Please transcribe this text

[Assistant] I know I promised not many equations, but this is so beautifully simple that we have to cover back propagation. It's also the thing


[User] Please transcribe this text

[Assistant] That if you're a little bit lazy and you go on the internet and start using the basic tutorials for tensorflow, you ignore how backpropagation work at your peril.


[User] Please transcribe this text

[Assistant] You kind of assume it just works. I give it some inputs, some outputs, and it's like Lego pieces. I can assemble them like you might have done with deep traffic. A bunch of layers, put them together, and then just press train.


[User] Please transcribe this text

[Assistant] And back propagation is the mechanism that neural networks currently, the best mechanism we know of that is used for training. So we need to understand the,


[User] Please transcribe this text

[Assistant] Simple power of back propagation, but also the dangers.


[User] Please transcribe this text

[Assistant] Say I'll put the top of the slide. There's an input to the network as an image. There's a bunch of neurons all with differentiable smooth activation functions on each neuron and then,


[User] Please transcribe this text

[Assistant] As you pass through those activation functions.


[User] Please transcribe this text

[Assistant] Taking the input, pass it through, uh, this net.


[User] Please transcribe this text

[Assistant] of differentiable compute nodes you produce an output and that output


[User] Please transcribe this text

[Assistant] You also have a ground truth. The correct, the truth.


[User] Please transcribe this text

[Assistant] That you hope, that you expect the network to produce. And then you can look at the difference between what the network actually produced and what you hoped it would produce, and that's an error. And then you back or propagate that error, punishing or rewarding the...


[User] Please transcribe this text

[Assistant] The, the weights, the parameters of the network that resulted to the net output.


[User] Please transcribe this text

[Assistant] The function does two things. It adds x and y, and then it multiplies that sum by z. And then we can, uh, formulate that as a circuit. Circuit of gates.


[User] Please transcribe this text

[Assistant] Where there's a plus gate and a multiplication gate.


[User] Please transcribe this text

[Assistant] And, uh, let's take some inputs shown in blue. Let's say x is negative two, y is five, z is negative four.


[User] Please transcribe this text

[Assistant] And let's do a forward pass through this circuit to produce the output.


[User] Please transcribe this text

[Assistant] So, negative two plus five equals three. Q is that intermediate value. Three.


[User] Please transcribe this text

[Assistant] And so important to understand that I just want to take my time through this, because everything else about neural networks just builds on these concepts. Right? Okay, so, uh, the add gate produces Q, in this case it's 3, and then 3 times negative 4 is 12. That's the output. The output of the circuit of this,


[User] Please transcribe this text

[Assistant] Network, if you think of it as such, is negative 12.


[User] Please transcribe this text

[Assistant] And so the forward pass is shown in blue, the backward pass will be shown in red in a second here. So what we want to do is, what would make us happy, what would make F happy is for the output to be as high as possible. Negative 12 is so-so, we could do better. So how do we teach it? How do we adjust X, Y, and Z such that it produces a higher F?


[User] Please transcribe this text

[Assistant] Makes us happier. Okay, let's, let's start backward. The backward pass.


[User] Please transcribe this text

[Assistant] We make the gradient on the output 1. Meaning we want this to increase. We want f to increase. That's how we encode our happiness. Is, uh, we want it to go up by 1.


[User] Please transcribe this text

[Assistant] That function, that, that fact that we want the, I have to go up by one. We have to look at the gradient on each one of the gates. Now what's a gradient?


[User] Please transcribe this text

[Assistant] The partial derivative of the output of a gate with respect to its inputs. You know, if you don't know what that means, it's just the...


[User] Please transcribe this text

[Assistant] When I change the inputs a little bit, what is the slope of that change? If I increase x for the first function of addition.


[User] Please transcribe this text

[Assistant] f of x, y equals x plus y. If I increase x by a little bit, what happens to f? If I increase y by a little bit, what happens to f? So taking a partial derivative of those with respect to x and y, you just get a slope of 1. So when you increase x, f increases linearly. Same with y. Multiplication is a little trickier.


[User] Please transcribe this text

[Assistant] When you increase x, f increases by y. So the partial derivative of f.


[User] Please transcribe this text

[Assistant] With respect to x is y, partial derivative of f with respect to y is x.


[User] Please transcribe this text

[Assistant] So if you think about it, what happens is the gradients, when you change x,


[User] Please transcribe this text

[Assistant] The gradient of change doesn't care about X. It cares about Y. So it's flipped.


[User] Please transcribe this text

[Assistant] So we can back propagate that one, the indication of what makes us happy, backwards.


[User] Please transcribe this text

[Assistant] So the, uh, the partial derivative of f with respect to q, that intermediate value.


[User] Please transcribe this text

[Assistant] The, that gradient will be negative four. It will take the value of z, as I said, it's a multiplication gate. It will take the value of z.


[User] Please transcribe this text

[Assistant] And the same for the, uh, partial derivative of f with respect to z, it'll assign that to q, the value of the forward pass from the q. So there's a 3 and a negative 4 on the forward pass, in blue, and then that's flipped, negative 4 and 3, on the backward pass. That's the gradient. And then we continue in the same exact process, but wait.


[User] Please transcribe this text

[Assistant] Uh, so what makes all of this work?


[User] Please transcribe this text

[Assistant] Is, is the chain rule. It's magical.


[User] Please transcribe this text

[Assistant] So what it allows us to do is to compute the gradient, the gradient of an f with respect to the inputs, x, y, z. We don't need to construct.


[User] Please transcribe this text

[Assistant] Uh, analytically. We can do it step by step, back propagating the gradients. We can multiply the gradients together. As opposed to doing partial derivative of f with respect to x. We have just the intermediate, the local gradient of f with respect to q and of q with respect to x and multiply them together.


[User] Please transcribe this text

[Assistant] The gradient of the, that giant function x plus y equals, uh, times z. In this case, it's not that giant, but it gets pretty giant when you're on networks. We just go step by step. We'll look at the first function. Simple addition.


[User] Please transcribe this text

[Assistant] The partial derivative of f, uh, with respect to x and y is computed by multiplying the gradient on the output, negative four,


[User] Please transcribe this text

[Assistant] Times the gradient on the inputs, which as we talked about when it's, when the operation is addition, that's just one. So it's negative four times one. That means...


[User] Please transcribe this text

[Assistant] What does that mean? Let's interpret those numbers.


[User] Please transcribe this text

[Assistant] You now have gradients on x, y, and z.


[User] Please transcribe this text

[Assistant] The gradient of, uh, the partial derivative of f with respect to x, y, z. That means the, so, for x and y it's negative four, for z it's three. That means in order to make f happy, we have to decrease the, uh, the inputs that have a negative gradient,


[User] Please transcribe this text

[Assistant] Hopefully I don't say the word beautiful too many times in this presentation, but this is very simple, beautifully simple, uh, because this gradient is a local worker.


[User] Please transcribe this text

[Assistant] It comp- it's- it propagates for you. It has no no- it has no knowledge of the broader, uh, happiness of F.


[User] Please transcribe this text

[Assistant] It just propagates the, it computes the gradient between the output and the input. And you can propagate this gradient, uh, based on,


[User] Please transcribe this text

[Assistant] Uh, in this case f, uh, a gradient of 1, but also just the error.


[User] Please transcribe this text

[Assistant] Instead of one, we could have on the output the error is the measure of happiness and then we could propagate that error backwards. These gates are important because you can break down almost every operation we could think of that we work with in neural networks into one of several gates like this and


[User] Please transcribe this text

[Assistant] The most popular are three, which is addition, multiplication, and the max operation. So for addition, what, uh, you do is, you ignore the, so, okay, the process is you take a forward pass through the network, so we have a value on every single gate. And then you take a backward pass, and through the backward pass you compute those gradients. For an add gate,


[User] Please transcribe this text

[Assistant] You equally distribute the gradients on the output to the input. So when the gradient on the output is negative four, you equally distribute it to negative, to negative four.


[User] Please transcribe this text

[Assistant] And you ignore the forward pass value. So that 3 is ignored when you back propagate it.


[User] Please transcribe this text

[Assistant] On the multiplication gate, on the multiply gate, it's trickier.


[User] Please transcribe this text

[Assistant] And multiplied by the value of the gradient in the output. If it's confusing, go through the slides slowly. You know, it'll make a lot more sense. Hopefully.


[User] Please transcribe this text

[Assistant] One more gate, there's the match gate, which takes the inputs and produces the output, the value that is larger.


[User] Please transcribe this text

[Assistant] And when computing the gradient of the max gate, it distributes the gradient, similar to the add gate, but to only one.


[User] Please transcribe this text

[Assistant] To, uh, to only one of the inputs. The largest one. So it, unlike the add gate pays attention to the input, the input values on the forward pass. Alright. Lots of numbers, but the whole point here is it's like, it's really simple.


[User] Please transcribe this text

[Assistant] What a neural network is just a simple collection of these gates. And there's, you take a forward pass.


[User] Please transcribe this text

[Assistant] You calculate some kind of function on the end, the gradient at the very end, and you propagate that back. So usually for neural networks that's an error function. A loss function, objective function, uh, cost function, all the same, all the same word.


[User] Please transcribe this text

[Assistant] So that's the, that's the sigmoid function there. When you have three weights, w0, w1, w2, and x, two inputs, x0, x1, that's gonna be the sigmoid function. That's how you compute the output.


[User] Please transcribe this text

[Assistant] But then you could decompose that neuron, you could separate it all into just a set of gates like this. Addition, multiplication, there's a exponential in there and a division.


[User] Please transcribe this text

[Assistant] They're all very similar. And you repeat the exact same process if the, the, uh,


[User] Please transcribe this text

[Assistant] There's five inputs, there's three weights, and two inputs, X0, X1.


[User] Please transcribe this text

[Assistant] In this case again, you want it to increase so that the gradient on the output is one, and you back propagate that gradient of one to the inputs. Now with neural networks,


[User] Please transcribe this text

[Assistant] There's a bunch of parameters that you're trying to through this process modify, and you don't get to modify the inputs.


[User] Please transcribe this text

[Assistant] You get to modify the weights along the way and the biases. The inputs are fixed, the outputs are fixed, the outputs that you hope, uh, uh, the network will produce. What you're modifying is the weights. So I get to try to adjust those weights in such that, uh, in the direction of the gradient.


[User] Please transcribe this text

[Assistant] That's the task of backpropagation. The main way that neural networks learn, is we update the weights and the biases to decrease the loss function. The lower the loss function the better. So in this case you have,


[User] Please transcribe this text

[Assistant] Uh, three weights on each of the inputs. There's a bias on the node, B, and it produces an output, A. And that little symbol is indicating a sigmoid function.


[User] Please transcribe this text

[Assistant] Where Y is the ground truth, the output that you want to be, uh, the network to produce.


[User] Please transcribe this text

[Assistant] And that loss function is back propagated in exactly the same way that we described before.


[User] Please transcribe this text

[Assistant] Alright, so the subtasks that are involved in this update of weights and biases is that the forward pass computes the network output at every neuron, and then, and then finally the output layer, computes the error, the difference between A and B, and then backward propagates the gradients.


[User] Please transcribe this text

[Assistant] Instead of one on the output, it'll be the error on the output, and you back propagate it.


[User] Please transcribe this text

[Assistant] And then once you know the gradient, you adjust the weights and the biases in the direction of the gradient.


[User] Please transcribe this text

[Assistant] Or actually the opposite of the direction of the gradient, because you want the loss to decrease. And the amount by which you make that adjustment is called the learning rate.


[User] Please transcribe this text

[Assistant] The learning rate could be the same across the entire network, or it could be individual to every weight.


[User] Please transcribe this text

[Assistant] of adjusting the weights and biases is just optimization.


[User] Please transcribe this text

[Assistant] It's, learning is an optimization problem. You have an objective function and you're trying to minimize it. And your variables are the parameters, the weights and biases.


[User] Please transcribe this text

[Assistant] And uh, you know, neural networks just happen to have tens, hundreds of thousands, millions of those parameters. So the space, the function that you're trying to minimize is highly non-linear. But it boils down to something like this. You have two weights. Here, two plots.


[User] Please transcribe this text

[Assistant] Or actually one weight, sorry. One weight. And then, uh, as you adjust it, the cost...


[User] Please transcribe this text

[Assistant] You adjust in such a way that minimizes the output cost.


[User] Please transcribe this text

[Assistant] And there's a bunch of optimization methods for doing this. You can, uh,


[User] Please transcribe this text

[Assistant] This is a convex function, so you can find the minimum.


[User] Please transcribe this text

[Assistant] The local minimum, if you know about these kind of terminologies, the local minimum is the same as the global minimum.


[User] Please transcribe this text

[Assistant] So there's not, it's not a weirdly hilly terrain where you can get stuck in, in, uh, So your goal is to get to the bottom of this thing, and if it's really complex terrain, it'll be hard to get to the bottom of it.


[User] Please transcribe this text

[Assistant] So there is a lot of, uh, different, this general approach is gradient descent.


[User] Please transcribe this text

[Assistant] And there's a lot of different ways to do gradient descent. Some adding, uh, in various ways of adding randomness into the process so you don't get stuck into the weird, uh, crevices of, uh, the terrain. Alright. But it's messy. You have to be really careful. This is the part you have to be aware of.


[User] Please transcribe this text

[Assistant] When, uh, you're designing a network for deep traffic and nothing is happening, this might be what's happening. Vanishing gradients or exploding gradients.


[User] Please transcribe this text

[Assistant] The, when the partial derivative is small, so if you take it as a sigmoid function, you know, the most popular


[User] Please transcribe this text

[Assistant] For a while, uh, activation function, the derivative is zero at the tails. So when the input to the sigmoid function is really high or really low,


[User] Please transcribe this text

[Assistant] That derivative is going to be zero. So the gradient that you compute


[User] Please transcribe this text

[Assistant] You know, gradient tells you how much I want to adjust the weights. The gradient might be zero.


[User] Please transcribe this text

[Assistant] And so you backpropagate that zero, a very low number, and it gets less and less as you backpropagate. And so your, the result is that you don't


[User] Please transcribe this text

[Assistant] You think that you don't need to adjust the weights at all. And when a large fraction of the network thinks that weights don't need to be adjusted, then they don't adjust the weights and you're not doing any learning. So the learning is slow.


[User] Please transcribe this text

[Assistant] There's different types of functions. Here's a piecewise.


[User] Please transcribe this text

[Assistant] The, the Rayleigh function, which is the most popular activation function. But again, it suffers.


[User] Please transcribe this text

[Assistant] If, if it's initialized, if the neurons are initialized poorly,


[User] Please transcribe this text

[Assistant] It might not, this, this function might not fire, it might be zero gradient for the entire data set. Nothing that you, nothing that you,


[User] Please transcribe this text

[Assistant] Produces input. You know, you run all your thousands of images of cats. And none of them fire at all. So that's the danger here.


[User] Please transcribe this text

[Assistant] So you have to pick these, both the, the optimization


[User] Please transcribe this text

[Assistant] You can't just play it plug and play like, uh, like they're Legos. You have to be aware of the function. Uh, SGD, stochastic gradient descent.


[User] Please transcribe this text

[Assistant] Is the, is the, that's the vanilla optimization algorithm for gradient descent.


[User] Please transcribe this text

[Assistant] Uh, for optimizing loss function over the gradients.


[User] Please transcribe this text

[Assistant] And so what's visualized here is, again, if you're in, if you've done any numerical optimization, nonlinear optimization, there's the famous saddle point that's tricky for these algorithms to deal with.


[User] Please transcribe this text

[Assistant] What, what happens is it's easy for them to oscillate, get stuck in that saddle and oscillate back and forth.


[User] Please transcribe this text

[Assistant] Uh, as opposed to what they want to do, which is go down into, you know, you get so, so happy that you found this like,


[User] Please transcribe this text

[Assistant] You forget that there's a much lower point. And so you get stuck with the gradient, the momentum of the gradient keeps rocking you back and forth without you going.


[User] Please transcribe this text

[Assistant] To a much greater global minimum. And there's a bunch of clever ways of solving that. The atom optimizer is one of those.


[User] Please transcribe this text

[Assistant] But in this case, as long as the gradients don't vanish,


[User] Please transcribe this text

[Assistant] SGD, the stochastic gradient descent, one of these algorithms will get you there. They might take a little while, but they'll get you there. And that's the main, yes, question. Uh, the question was, uh,


[User] Please transcribe this text

[Assistant] You're dealing with a function that's not non-convex and how do we ensure anything about, uh, it converging to anything that's reasonably good. The local optimum converges to, and the answer is you can't. This isn't only a nonlinear function. It's a highly nonlinear function. The power and the beauty of neural networks is that it can represent these like


[User] Please transcribe this text

[Assistant] It's, it's incredible, right? And you can learn those functions from data. But,


[User] Please transcribe this text

[Assistant] The, the reason, you know, people refer to neural networks training as art...


[User] Please transcribe this text

[Assistant] It's, it's, you're trying to play with parameters that don't get stuck in these local optima for stupid reasons and for clever reasons. Here's a question.


[User] Please transcribe this text

[Assistant] So the question, yeah, so continue on the same thread.


[User] Please transcribe this text

[Assistant] So the thing is, we're dealing with functions where we don't know what the global optimal is. That's sort of the crux of it.


[User] Please transcribe this text

[Assistant] Everything we talk about, interpreting text, interpreting video.


[User] Please transcribe this text

[Assistant] You know, even driving, like what is the optimal for driving? Never crashing? You know.


[User] Please transcribe this text

[Assistant] It sounds easier to say that, but you actually have to formulate the world under which you define all of those things and that becomes really non-linear objective function for which you don't know what the optimal is. It's just...


[User] Please transcribe this text

[Assistant] That's why you just keep trying and get impressed every time it gets better. It's essentially the process.


[User] Please transcribe this text

[Assistant] And you know, and, and you can also compare, you know, you can compare to human level performance. So for ImageNet...


[User] Please transcribe this text

[Assistant] We can tell the difference between cats and dogs in the top 5 categories in 90, uh,


[User] Please transcribe this text

[Assistant] Shoot, uhm, 96% of the time, whatever, accuracy. And then you get impressed when, uh, a machine can do better than that. But you don't know what the best is.


[User] Please transcribe this text

[Assistant] These videos could be watched for hours. I won't play it until I explain the slide.


[User] Please transcribe this text

[Assistant] The question was, how do you practically know when you've hit the vanishing gradient problem? So the vanishing gradient could be, so that, that,


[User] Please transcribe this text

[Assistant] Happens when the activation is exploding. It's like really high values and really low values. The really high values is easy because they're like your network is just going crazy. Uh, is producing very large values and you can fix a lot of those things by just, uh, capping the activations. The, the values being really low,


[User] Please transcribe this text

[Assistant] Resulting in a vanishing gradient are really hard to detect.


[User] Please transcribe this text

[Assistant] This is, I mean, uhm, there's a lot of research in trying to figure out how to...


[User] Please transcribe this text

[Assistant] How to detect these things, but it's, if you're not careful, it's oftentimes you can, you can find that.


[User] Please transcribe this text

[Assistant] And this isn't hard to do, uh, where like 40-50% of the network of the neurons are, uh, are dead. Where they're called, like, for Rayleigh-U, they're dead Rayleigh-U nodes. They're not producing, they're not firing at all. How do you detect that?


[User] Please transcribe this text

[Assistant] That's part of learning. So if they never fire, you can detect that by running it through the entire training set. I mean, there's a lot of tricks, but that's the problem is...


[User] Please transcribe this text

[Assistant] You try to learn and then the, you look at the loss function and it's not,


[User] Please transcribe this text

[Assistant] Converging to anything reasonable. It's either going all over the place or just converging very slowly. And that's an indication that something is wrong.


[User] Please transcribe this text

[Assistant] That something could be the loss function is bad, that something could be that you've already found the optimal, or that something could be the vanishing gradient. And again, that's why it's an art.


[User] Please transcribe this text

[Assistant] But certainly some, at least, at least some fraction of the neurons need to be firing. Otherwise the initialization is really poorly done. Okay, so to reflect on the simplicity of backpropagation. And the power of it. So this is,


[User] Please transcribe this text

[Assistant] This kind of step of backpropagating the loss function to the gradients locally.


[User] Please transcribe this text

[Assistant] Is the way neural networks learn. We don't have, uh, it's really the only way that we've effectively been able to, to train


[User] Please transcribe this text

[Assistant] A neural network to learn a function. To adjusting the weights and biases. The huge number of weights and biases, the parameters.


[User] Please transcribe this text

[Assistant] Uh, is just through this optimization, is backpropagating the error where you have the supervised ground truth.


[User] Please transcribe this text

[Assistant] So the question is whether this process is just like fitting


[User] Please transcribe this text

[Assistant] Adjusting the parameters of a highly nonlinear function to minimize a single objective is the way you achieve


[User] Please transcribe this text

[Assistant] Intelligence. Human level intelligence. And that's something to think about.


[User] Please transcribe this text

[Assistant] You have to think about the, for driving purposes, what is the limitation of this approach?


[User] Please transcribe this text

[Assistant] So what's not happening? The neural network design, the architecture, is not being adjusted. You're not evolving the, any of the edges, the layers, nothing is being evolved.


[User] Please transcribe this text

[Assistant] And so there are other, uh, optimization approaches that I think are more...


[User] Please transcribe this text

[Assistant] Interesting and inspiring than effective. So for example, this is, uh,


[User] Please transcribe this text

[Assistant] Uh, using soft cubes to, uh, so this is falling out of the field of evolutionary robotics, where you evolve the dynamics of a robot using genetic algorithms. And that's...


[User] Please transcribe this text

[Assistant] So what, these robots are being taught to, in simulation obviously, to uh, to walk and to swim. That one is swimming.


[User] Please transcribe this text

[Assistant] But, you could, the, the nice thing here is the dynamics, that highly nonlinear space as well, that controls the dynamics of this, uh, weird shaped robot.


[User] Please transcribe this text

[Assistant] With a lot of degrees of freedom is the same kind of thing as the neural network. And in fact, people have applied genetic algorithms and, uh, uh, and colony optimization, all kinds of sort of nature inspired algorithms for optimizing the weights and the biases, but they don't seem to currently work that well, but it's kind of,


[User] Please transcribe this text

[Assistant] It's a cool idea to be using nature type evolutionary algorithms to evolve something that's already nature inspired, which is neural networks. But something to think about is, you know,


[User] Please transcribe this text

[Assistant] The backpropagation, while really simple, is, is kinda dumb. And the question is whether general intelligence reasoning can be achieved with this process. Alright, recurrent neural networks.


[User] Please transcribe this text

[Assistant] So, uh, on the left there, there's an input X with weights on the input U.


[User] Please transcribe this text

[Assistant] On the edge connecting the hidden states to each other, and then more weights V on the output. Oh. It's a really simple network. There's inputs, there is hidden states, the memory of this network, and there's outputs.


[User] Please transcribe this text

[Assistant] But the re- the fact that there is this loop where the hidden states are connected to each other means that as opposed to producing a single input,


[User] Please transcribe this text

[Assistant] The network takes arbitrary number of inputs. It just keeps taking x one at a time and produces a sequence of x's through time. And so,


[User] Please transcribe this text

[Assistant] Depending on the duration of the sequences you're interested in, you can think of this network in its unrolled state.


[User] Please transcribe this text

[Assistant] So you can unroll this neural network, where the inputs are on the bottom, xt-1, xt, xt-1.


[User] Please transcribe this text

[Assistant] The same with the outputs, 0, oh sorry, OT minus 1, OT, OT plus 1. And it becomes like a regular neural network.


[User] Please transcribe this text

[Assistant] Unrolled some arbitrary number of times.


[User] Please transcribe this text

[Assistant] It's similar to CNNs, convolutional neural networks, in that it's, uh, just like convolutional neural networks make certain spatial, uh, consistency assumptions, the, uh, recurrent neural networks assume temporal consistency amongst the parameters. So it shares the parameters.


[User] Please transcribe this text

[Assistant] That w, that u, that v is the same for every single time step. So you're learning.


[User] Please transcribe this text

[Assistant] The same parameter, no matter the duration of the sequence.


[User] Please transcribe this text

[Assistant] And that allows you to look at arbitrary long sequences, uh, without having an explosion of parameters.


[User] Please transcribe this text

[Assistant] And this process is the same exact process that's repeated based on the different variants that we talked about before in terms of inputs and outputs. One to many, many to one, many to many.


[User] Please transcribe this text

[Assistant] And the backpropagation process is exactly the same as for regular neural networks. It has a fancy name of backpropagation through time, BPTT.


[User] Please transcribe this text

[Assistant] Uh, unrolled, uh, recurrent neural network, where the errors are on the, computed on the outputs, the gradients are computed, uh, propagate, back propagated, and computed on the inputs.


[User] Please transcribe this text

[Assistant] Again, suffering from the same exact problem of vanishing gradients. Now the problem is that the depth of these networks can be arbitrarily long, right?


[User] Please transcribe this text

[Assistant] If, if at any point the gradient hits a low number, zero,


[User] Please transcribe this text

[Assistant] It becomes, that neuron becomes saturated. That gradient, well it's called saturated, that gradient gets


[User] Please transcribe this text

[Assistant] So it's easy to run into a problem where you're really ignoring majority of the sequence.


[User] Please transcribe this text

[Assistant] This is just another Python way to, uh, pseudocode way to look at it.


[User] Please transcribe this text

[Assistant] They have a negative value that results in a gradient that goes to zero.


[User] Please transcribe this text

[Assistant] That propagates through the rest. So that's the pseudocode for back propagation, the backward pass through the RNN. That WHH propagates back.


[User] Please transcribe this text

[Assistant] And so you get these things with exploding and vanishing gradients where


[User] Please transcribe this text

[Assistant] This is for example a error surface for a single hidden unit RNN. So these visualizing the gradient. The value of the weight, the value of the bias and the error. So the error could be really flat or could explode and both are going to lead


[User] Please transcribe this text

[Assistant] To you not making the gre-uh, either making steps that are too gradual or too big.


[User] Please transcribe this text

[Assistant] It's the geometric interpretation. Okay, what other variants that we'll look at a little bit?


[User] Please transcribe this text

[Assistant] Are there for RNNs. It doesn't have to be only one way. It can be bi-directional. So there could be, uh, edges going forward and edges going back.


[User] Please transcribe this text

[Assistant] What that's needed for is things like, uhm, filling in missing, whatever the data is, filling in missing elements of that data, whether that's, uh, images or words or audio.


[User] Please transcribe this text

[Assistant] And generally, as always is the case with neural networks, the deeper you go, the better. So, uh, this is that deep referring to the number of layers in a single temporal instance.


[User] Please transcribe this text

[Assistant] So on the right of the slide is, is, uh, we're stacking in the, uh, not in the temporal domain.


[User] Please transcribe this text

[Assistant] Each of those layers has its own set of weights and its own sets of biases.


[User] Please transcribe this text

[Assistant] These things are awesome, but they need a lot of data when you, uh, when you, when you make, uh, when you add extra layers in this way.


[User] Please transcribe this text

[Assistant] Okay, so the problem is while recurrent neural networks, in theory, are supposed to be able to learn any kind of sequence.


[User] Please transcribe this text

[Assistant] The reality is, they're not really good at remembering what happened a while ago. The long-term dependency. So here's, uh, a silly example. Uh, let's think of, uh, a story about Bob. Bob is eating an apple.


[User] Please transcribe this text

[Assistant] So the apple part is generated by the recurrent neural network. Right?


[User] Please transcribe this text

[Assistant] Recurrent neural networks can learn to generate apple because they've seen a lot of sentences with Bob and eating and they can generate the word apple.


[User] Please transcribe this text

[Assistant] You have to maintain the state that we're talking about Bob, and we're talking about apples, through several discreet semantic sentences.


[User] Please transcribe this text

[Assistant] And that kind of long-term memory is not


[User] Please transcribe this text

[Assistant] Recurrent neural networks, these days,


[User] Please transcribe this text

[Assistant] They're talking about LSTMs. Long Shirt Short Term Memory Networks.


[User] Please transcribe this text

[Assistant] So all the impressive results on time series and audio and video, all of that, that requires LSTMs. And so again, vanilla RNNs up on top of the slide.


[User] Please transcribe this text

[Assistant] Here we'll use 10H as the activation function.


[User] Please transcribe this text

[Assistant] It's, uh, it's just another popular sigmoid type activation function.


[User] Please transcribe this text

[Assistant] LSTM's are more complicated. Or they look more complicated, but.


[User] Please transcribe this text

[Assistant] In some ways they're more intuitive for us to understand. There's a bunch of gates in each cell.


[User] Please transcribe this text

[Assistant] We'll go through those. In yellow are different neural network layers. Sigma and 10H.


[User] Please transcribe this text

[Assistant] Are just, are different types of activation functions. 10h is an activation function that, that squishes the input into the range of negative one to one. Uh, uh, sigmoid function, uh, squishes it between zero and one. And that serves different purposes.


[User] Please transcribe this text

[Assistant] There is some pointwise operations, addition, uh, multiplication, and there is connections.


[User] Please transcribe this text

[Assistant] So data being passed from layer to layer, shown by the arrows.


[User] Please transcribe this text

[Assistant] There's concatenation and there's a copy operation on the output.


[User] Please transcribe this text

[Assistant] The output of each cell is copied to the next cell and to the output. Okay, let me try to make it clarify


[User] Please transcribe this text

[Assistant] So there's this conveyor belt going through inside each individual cell. And they all have, there's really three steps in the conveyor belt.


[User] Please transcribe this text

[Assistant] What to forget and what to ignore. So it's, it's responsible for taking in the input, the new input, xt, taking in the, the, the state of the previous,


[User] Please transcribe this text

[Assistant] The output of the previous, uh, cell, previous time step.


[User] Please transcribe this text

[Assistant] And deciding, do I wanna keep that in my memory or not? And do I want to integrate the new input into my memory or not?


[User] Please transcribe this text

[Assistant] So this allows you to be selective about the information which you learn.


[User] Please transcribe this text

[Assistant] So for example, the sentence, Bob and Alice are having lunch. Bob likes apples. Alice likes oranges. She's eating an orange. So the way...


[User] Please transcribe this text

[Assistant] So Bob and Alice are having lunch. Bob likes apples. Right now, if you say you had a hidden state, keeping track of the gender of the person we're talking about. So,


[User] Please transcribe this text

[Assistant] You might say that there's both genders in the first sentence, there's male in the second sentence, female in the third sentence.


[User] Please transcribe this text

[Assistant] And that way, when you have to generate a sentence about who's eating what, you'll know, you'll keep the gender information.


[User] Please transcribe this text

[Assistant] In order to make an accurate generation of text corresponding to, uhm, to the proper person.


[User] Please transcribe this text

[Assistant] So you have to forget certain things, like forget that Bob existed at that moment.


[User] Please transcribe this text

[Assistant] And you have to forget Bob likes apples, but you have to remember that Alice likes oranges.


[User] Please transcribe this text

[Assistant] So you have to selectively remember and forget certain things. That's LSTM in a nutshell.


[User] Please transcribe this text

[Assistant] So you decide what to forget, decide what to remember, and decide what to output at that cell.


[User] Please transcribe this text

[Assistant] Alright, so zoom in a little bit. This is pretty cool.


[User] Please transcribe this text

[Assistant] There is a state running through the cell. This conveyor belt.


[User] Please transcribe this text

[Assistant] That's the state that you're keeping track of, and that's running through the cell. And then there is three sigmoid layers, outputting,


[User] Please transcribe this text

[Assistant] And so first sigmoid function is we decide what to forget and what to ignore. That's the first one. You take the inputs from the previous time step.


[User] Please transcribe this text

[Assistant] The input to the network from the current time step and decide do I want to forget? Do I want to ignore those? Then.


[User] Please transcribe this text

[Assistant] We decide which part of the state to update. What part of our memory do we update with this information? And what values to insert in that update?


[User] Please transcribe this text

[Assistant] So that's where you have the sigmoid function is you just multiply it.


[User] Please transcribe this text

[Assistant] And finally, we produce an output from the cell.


[User] Please transcribe this text

[Assistant] So if it's translation, it's producing an output in the English language where the input was in the Spanish language. And then that same output is copied to the next cell.


[User] Please transcribe this text

[Assistant] Okay, so what can we get done with this kind of approach?


[User] Please transcribe this text

[Assistant] We can look at machine translation. I guess what I'm trying to, uh, uh, this question.


[User] Please transcribe this text

[Assistant] Multiply by the way, so it's the outputs of the, uh, the sigmoid or the 10H activations. So the...


[User] Please transcribe this text

[Assistant] There's a bunch of neurons and they're firing a number between negative one and one or between zero and one. And that's, that holds the state. It's just calling it state is sort of simplifying. But the point is that there's a bunch of numbers being constantly modified by the weights and the biases. And those numbers hold the state.


[User] Please transcribe this text

[Assistant] And the modification of those numbers is controlled by the weights.


[User] Please transcribe this text

[Assistant] And then once all that is done, the resulting output of the recurrent neural network is compared to the desired output and the air is then back propagated through the weights.


[User] Please transcribe this text

[Assistant] So machine translation is one popular application.


[User] Please transcribe this text

[Assistant] And all of it is, uh, uh, all of it is the same.


[User] Please transcribe this text

[Assistant] All of these networks that I'll talk about, they're really similar constructs. You have some inputs, whatever language that is again, uh, German maybe, I think everything is German.


[User] Please transcribe this text

[Assistant] And the output, so the inputs are in one language, a set of characters that compose a word in one language, there's a state being propagated, and once that sentence is over,


[User] Please transcribe this text

[Assistant] You start, as opposed to collecting inputs, you start producing outputs. And you can output in the English language.


[User] Please transcribe this text

[Assistant] That's, there, there, there's a ton of great work on machine translation. It's what Google is mostly using for their translator. Same thing, uh, I showed this previously.


[User] Please transcribe this text

[Assistant] But now you know how it works. Same exact thing. LSTMs. Generating handwritten characters. So handwriting in arbitrary styles. So controlling the drawing.


[User] Please transcribe this text

[Assistant] Where the input is text and the output is handwriting. And it's again the same kind of network with some depth here.


[User] Please transcribe this text

[Assistant] Character level text generation. This is the...


[User] Please transcribe this text

[Assistant] The thing that told us about life, the meaning of life, literary recognition and the tradition of ancient human reproduction. That's again the same process. Input one character at a time.


[User] Please transcribe this text

[Assistant] Where you see there's an encoding of the characters on the input layer. There's a hidden state.


[User] Please transcribe this text

[Assistant] Hidden layer that's keeping track of those activations, the outputs of the, of the activation functions, and every single,


[User] Please transcribe this text

[Assistant] You want to ignore the output until the sentence is, the input sentence is over.


[User] Please transcribe this text

[Assistant] And then you start listening to the output. But the point is it just keeps generating text, whether it's given input or not.


[User] Please transcribe this text

[Assistant] So you producing input is just adding, uh, steering the recurrent neural network. You can answer questions.


[User] Please transcribe this text

[Assistant] So the input it hid there, so you could almost arbitrarily stack things together. So you take an image as an input, bottom left there. Put it into convolutional neural network.


[User] Please transcribe this text

[Assistant] There's something called word embeddings. It's to broaden the representative meaning of the words. So how many books is the question? So you want to take the word embeddings and the image and produce your best estimate of the answer. So for a question of what color is the cat?


[User] Please transcribe this text

[Assistant] It could be gray or black. It's the different LSTM flavors producing that answer. Same with counting chairs. You can give an image of a chair.


[User] Please transcribe this text

[Assistant] And ask the question how many chairs are there, and it can produce an answer of three.


[User] Please transcribe this text

[Assistant] So this is, so I should say that this is really hard, right? And it's an arbitrary question asked of an arbitrary image. So you're both interpreting, you're doing natural language processing and you're doing computer vision all in one network.


[User] Please transcribe this text

[Assistant] You can, uh, detect the different objects in the scene, generate those words.


[User] Please transcribe this text

[Assistant] Stitch them together in syntactically correct sentences and re-rank the sentences. All of those are LSTMs, the second and the third step. The first is computer vision detecting the objects. Segmenting the image and detecting the objects. In that way you can generate a caption that says a man is sitting in a chair with a dog in his lap.


[User] Please transcribe this text

[Assistant] Again, LSTMs for video. Caption generation for video.


[User] Please transcribe this text

[Assistant] The input at every frame is an image that goes into the LSTM. The input is an image.


[User] Please transcribe this text

[Assistant] Into, uh, a representation inside the network. And then you start generating words about that video.


[User] Please transcribe this text

[Assistant] Take in the video, say a man is taking, talking, whatever.


[User] Please transcribe this text

[Assistant] And because the input and the output is arbitrary, there also has to be indicators of the beginnings and uh, uh, the ends of a sentence. So in this case, end of sentence. So you want to know when you stop.


[User] Please transcribe this text

[Assistant] In order to generate syntactically correct sentences, you want to also be able to generate a period that indicates the end of a sentence.


[User] Please transcribe this text

[Assistant] That's used to classify what's contained in that image. So here, a CNN being steered by a recurrent neural network.


[User] Please transcribe this text

[Assistant] In order to, uh, convert this, uh, image into the number that's associated with the house number.


[User] Please transcribe this text

[Assistant] As, so the output of the, it's a LSTM, where the output at every time step is visual. And this way you can draw numbers.


[User] Please transcribe this text

[Assistant] NLSTM that takes convolutional, that has convolutional layers for every single frame. Takes images as input.


[User] Please transcribe this text

[Assistant] And produces a spectrogram, audio as output.


[User] Please transcribe this text

[Assistant] The training set is a person hitting an object with a drumstick and your task is to generate, given a silent video, generate...


[User] Please transcribe this text

[Assistant] The sound that a drumstick would make when in contact with that, with that object. Okay, there, eh, medical diagnosis.


[User] Please transcribe this text

[Assistant] That's actually, so I've listed some places where it's been really successful and pretty cool. But it's also beginning to be applied in places where it can actually really help.


[User] Please transcribe this text

[Assistant] Civilization, right? In medical applications. So for medical diagnosis...


[User] Please transcribe this text

[Assistant] Sequence of information in the form of, uh, for example, patient electronic health records. So every time you visit a doctor, there's some test being done and that information is there and you can look at it as a sequence over a period of time. And then given that data, that's the input. The output is the diagnosis.


[User] Please transcribe this text

[Assistant] A medical diagnosis. So in this case we can look at predicting diabetes, scoliosis, asthma, so on.


[User] Please transcribe this text

[Assistant] There's something that all of us wish we could do.


[User] Please transcribe this text

[Assistant] So you can input, for example, well first of all you can input the raw stock data, right, the order books and so on, financial data. But you can also look at news articles from all over the web.


[User] Please transcribe this text

[Assistant] And take those as input, as shown here, on the x-axis is time. So articles from different days. LSTM, once again.


[User] Please transcribe this text

[Assistant] And produce an output of your prediction, binary prediction, whether the stock will go up or down. And,


[User] Please transcribe this text

[Assistant] Nobody's been able to really successfully do this, but there is a bunch of results and trying to perform above random.


[User] Please transcribe this text

[Assistant] Significantly above random, on the prediction of is it going up or down, so you can buy or sell.


[User] Please transcribe this text

[Assistant] And especially when there's a, uh, in the cases when there was crashes, it's easier to predict. So you can predict an encroaching crash.


[User] Please transcribe this text

[Assistant] These are shown in the table, the error rates for different stocks. Automotive stocks.


[User] Please transcribe this text

[Assistant] You can also generate audio. This exact same process you generate language. You can generate audio. Here's trained on a single speaker.


[User] Please transcribe this text

[Assistant] Uh, a few hour epics of them speaking and you just learn that's raw audio of the speaker.


[User] Please transcribe this text

[Assistant] And, it's learning slowly to generate.


[User] Please transcribe this text

[Assistant] Uh, on a compressed spectrogram of the audio, raw audio.


[User] Please transcribe this text

[Assistant] Over just a few epics is producing something that sounds like words. That could do this lecture for me, I wish.


[User] Please transcribe this text

[Assistant] And there's a lot of work in voice recognition and audio recognition. You're mapping.


[User] Please transcribe this text

[Assistant] Let me turn it up. You're mapping any kind of audio to a classification.


[User] Please transcribe this text

[Assistant] And that's the spectrogram on the bottom there being shown. And you could detect whether the road is wet or the road is dry.


[User] Please transcribe this text

[Assistant] Recognizing the gender of the speaker. Recognizing many to many map of the actual words being spoken. Speech recognition.


[User] Please transcribe this text

[Assistant] The NVIDIA approach, the thing that actually powers Deep Tesla, JS, is a simple convolutional neural network.


[User] Please transcribe this text

[Assistant] There's five convolutional layers in their approach, uh, three fully connected layers. You can add as many layers as you want in Deep Tesla.


[User] Please transcribe this text

[Assistant] Parameters, uh, to optimize. And all you're taking is a single image, no temporal information, single image, and producing a steering angle. That's the approach. That's the DeepTesla way.


[User] Please transcribe this text

[Assistant] And a learning of aggression of a steering angle. Now, uh, so one of the,


[User] Please transcribe this text

[Assistant] Prizes for the competition is the Udacity Self-Driving Car Engineer Nano Degree for free.


[User] Please transcribe this text

[Assistant] This thing is awesome. I encourage everyone to check it out. But they did a competition.


[User] Please transcribe this text

[Assistant] Uh, that's very similar to ours.


[User] Please transcribe this text

[Assistant] But they have a very large group of obsessed people. So they, uh, they were very clever. They went beyond just convolutional neural networks of predicting steering. So taking a sequence of images and predicting steering.


[User] Please transcribe this text

[Assistant] What they did is, uh, the winners, at least the first, and I'll talk about the second place winner tomorrow.


[User] Please transcribe this text

[Assistant] On, uh, 3D convolutional neural networks, but the first and the third place winners used RNNs, used LSTMs, recurrent neural networks.


[User] Please transcribe this text

[Assistant] And map a sequence of images to a sequence of steering angles. For anyone, statistically speaking,


[User] Please transcribe this text

[Assistant] Anybody here who's not a computer vision person, most likely what you want to use for whatever application you're interested in is RNNs. It's just the world is full of time series data. Very few of us are working on.


[User] Please transcribe this text

[Assistant] Data that's not time series data. In fact, whenever it's just snapshots, you're really just, uh, reducing the problem to, uh, the size that you can handle. But most data re- the world is time series data. So this is the approach you will end up using if, if you want to, uh, apply it in your own research. So this is, this is, this, uh, RNNs is the way to go. So,


[User] Please transcribe this text

[Assistant] Put images into a recurrent neural network. It's the same thing. You take


[User] Please transcribe this text

[Assistant] You have to convert an image into numbers in some kind of way. A powerful way of doing that is convolutional neural networks. So you can take


[User] Please transcribe this text

[Assistant] Either 3D convolutional neural networks or 2D convolutional neural networks, ones that take time into consideration and one not.


[User] Please transcribe this text

[Assistant] So process that image to extract the representation of that image. And that becomes the input to the LSTM.


[User] Please transcribe this text

[Assistant] And the output at every single cell, at every single time step, is the predicted steering angle, the speed of the vehicle, and the torque. That's what the first place winner did. They didn't just do steering angle, they also did the speed and the torque. And the sequence length that they were using, for training and for testing, for the input and the output, is a sequence length of 10.


[User] Please transcribe this text

[Assistant] Uh, the question was, do they use supervised learning? Yep. So they were given the same thing as in deep Tesla, a sequence of frames where they have a sequence of steering angle, speed and torque. I think there's other information to available. So yeah, there's no, there's no reinforcement learning here. Question.


[User] Please transcribe this text

[Assistant] It's, it's, it's true that these diagrams kind of hide the number of parameters here. It's, but it's arbitrary just like convolutional neural networks are arbitrary. So,


[User] Please transcribe this text

[Assistant] The size of the input is arbitrary, the size of the sigmoid function, the tanh, is arbitrary, so you can make it as large as you want, as deep as you want. And the deeper and larger, the better.


[User] Please transcribe this text

[Assistant] What these folks actually use, so the way these competitions work and


[User] Please transcribe this text

[Assistant] If you've, uh, I encourage you if you're interested in machine learning to participate in Kaggle, I don't know how to pronounce it, competitions.


[User] Please transcribe this text

[Assistant] Where basically everyone's doing the same thing. You're using LSTMs, or if it's a one-to-one mapping, you're using convolutional neural networks, or fully connected networks, with some clever pre-processing. And the whole job is, that takes months, and you probably, if you're a researcher, that's what you'd be doing, your own research, is playing with parameters.


[User] Please transcribe this text

[Assistant] Playing with preprocessing of the data, playing with the different parameters that control the size of the network, the learning rate, I mentioned, you know, this type of optimizer, all these kinds of things, that's what you're playing with.


[User] Please transcribe this text

[Assistant] Using your own human intuition and you're using your...


[User] Please transcribe this text

[Assistant] Whatever probing you can do in monitoring the performance of the network through time.


[User] Please transcribe this text

[Assistant] You said that there is a memory of 10 in this LCM and I thought RNNs are supposed to be arbitrary or whatever. So it has to do with the training.


[User] Please transcribe this text

[Assistant] The, uh, how the network is trained. So it's trained with sequences of 10. The structure is still the same. You only have one.


[User] Please transcribe this text

[Assistant] Uh, cell that's looping onto each other. But the question is what, in what chunks?


[User] Please transcribe this text

[Assistant] Of s- what is the size of the sequence of which you're doing the training and then the testing? So.


[User] Please transcribe this text

[Assistant] You don't have to, it can be arbitrary length, it's just usually better to be consistent, uh, and have a fixed length.


[User] Please transcribe this text

[Assistant] But it's, it's the same, there's no, uh, you're not stacking 10 cells together. It's just a single cell still.


[User] Please transcribe this text

[Assistant] So the third place winner, team chauffeur,


[User] Please transcribe this text

[Assistant] Use something called transfer learning and it's something I don't think I mentioned uhm, uh, but it's kind of implied.


[User] Please transcribe this text

[Assistant] The amazing power of neural networks, so first you need a lot of data to do anything.


[User] Please transcribe this text

[Assistant] So that's a cost, that's a limitation of neural networks. But what you could do is, so there's, uh, there's,


[User] Please transcribe this text

[Assistant] Neural networks that have been trained on very large datasets. On ImageNet.


[User] Please transcribe this text

[Assistant] Uh, there's a VGGnet, Alexnet, Resnet, all these networks that train in huge amounts of data.


[User] Please transcribe this text

[Assistant] But those networks were trained to tell the difference between a cat and a dog, or whatever, or the specific object recognition in single images.


[User] Please transcribe this text

[Assistant] How do I, I then take that network and apply it to my problem, say of driving, of lane detection, or classifying, uh, medical diagnosis of cancer or not? The beauty of neural networks is you don't


[User] Please transcribe this text

[Assistant] It depends, but the transfer, the promise of transfer learning is that you can just take that network, chop off the final layer,


[User] Please transcribe this text

[Assistant] It teaches you to predict lane or no lane, truck or no truck.


[User] Please transcribe this text

[Assistant] And so, as long as the visual space under which the networks operate is similar, or the data space, like if it's audio or whatever, if it's similar, if the features are useful that you learned in studying the problem of cat versus dog deeply,


[User] Please transcribe this text

[Assistant] You have learned actually how to see the world. And so you can apply that visual knowledge, you can transfer that learning to another domain. And that's, that's the beautiful power of neural networks is they're transferable.


[User] Please transcribe this text

[Assistant] And so what they did here is they took


[User] Please transcribe this text

[Assistant] I didn't, uh, spend enough time looking through the code. So they, but they, they, uh, I'm not sure which of the giant networks they took, but they took a giant convolution neural network. They pruned it down to, they chopped off.


[User] Please transcribe this text

[Assistant] The end layer, which produced 3,000 features, and they took those 3,000 features at every single image frame, and that's the XT. They gave that as the input to LSTM. And the sequence length in that case was 50. So this, this process is pretty, uh,


[User] Please transcribe this text

[Assistant] It's, it's uh, it's pretty similar across domains. That's the beauty of it.


[User] Please transcribe this text

[Assistant] And the art of neural networks is in the, well that's a good sign. I guess I should wrap it up.


[User] Please transcribe this text

[Assistant] The, the, the art of neural networks is in the hyperparameter tuning. And that's the tricky part. And that's the part you can be taught. That's experience.


[User] Please transcribe this text

[Assistant] Sadly enough. That's why they called, I talked about


[User] Please transcribe this text

[Assistant] Stochastic Gradient Descent, SGD. That's why, uh, whoever it was, Jeffrey Hinton, refers to it as, uh, Stochastic Graduate Student Descent.


[User] Please transcribe this text

[Assistant] Meaning you just keep hiring graduate students to play with the hyperparameters until the problem is solved.


[User] Please transcribe this text

[Assistant] I have about a hundred plus slides on driver state, which is the thing that I'm most passionate about. And I think we'll save the best for last, right? And I'll talk about that tomorrow. We have a guest speaker.


[User] Please transcribe this text

[Assistant] From the White House, they talk about the future of artificial intelligence from the perspective of policy and


[User] Please transcribe this text

[Assistant] What I'd like you to do first off, you register to submit the two tutorials, assignments, and uh, pick up.


[User] Please transcribe this text

[Assistant] Can we just set up boxes right here or something? Yea, just stop by and pick up a shirt. And give us a card on the way. Alright, thanks guys.


[User] Please transcribe this text

[Assistant] Move, Max! Don't waste your power on getting busted!


[User] Please transcribe this text

[Assistant] As far as we know, it's Nathan Prescott who's the real threat so far.


[User] Please transcribe this text

[Assistant] At least we're showing our faces together in public. So you absolutely should go with us. Borrow one of my outfits and come as a cute geek couple with your bitch Warren. Uh, I'm not big on cosplay.


[User] Please transcribe this text

[Assistant] Don't let life pass you by, Max. So, are you coming to the party or not?


[User] Please transcribe this text

[Assistant] You could explain it. I'm serious. Even meteorologists can't explain that snow. I bet Warren can. Did you talk to him yet?


[User] Please transcribe this text

[Assistant] I will later. Lucky you. I'm sure he'll track you down, Max. I'll see you around.


[User] Please transcribe this text

[Assistant] What's up, Kate? School. That's it. That video of you clubbing didn't look like homework.


[User] Please transcribe this text

[Assistant] Don't be shy. I think it's awesome you set a tongue record on video. You're going to be sorry someday.


[User] Please transcribe this text

[Assistant] Oh boohoo, I'm sorry you're a viral slut. I'm sure she had fun. Looks like it. I know Nathan hooked her up. And you know he has the good shit. Preach it, sister.


[User] Please transcribe this text

[Assistant] Yup, I still have goddamn paint all over my face. Good thing my faithful minions took their sweet time bringing me a towel.


[User] Please transcribe this text

[Assistant] We ran all the way- Give it a rest, Taylor. Now I know if I'm in an accident, I won't rely on you or Courtney for help. You can hang out with Kate. Or Max.


[User] Please transcribe this text

[Assistant] She's a weirdo with that dumb camera. I hate that I'm so quirky crap. Anyway.


[User] Please transcribe this text

[Assistant] Let's leave the link to Kate's video so everybody gets a chance to see her in action. You are such an evil biatch. I love it.


[User] Please transcribe this text

[Assistant] I know I can trust you since you were so nice asking about my mom in the hospital, but Victoria will like, make my life hell if I blab, okay?


[User] Please transcribe this text

[Assistant] Dude, do not even torture yourself like that. Let's focus on looking for clues, okay?


[User] Please transcribe this text

[Assistant] Not as much as I hate thinking it. Max, we have to find Rachel soon. We have to.


[User] Please transcribe this text

[Assistant] I just don't want you to get into any more trouble. Look at all the trouble dropping in Arcadia Bay. At this point, who gives a fuck anymore? We're in it to win it, Max. Lead the way. I'm so glad you're my partner in crime. As long as you're my partner in time.


[User] Please transcribe this text

[Assistant] Well now we definitely have to get this door open. Believe it or not, I know a little about lockpicking. Thanks to Frank. I might as well test out my thief skill.


[User] Please transcribe this text

[Assistant] But I'll use my DIY lockpick tools while you come up with a better plan.


[User] Please transcribe this text

[Assistant] I'm gonna go put it together. Can you stay here and not get caught? I might get on the other side of that door before you, Lupin.


[User] Please transcribe this text

[Assistant] Man, I can see why the principal locks this room up. Fancy floor crap. He must want everybody to know he has money. But no taste.


[User] Please transcribe this text

[Assistant] How can you trust somebody who has a fucking bronze bird in his office? I'm glad I was expelled.


[User] Please transcribe this text

[Assistant] Find us some clues about Rachel, or Kate, or Nathan. Anybody. I'm on the case.


[User] Please transcribe this text

[Assistant] Nathan accuses Rachel of bringing drugs on campus, and my step troll went along because he thinks Rachel was a bad influence on me.


[User] Please transcribe this text

[Assistant] And you know the Prescott's dropped major bank to bury Nathan's real file.


[User] Please transcribe this text

[Assistant] David was stalking Kate, hassling me, and now we know he was all over Rachel too. Oh, we are so going into his garage files. Plus I'm getting a little paranoid in here. We got our info, let's bail.


[User] Please transcribe this text

[Assistant] Or did you just rewind because I tried to steal the chair? Shit, I'm confused.


[User] Please transcribe this text

[Assistant] I've got my power to protect you, right? There's a lot of power in that horse-choking wad of cash. But yes, Moral Max is right again. I guess. Let's get the hell out of this office morgue.


[User] Please transcribe this text

[Assistant] You wanna take that risk now? It's been a cray week. You didn't let me take that money to pay off Frank, so if he pops a cap in my skull, at least allow me to have a little carefree fun for a few minutes. Splish splash. You're right. We hella deserve it. Splish splash. Did you actually just say hella? I think I'm a good bad influence on you.


[User] Please transcribe this text

[Assistant] Let's check the door. A friend gets so wasted he sometimes forgets to lock it.


[User] Please transcribe this text

[Assistant] I can't believe you hung out with him. Not anymore. We have to be casual ninjas here.


[User] Please transcribe this text

[Assistant] Okay, here's the plan. I'll go to the diner and distract Frank by telling him I have his money, but he needs to come with me. Then you come in and rewind so Frank doesn't see me. Then you can tell Frank he needs to check out his RV, and then you rewind after you get the key, and, uh, and... Chloe.


[User] Please transcribe this text

[Assistant] Don't let my epic plan get in the way of yours. I'll try to scrounge up a Scooby snack to distract his mangy mutt.


[User] Please transcribe this text

[Assistant] Damn, I thought my room was a shit hole. You're not a creepy drug dealer. Frank has issues, but he's not creepy. At least I didn't think so until I saw him with Rachel's bracelet.


[User] Please transcribe this text

[Assistant] Crappy wifi out here. Let's take him days to download the porn. Yuck!


[User] Please transcribe this text

[Assistant] That makes me ill that Rachel posed like this for Frank. I wrote him love letters. I can't believe she was banging Frank!


[User] Please transcribe this text

[Assistant] Rachel straight up lied to my face. Why didn't she say anything? Because she knew how you would react. And she wasn't much of a friend, huh? Just another person who shits all over me.


[User] Please transcribe this text

[Assistant] Yes, Kate Marsh almost killed herself. Such sad, okay? That doesn't make me feel any better about my fucked up life. Get it? So who do you most want to blame? My fucking dad, of course. Hello?


[User] Please transcribe this text

[Assistant] Really? Yes, I do. Damn right. He chose to go out that door and leave me forever.


[User] Please transcribe this text

[Assistant] I know that, Max. My mom actually blames herself. Just because she wanted a ride home from work. Sometimes even I blame her. No you don't. Yes, Max, I do.


[User] Please transcribe this text

[Assistant] But I was with you that day. It was just a terrible accident. I wish that made me feel better. But ever since he died, my life has been dipped in shit.


[User] Please transcribe this text

[Assistant] Even though you left me with said bully and ran off with that girl, she's pretty punk rock. How do you know her?


[User] Please transcribe this text

[Assistant] I bet you were glad to see her blast up in that truck like a rockstar. Which begs the question, why is Nathan Prescott getting psycho on you?


[User] Please transcribe this text

[Assistant] Well that was easier than I thought. Cool, I'll text you the info.


[User] Please transcribe this text

[Assistant] Justin. Getting my coffee on before I cut class and destroy some rails. Shouldn't you be taking photos?


[User] Please transcribe this text

[Assistant] Speaking of buds, I saw your bro Trevor sneaking out of Dana's room. Trevor finally told me about him and Dana hooking up. He got there first, so I snooze and I lose. See you later, Max.


[User] Please transcribe this text

[Assistant] If by big you mean none, then sure. I caught a lot of big ones today. Now let me mourn over my eggs.


[User] Please transcribe this text

[Assistant] Story number 12. The Adventure of the Copper Beaches.


[User] Please transcribe this text

[Assistant] "'To the man who loves art for its own sake,' remarked Sherlock Holmes, tossing aside the advertisement sheet of the Daily Telegraph.


[User] Please transcribe this text

[Assistant] It is frequently in its least important and lowliest manifestations that the keenest pleasure is to be derived.


[User] Please transcribe this text

[Assistant] It is pleasant to me to observe, Watson, that you have so far grasped this truth, that in these little records of our cases which you have been good enough to draw up,


[User] Please transcribe this text

[Assistant] And I am bound to say, occasionally to embellish, you have given prominence not so much to the many cause célèbre and sensational trials in which I have figured


[User] Please transcribe this text

[Assistant] But rather to those incidents which may have been trivial in themselves, but which have given room for those faculties of deduction and of logical synthesis which I have made my special province.


[User] Please transcribe this text

[Assistant] And yet," said I, smiling, I cannot quite hold myself absolved from the charge of sensationalism which has been urged against my records.


[User] Please transcribe this text

[Assistant] You have erred, perhaps," he observed, taking up a glowing cinder with the tongs, and lighting with it the long cherry-wood pipe which was wont to replace his clay when he was in a disputatious rather than a meditative mood.


[User] Please transcribe this text

[Assistant] You have erred perhaps in attempting to put colour and life into each of your statements, instead of confining yourself to the task of placing upon record that severe reasoning from cause to effect, which is really the only notable feature about the thing.


[User] Please transcribe this text

[Assistant] It seems to me that I've done you full justice in the matter," I remarked with some coldness.


[User] Please transcribe this text

[Assistant] For I was repelled by the egotism which I had more than once observed to be a strong factor


[User] Please transcribe this text

[Assistant] No, it is not selfishness or conceit," said he, answering, as was his won't, my thoughts rather than my words.


[User] Please transcribe this text

[Assistant] If I claim full justice for my art, it is because it is an impersonal thing, a thing beyond myself.


[User] Please transcribe this text

[Assistant] Therefore it is upon the logic, rather than upon the crime, that you should dwell.


[User] Please transcribe this text

[Assistant] You have degraded what should have been a course of lectures into a series of tales.


[User] Please transcribe this text

[Assistant] It was a cold morning of the early spring, and we sat after breakfast on either side of a cheery fire in the old room at Baker Street.


[User] Please transcribe this text

[Assistant] A thick fog rolled down between the lines of dun-coloured houses, and the opposing windows loomed like dark, shapeless blurs through the heavy yellow wreaths.


[User] Please transcribe this text

[Assistant] Our gas was lit and shone on the white cloth and glimmer of china and metal, for the table had not been cleared yet.


[User] Please transcribe this text

[Assistant] Sherlock Holmes had been silent all the morning, dipping continuously into the advertisement columns of a succession of papers, until at last, having apparently given up his search, he had emerged in no very sweet temper to lecture me upon my literary shortcomings.


[User] Please transcribe this text

[Assistant] At the same time, he remarked after a pause, during which he had sat puffing at his long pipe and gazing down into the fire,


[User] Please transcribe this text

[Assistant] You can hardly be open to a charge of sensationalism.


[User] Please transcribe this text

[Assistant] For out of these cases, which you have been so kind as to interest yourself in, a fair proportion do not treat of crime in its legal sense at all.


[User] Please transcribe this text

[Assistant] The small matter in which I endeavoured to help the King of Bohemia,


[User] Please transcribe this text

[Assistant] The singular experience of Miss Mary Sutherland, the problem connected with the man with the


[User] Please transcribe this text

[Assistant] We're all matters which are outside the pale of the law.


[User] Please transcribe this text

[Assistant] But in avoiding the sensational I fear that you may have bordered on the trivial. The end may have been so, I answered, but the methods I hold to have been novel and of interest.


[User] Please transcribe this text

[Assistant] My dear fellow, what do the public, the great unobservant public, who could hardly tell a weaver by his tooth or a compositor by his left thumb, care about the finer shades of analysis and deduction?


[User] Please transcribe this text

[Assistant] But indeed, if you are trivial, I cannot blame you, for the days of the great cases are past.


[User] Please transcribe this text

[Assistant] Man, or at least criminal man, has lost all enterprise and originality. As to my own little practice, it seems to be degenerating into an agency for recovering lost-lead pencils and giving advice to young ladies from boarding schools. I think that I have touched bottom at last, however.


[User] Please transcribe this text

[Assistant] This note I had this morning marks my zero point, I fancy. Read it. He tossed a crumpled letter across to me.


[User] Please transcribe this text

[Assistant] It was dated from Montague Place upon the preceding evening, and ran thus.


[User] Please transcribe this text

[Assistant] Dear Mr Holmes, I am very anxious to consult you as to whether I should or should not accept a situation which has been offered to me as governess.


[User] Please transcribe this text

[Assistant] I shall call at half-past ten tomorrow, if I do not inconvenience you, yours faithfully, Violet Hunter.


[User] Please transcribe this text

[Assistant] Do you know the young lady?" I asked. Not I. It is half-past ten now. Yes, and I have no doubt that is her ring. It may turn out to be of more interest than you think.


[User] Please transcribe this text

[Assistant] You remember that the affair of the blue carbuncle, which appeared to be a mere whim at first, developed into a serious investigation. It may be so in this case also.


[User] Please transcribe this text

[Assistant] Well, let's hope so, but our doubts will very soon be solved, for here, unless I am much mistaken, is the person in question.


[User] Please transcribe this text

[Assistant] As he spoke the door opened, and a young lady entered the room. She was plainly but neatly dressed, with a bright, quick face, freckled like a plover's egg, and with the brisk manner of a woman who has had her own way to make in the world.


[User] Please transcribe this text

[Assistant] You will excuse me for troubling you, I am sure," said she, as my companion rose to greet her.


[User] Please transcribe this text

[Assistant] But I have had a very strange experience, and as I have no parents or relations of any sort from whom I could ask advice, I thought that perhaps you would be kind enough to tell me what I should do. Pray take a seat, Miss Hunter. I shall be happy to do anything that I can to serve you.


[User] Please transcribe this text

[Assistant] I could see that Holmes was favourably impressed by the manner and the speech of his new client.


[User] Please transcribe this text

[Assistant] He looked her over in his searching fashion, and then composed himself with his lids drooping and his fingertips together to listen to her story.


[User] Please transcribe this text

[Assistant] I have been a governess for five years," said she, in the family of Colonel Spence Munro.


[User] Please transcribe this text

[Assistant] But two months ago the Colonel received an appointment at Halifax in Nova Scotia and took his children over to America with him, so that I found myself without a situation.


[User] Please transcribe this text

[Assistant] I advertised and I answered advertisements, but without success.


[User] Please transcribe this text

[Assistant] At last the little money which I had saved began to run short, and I was at my wits' end as to what I should do.


[User] Please transcribe this text

[Assistant] There is a well-known agency for governesses in the West End called Westerways, and there I used to call about once a week, in order to see whether anything had turned up which might suit me. Westerway was the name of the founder of the business, but it is really managed by Miss Stoper.


[User] Please transcribe this text

[Assistant] She sits in her own little office and the ladies who are seeking employment wait in an anteroom and are then shown in one by one when she consults her ledgers and sees whether she has anything which would suit them.


[User] Please transcribe this text

[Assistant] Well, when I called last week, I was shown into the little office as usual, but I found that Miss Stoper was not alone.


[User] Please transcribe this text

[Assistant] A prodigiously stout man, with a very smiling face, and a great heavy chin, which rolled down fold upon fold over his throat, sat at her elbow with a pair of glasses on his nose, looking very earnestly at the ladies who entered.


[User] Please transcribe this text

[Assistant] He seemed quite enthusiastic and rubbed his hands together in the most genial fashion.


[User] Please transcribe this text

[Assistant] He was such a comfortable looking man that it was quite a pleasure to look at him. You are looking for a situation, miss? he asked. Yes, sir. As governess? Yes, sir. And what salary do you wish?


[User] Please transcribe this text

[Assistant] I had £4 a month in my last place with Colonel Spence Munro.


[User] Please transcribe this text

[Assistant] Throwing his fat hands out into the air like a man who is in a boiling passion.


[User] Please transcribe this text

[Assistant] How could anyone offer so pitiful a sum to a lady with such attractions and accomplishments?


[User] Please transcribe this text

[Assistant] My accomplishments, sir, may be less than you imagine," said I.


[User] Please transcribe this text

[Assistant] A little French, a little German, music and drawing. Tut tut! he cried. This is all quite beside the question.


[User] Please transcribe this text

[Assistant] The point is, have you or have you not the bearing and deportment of a lady?


[User] Please transcribe this text

[Assistant] There it is in a nutshell. If you have not, you are not fitted for the rearing of a child who may some day play a considerable part in the history of the country.


[User] Please transcribe this text

[Assistant] But if you have, why? Then how could any gentleman ask you to condescend to accept anything under the three figures?


[User] Please transcribe this text

[Assistant] You may imagine, Mr Holmes, to me, destitute as I was, such an offer seemed almost too good to be true.


[User] Please transcribe this text

[Assistant] The gentleman, however, seeing perhaps the look of incredulity upon my face, opened a pocket-book and took out a note.


[User] Please transcribe this text

[Assistant] It is also my custom," said he, smiling in the most pleasant fashion, until his eyes were just two little shining slits amid the white creases of his face.


[User] Please transcribe this text

[Assistant] To advance to my young ladies half their salary beforehand, so that they may meet any little expenses of their journey and their wardrobe.


[User] Please transcribe this text

[Assistant] It seemed to me that I had never met so fascinating and so thoughtful a man.


[User] Please transcribe this text

[Assistant] As I was already in debt to my tradesmen, the advance was a great convenience.


[User] Please transcribe this text

[Assistant] And yet there was something unnatural about the whole transaction.


[User] Please transcribe this text

[Assistant] Which made me wish to know a little more before I quite committed myself.


[User] Please transcribe this text

[Assistant] Hampshire, charming rural place. The copper beaches, five miles on the far side of Winchester. It is the most lovely country, my dear young lady, and the dearest old country house. And my duties, sir. I should be glad to know what they would be.


[User] Please transcribe this text

[Assistant] One child, one dear little romper, just six years old. Oh, if you could see him, killing cockroaches with a slipper. Smack, smack, smack. Three gone before you could wink.