[User] Please transcribe this text

[Assistant] So how are these things trained? Very similar to supervised learning.


[User] Please transcribe this text

[Assistant] And the discounted expected reward from future states.


[User] Please transcribe this text

[Assistant] The loss function here for a neural network, the neural network learns with a loss function. It takes the reward received at the current state.


[User] Please transcribe this text

[Assistant] Does a forward pass through a neural network to estimate the value of the future state of the best action to take in the future state and then subtracts that


[User] Please transcribe this text

[Assistant] From the forward pass, uh, uh, through the network for the current state in action.


[User] Please transcribe this text

[Assistant] So you take the difference between what your Q estimator, the neural network, believes the value of the current state is and what it more likely is to be.


[User] Please transcribe this text

[Assistant] Based on the value of the future states that are reachable based on the actions you can take.


[User] Please transcribe this text

[Assistant] Input is the state, output is the Q-value for each action, or in this diagram, input is the state in action and the output is the Q-value. It's very similar architectures. So given a transition of S, A, R, S'


[User] Please transcribe this text

[Assistant] The, the update is do a feedforward pass to the network for the current state. Do a feedforward pass for each of the possible actions taken in the next state. And that's how we compute the two parts of the loss function.


[User] Please transcribe this text

[Assistant] Again, loss function, backpropagation is how the network is trained.


[User] Please transcribe this text

[Assistant] This has actually been around for much longer than Deep Mind. A few tricks made it, made it really work.


[User] Please transcribe this text

[Assistant] So as the games are played through simulation, or if it's a physical system, as it acts in the world,


[User] Please transcribe this text

[Assistant] It's actually collecting the observations into a library of experiences and the training is performed by randomly sampling the library in the past.


[User] Please transcribe this text

[Assistant] By randomly sampling the previous experiences and batches.


[User] Please transcribe this text

[Assistant] So you're not always training on the natural continuous evolution of the system. You're training on randomly picked batches of those experiences. That's a huge


[User] Please transcribe this text

[Assistant] It's a, it's a, seems like a subtle trick, but it's a really important one.


[User] Please transcribe this text

[Assistant] Again, subtle trick, as in a lot of deep learning approaches, the subtle tricks make all the difference, is fixing the target network. For the loss function, if you notice,


[User] Please transcribe this text

[Assistant] You have to use the neural network, the singular neural network, the GQN network to estimate the value of the current state and action pair.


[User] Please transcribe this text

[Assistant] And the next. So you're using it multiple times.


[User] Please transcribe this text

[Assistant] And as you perform that operation, you're updating the network. Which means the target function inside that loss function is always changing.


[User] Please transcribe this text

[Assistant] So your, the very nature of your loss function is changing all the time as you're learning. And that's a big problem for stability.


[User] Please transcribe this text

[Assistant] That can create big problems for the learning process. So this little trick is to fix the network and only update it every say thousand steps.


[User] Please transcribe this text

[Assistant] So as you train the network, the network that's used to compute the target function inside the loss function is fixed.


[User] Please transcribe this text

[Assistant] It produces a more stable computational loss function. So the ground doesn't shift under you as you're trying to find a minimal.


[User] Please transcribe this text

[Assistant] The loss function doesn't change in unpredictable, difficult to understand ways.


[User] Please transcribe this text

[Assistant] And reward clipping, which is always true with general, the systems that are operating, uh, uh, seeking to operate in a generalized way, is for very, for these various games.


[User] Please transcribe this text

[Assistant] The points are different. Some points are low, some points are high, some go positive and negative, and they're all normalized to a point where the good points or the positive points are a 1 and negative points are a negative 1. That's reward clipping. Simplify the reward structure.


[User] Please transcribe this text

[Assistant] And because a lot of the games are 30 FPS or 60 FPS and the actions are not


[User] Please transcribe this text

[Assistant] It's not valuable to take actions at such a high rate inside of these particular Atari games that you only take an action every four steps, while still taking in the frames as part of the temporal window to make decisions. Tricks, but hopefully gives you a sense


[User] Please transcribe this text

[Assistant] of the kind of things necessary for both


[User] Please transcribe this text

[Assistant] Seminal papers like this one, and for the more important accomplishment of winning deep traffic, is the, is the tricks make all the difference. Here on the bottom is...


[User] Please transcribe this text

[Assistant] The circle is when the technique is used and when it's not. Looking at replay and target. Fixed target network and experienced replay. When both are used for the game of breakout, river raid, sea quest and space invaders, the higher the number, the better it is. The more points achieved.


[User] Please transcribe this text

[Assistant] So when, it gives you a sense that when replay and target both give significant improvements in the performance of the system.


[User] Please transcribe this text

[Assistant] Order of magnitude improvements. Two orders of magnitude for breakout.


[User] Please transcribe this text

[Assistant] And here is pseudocode of implementing DQN, the learning.


[User] Please transcribe this text

[Assistant] The key thing to notice, and you can look through the slides, is the, the, the loop, the wild loop.


[User] Please transcribe this text

[Assistant] Of playing through the games and selecting the actions to play.


[User] Please transcribe this text

[Assistant] It's not part of the training. It's part of the saving.


[User] Please transcribe this text

[Assistant] Uh, the observations, the state, action, reward, next state observations and saving them into replay memory into that library. And then you sample randomly from that replay memory to then train the network based on the loss function.


[User] Please transcribe this text

[Assistant] And with probability up, up top, the probability epsilon, select a random action. That epsilon is the probability of exploration that decreases. That's something you'll see in deep traffic as well.


[User] Please transcribe this text

[Assistant] Is the rate at which that exploration decreases over time through the training process. You want to explore a lot first and less and less over time.


[User] Please transcribe this text

[Assistant] So this algorithm has been able to accomplish in 2015 and since a lot of incredible things.


[User] Please transcribe this text

[Assistant] It's for the first time that raw sensory information was used to create a system that acts and makes sense to the world. Makes sense to the physics of the world enough to be able to succeed in it from very little information. But these games are trivial.


[User] Please transcribe this text

[Assistant] This DQN approach has been able to outperform a lot of the Atari games. That's what's been reported on. Outperform the human level performance. But again, these games are trivial.


[User] Please transcribe this text

[Assistant] What I think, and perhaps biased, I'm biased, but one of the greatest accomplishments of artificial intelligence in the last decade.


[User] Please transcribe this text

[Assistant] At least from the philosophical or the research perspective.


[User] Please transcribe this text

[Assistant] It's DeepMind's system that beat the best in the world in the game of Go. So what's the game of Go? Simple.


[User] Please transcribe this text

[Assistant] I won't get into the rules, but basically it's a 19 by 19 board, shown on the bottom of the slide.


[User] Please transcribe this text

[Assistant] The number of legal game positions is 2 times 10 to the power of 170.


[User] Please transcribe this text

[Assistant] It's a very large number of possible positions to consider. At any one time, especially the game evolves, the number of possible moves is huge. Much larger than in chess. So that's why AI community thought that this game is not solvable.


[User] Please transcribe this text

[Assistant] When AlphaGo used this, used human expert position play


[User] Please transcribe this text

[Assistant] To seed in a supervised way, reinforcement learning approach. And I'll describe it in a little bit of detail in a couple of slides here. To beat the best in the world.


[User] Please transcribe this text

[Assistant] And beat the best in the world in an extremely complex game. This is not Atari.


[User] Please transcribe this text

[Assistant] This is a much higher order difficulty game. And the quality of players that it's competing in is much higher. And it's able to extremely quickly here to achieve a rating that's better than AlphaGo.


[User] Please transcribe this text

[Assistant] And better than the different variants of AlphaGo. And certainly better than the best of the human players. In 21 days. Of self play.


[User] Please transcribe this text

[Assistant] So how does it work? All of these approaches, much, much like the previous ones, the traditional ones, they're not based on deep learning.


[User] Please transcribe this text

[Assistant] Are using Monte Carlo Tree Search, MCTS.


[User] Please transcribe this text

[Assistant] Which is, when you have such a large state space, you start at a board and you play and you choose moves.


[User] Please transcribe this text

[Assistant] With some exploitation, exploration, balancing.


[User] Please transcribe this text

[Assistant] Choosing to explore totally new positions or to go deep in the positions you know are good until the bottom of the game is reached until the final state is reached and then you back propagate the


[User] Please transcribe this text

[Assistant] Quality of the choices you made leading to that position. And in that way you learn the value of, of board positions and play. That's been used by the most successful Go playing engines before and Alpha Go since.


[User] Please transcribe this text

[Assistant] But you might be able to guess what's the difference with AlphaGo versus the previous approaches.


[User] Please transcribe this text

[Assistant] They use the neural network as the intuition, quote unquote, to what are the good states, what are the good next board positions to explore.


[User] Please transcribe this text

[Assistant] And the key things, again, the tricks make all the difference. That made alpha go zero work.


[User] Please transcribe this text

[Assistant] And work much better than AlphaGo is first, because there was no expert play, instead of human games.


[User] Please transcribe this text

[Assistant] AlphaGo used that very same Monte Carlo tree search algorithm, MCTS, to do an intelligent look ahead based on the neural network prediction of what are the good states to take.


[User] Please transcribe this text

[Assistant] It checked that, instead of human expert play, it checked how good indeed are those states. It's a simple look ahead action.


[User] Please transcribe this text

[Assistant] That does the ground truth, that does the target, uh, correction that produces the loss function. The second part is the multi-task learning, or what's now called multi-task learning, is the network is, is uh, quote unquote, two-headed.


[User] Please transcribe this text

[Assistant] In the sense that first it outputs the probability of which move to take, the obvious thing, and it's also producing a probability of winning.


[User] Please transcribe this text

[Assistant] And there's a few ways to combine that information and continuously train both parts of the network depending on the choice taken. So you want to take the best choice in the short term and achieve the positions that are at the highest likelihood of winning for the player that's whose turn it is. And another big step


[User] Please transcribe this text

[Assistant] Is that they updated from 2015, they updated the state-of-the-art architecture, which are now the architecture that one ImageNet is residual networks. ResNet for ImageNet.


[User] Please transcribe this text

[Assistant] Those, that's it. And those little changes made all the difference.


[User] Please transcribe this text

[Assistant] So that takes us through deep traffic and the 8 billion hours stuck in traffic.


[User] Please transcribe this text

[Assistant] America's pastime. So we tried to simulate driving, the behavioral layer of driving.


[User] Please transcribe this text

[Assistant] So not the immediate control, not the motion planning, but beyond that on top.


[User] Please transcribe this text

[Assistant] On top of those control decisions, the human interpretable decisions of changing lane, of speeding up, slowing down, modeling that


[User] Please transcribe this text

[Assistant] In a micro traffic simulation framework that's popular in traffic engineering, the kind shown here.


[User] Please transcribe this text

[Assistant] We apply the deep reinforcement learning to that. We call it deep traffic.


[User] Please transcribe this text

[Assistant] The goal is to achieve the highest average speed over a long period of time, weaving in and out of traffic.


[User] Please transcribe this text

[Assistant] For students here, the requirement is to follow the tutorial and achieve a speed of 65 miles an hour.


[User] Please transcribe this text

[Assistant] And if you really want to achieve a speed over 70 miles an hour, which is what's required to win. And perhaps upload your own image to make sure you look good doing it.


[User] Please transcribe this text

[Assistant] What you should do. Clear instructions. To compete, read the tutorial.


[User] Please transcribe this text

[Assistant] You can change parameters in the code box on that website, cars.mit.edu.


[User] Please transcribe this text

[Assistant] Click the white button that says apply code, which applies the code that you write. These are the parameters that you specify for the neural network.


[User] Please transcribe this text

[Assistant] It applies those parameters, creates the architecture that you specify and now you have a network written in JavaScript living in the browser ready to be trained. Then you click the blue button that says run training and that trains the network.


[User] Please transcribe this text

[Assistant] Much faster than what's actually being visualized in the browser. A thousand times faster.


[User] Please transcribe this text

[Assistant] By evolving the game, making decisions, taking in the grid space, as I'll talk about here in a second. The speed limit is 80 miles an hour.


[User] Please transcribe this text

[Assistant] Based on the various adjustments we made to the game, reaching 80 miles an hour is certainly impossible on average. And reaching some of the speeds that we've achieved last year is much, much, much more difficult. Finally, when you're happy and the training is done, submit the model to competition.


[User] Please transcribe this text

[Assistant] You can click the request visualization specifying the custom image and the color.


[User] Please transcribe this text

[Assistant] Okay, so here's the simulation. Speed limit 80 miles an hour. Cars, 20 on the screen. One of them is a red one in this case. That one is controlled by a neural network. It's allowed the actions to speed up, slow down, change lanes, left, right, or stay exactly the same.


[User] Please transcribe this text

[Assistant] The other cars are pretty dumb. They speed up, slow down, turn left, right, but they don't have a purpose in their existence. They do so randomly. Or at least purpose has not been discovered.


[User] Please transcribe this text

[Assistant] The road, the car, the speed. The road is a grid space. An occupancy grid that specifies


[User] Please transcribe this text

[Assistant] And when there's other cars that are going slow, the value in that grid is the speed of that car. That's the state space. That's the state representation. And you can choose how much, what slice of that state space you take in. That's the input to the neural network.


[User] Please transcribe this text

[Assistant] For visualization purposes, you can choose normal speed or fast speed for watching the network operate.


[User] Please transcribe this text

[Assistant] And there's display options to help you build intuition about what the network takes in and what space the car is operating in. The default is no extra information is added. Then there's the learning input which visualizes exactly which part of the road


[User] Please transcribe this text

[Assistant] Here's a safety system. You can think of this system as ACC, basic radar ultrasonic sensors helping you avoid the obvious collisions to obviously detectable objects around you. And the task for this red car, for this neural network is to move about this space


[User] Please transcribe this text

[Assistant] Uh, is to move about the space under the constraints of the safety system.


[User] Please transcribe this text

[Assistant] So the goal for the car is to not get stuck in traffic.


[User] Please transcribe this text

[Assistant] The input, like DQN, is the state space, the output is the value of the different actions, and based on the epsilon parameter through training and through inference evaluation process, you choose how much exploration you want to do. These are all parameters. The learning is done in the browser, on your own computer.


[User] Please transcribe this text

[Assistant] The brain is where the neural network is contained, both for the training and the evaluation.


[User] Please transcribe this text

[Assistant] Forward length and backward length. Lane side, number of lanes to the side that you see. Patches ahead is the patches ahead that you see. Patches behind is patches behind that you see. New this year can control the number of agents that are controlled by the neural network. Anywhere from 1 to 10.


[User] Please transcribe this text

[Assistant] And the evaluation is performed exactly the same way. You have to achieve the highest average speed for the agents.


[User] Please transcribe this text

[Assistant] The very critical thing here is the agents are not aware of each other. So they're not jointly planning. The network is trained.


[User] Please transcribe this text

[Assistant] Under the joint objective of achieving the average speed for all of them. But the actions are taking in a greedy way for each. It's very interesting what can be learned in this way.


[User] Please transcribe this text

[Assistant] Because these kinds of approaches are scalable to an arbitrary number of cars. And you can imagine us plopping down the best cars from this class together and having them compete in this way. The best neural networks.


[User] Please transcribe this text

[Assistant] Because they're full in their greedy operation, the number of networks that can concurrently operate is fully scalable. There's a lot of parameters.


[User] Please transcribe this text

[Assistant] The layers, the many layers, types that can be added. Here's a fully connected layer with 10 neurons. The activation functions, all of these things can be customized as is specified in the tutorial.


[User] Please transcribe this text

[Assistant] The final layer, a fully connected layer with


[User] Please transcribe this text

[Assistant] I'll put a five regression giving the value of each of the five actions. And there's a lot of more specific parameters, some of which I've discussed. From gamma to epsilon to experience replay size to learning rate and temporal window.


[User] Please transcribe this text

[Assistant] The optimizer, the learning rate, momentum, batch size, L2L1 decay for regularization and so on. There's a big white button that says apply code that you press.


[User] Please transcribe this text

[Assistant] That kills all the work you've done up to this point, so be careful doing it. You should be doing it only at the very beginning.


[User] Please transcribe this text

[Assistant] And the network state gets shipped to the main simulation from time to time. So the thing you see in the browser as you open up the website is running the same network that's being trained. And regularly it updates that network. So it's getting better and better even if the training takes weeks for you.


[User] Please transcribe this text

[Assistant] It's constantly updating the network you see on the left. So if the car for the network that you're training is just standing in place and not moving, it's probably


[User] Please transcribe this text

[Assistant] Time to restart and change the parameters, maybe add a few layers to your network.


[User] Please transcribe this text

[Assistant] Number of iterations is certainly an important parameter to control.


[User] Please transcribe this text

[Assistant] And the evaluation is something we've done a lot of work on since last year to remove the degree of randomness, to remove the, the incentive to submit the same code over and over again to hope to produce a higher reward, uh, a higher evaluation score.


[User] Please transcribe this text

[Assistant] The method for evaluation is to collect the average speed over 10 runs about 45


[User] Please transcribe this text

[Assistant] Seconds of game each, not minutes, 45 simulated seconds. And there is 500 of those, and we take the median speed of the 500 runs. It's done server side, so extremely difficult to cheat. I urge you to try.


[User] Please transcribe this text

[Assistant] You can try it locally, there's a start evaluation run, but that one doesn't count. That's just for you to feel better about your network.


[User] Please transcribe this text

[Assistant] Loading is saving. If the network is huge and you want to switch computers, you can save the network. It saves both the architecture of the network and the weights on the network and you can load it back in.


[User] Please transcribe this text

[Assistant] Obviously, when you load it in, it's not, uh, saving any of the data you've already done. You can't do transfer learning with JavaScript in the browser yet.


[User] Please transcribe this text

[Assistant] Submitting your network, submit model to competition, and make sure you run training first. Otherwise, it'll be initiated, the weights are initiated randomly and it will not do so well.


[User] Please transcribe this text

[Assistant] You can resubmit as often you like and the highest score is what counts.


[User] Please transcribe this text

[Assistant] The coolest part is you can load your custom image, specify colors, and request a visualization.


[User] Please transcribe this text

[Assistant] We have not yet shown the visualization, but I promise you it's going to be awesome.


[User] Please transcribe this text

[Assistant] Again, read the tutorial, change the parameters in the code box, click apply code, run training.


[User] Please transcribe this text

[Assistant] Everybody in this room, on the way home, on the train, hopefully not in your car, should be able to do this in the browser. And then you can visualize, request visualization, because it's an expensive process. You have to want it for us to do it, uh, because we have to run it server side.


[User] Please transcribe this text

[Assistant] Competition link is there. GitHub starter code is there. And the details for those that truly want to win is in the archive paper. So the question,


[User] Please transcribe this text

[Assistant] That will come up throughout is whether these reinforcement learning approaches are at all or rather if action, planning, control is amenable to learning.


[User] Please transcribe this text

[Assistant] Certainly in the case of driving, we can't do what AlphaGo Zero did. We can't.


[User] Please transcribe this text

[Assistant] Learn from scratch from self-play because that would result in millions of crashes in order to learn to avoid the crashes. Unless we're working like we are with DeepCrash on the RC car or we're working on simulation.


[User] Please transcribe this text

[Assistant] So we can look at expert data, we can look at driver data, which we have a lot of and learn from. It's an open question whether this is applicable. To date, and I bring up two companies because they're both guest speakers.


[User] Please transcribe this text

[Assistant] Deep IRL is not involved in the most successful robots operating in the real world.


[User] Please transcribe this text

[Assistant] Most of the perception, control and planning, like in this robot.


[User] Please transcribe this text

[Assistant] Does not involve learning approaches, except with minimal addition on the perception side. Best of our knowledge. And certainly the same is true with Waymo, as the speaker on Friday will talk about.


[User] Please transcribe this text

[Assistant] Deep learning is used a little bit in perception on top, but most of the work is done from the sensors and the optimization based, the model based approaches. Trajectory generation and optimizing which trajectory is best to avoid collisions. Deep IRL is not involved.


[User] Please transcribe this text

[Assistant] And coming back and back again, the unexpected loco pock is a higher award, which arise in all of these situations when applied in the real world. So for the cat video, that's pretty short where the cats are ringing the bell and they're learning that the ring of the bell


[User] Please transcribe this text

[Assistant] I urge you to think about how that can evolve over time in unexpected ways that may not have a desirable effect. Where the final reward is in the form of food and the intended effect is to ring the bell.


[User] Please transcribe this text

[Assistant] That's where AI safety comes in. For the artificial general intelligence course in two weeks, that's something we'll explore extensively. It's how


[User] Please transcribe this text

[Assistant] These reinforcement learning, planning algorithms will evolve in ways that are not expected.


[User] Please transcribe this text

[Assistant] And how we can constrain them, how we can design reward functions that result in safe operation.


[User] Please transcribe this text

[Assistant] So I encourage you to come to the talk on Friday at 1 p.m. as a reminder. So 1 p.m. not 7 p.m. in Stata 32123 and to the awesome talks in two weeks from Boston Dynamics to Ray Kurzweil and so on for AGI.


[User] Please transcribe this text

[Assistant] Now tomorrow we'll talk about computer vision and psychfuse. Thank you everybody.


[User] Please transcribe this text

[Assistant] Thank you everyone for braving the cold and the snow to be here.


[User] Please transcribe this text

[Assistant] Which is a set of techniques that have taken a leap in the last decade for our understanding of what artificial intelligence systems are capable of doing. And self-driving cars, which is


[User] Please transcribe this text

[Assistant] In a meaningful, profound way, enter our daily lives in a way that transforms society.


[User] Please transcribe this text

[Assistant] So that's why both of these topics are extremely important and extremely exciting. My name is Lex Friedman, and I'm joined by an amazing team of engineers in Jack Terwilliger, Julia Kindelsberger, Dan Brown, Michael Glazer, Lee Ding, Spencer Dodd, and Benedict Jenick, among many others.


[User] Please transcribe this text

[Assistant] We build autonomous vehicles here at MIT. Not just ones that perceive and move about the environment, but ones that interact, communicate, and earn the trust


[User] Please transcribe this text

[Assistant] And understanding of human beings inside the car, the drivers and the passengers, and the human beings outside the car, the pedestrians and other drivers and cyclists.


[User] Please transcribe this text

[Assistant] The website for this course, selfdrivingcars.mit.edu, if you have questions, email deepcars.mit.edu, slack deep-mit.edu.


[User] Please transcribe this text

[Assistant] For registered MIT students, you have to register on the website, and by midnight, Friday, January 19th,


[User] Please transcribe this text

[Assistant] Build a neural network and submit it to the competition that achieves a speed of 65 miles per hour on the new DeepTraffic 2.0.


[User] Please transcribe this text

[Assistant] It's much harder and much more interesting than last year's for those of you who participated.


[User] Please transcribe this text

[Assistant] There's three competitions in this class. Deep traffic, segfuse, deep crash. There's guest speakers that come from Waymo, Google, Tesla.


[User] Please transcribe this text

[Assistant] And, we have shirts. For those of you who brave the snow and continue to do so, towards the end of the class, there will be free shirts. Yes, I said free and shirts in the same sentence. You should be here. Okay, first, the deep traffic competition. There's a lot of updates and we'll cover those on Wednesday. It's a deep reinforcement learning competition.


[User] Please transcribe this text

[Assistant] Last year, we received over 18,000 submissions. This year, we're going to go bigger.


[User] Please transcribe this text

[Assistant] Not only can you control one car within your network, you can control up to ten. This is multi-agent, deep reinforcement learning. This is super cool.


[User] Please transcribe this text

[Assistant] Second, SegFuse, dynamic driving scene segmentation competition, where you're given the raw video, the kinematics of the vehicle, so the movement of the vehicle, the state of the art segmentation, for the training set you're given


[User] Please transcribe this text

[Assistant] Ground truth labels, pixel level labels, scene segmentation and optical flow. And with those pieces of data, you're tasked to try to perform better than the state of the art in image based segmentation. Why is this critical and fascinating and open research problem? Because


[User] Please transcribe this text

[Assistant] Robots that act in, in this world, in the physical space, not only must interpret, use these deep learning methods to interpret the spatial visual characteristics of a scene, they must also interpret, understand and track the temporal dynamics of the scene. This competition is about temporal propagation of information, not just scene segmentation. You must understand the space and time.


[User] Please transcribe this text

[Assistant] Deep crash, where we use deep reinforcement learning to slam cars thousands of times at, uh, here at MIT at the gym.


[User] Please transcribe this text

[Assistant] You're given data on a thousand runs.


[User] Please transcribe this text

[Assistant] Where a car, where a car knowing nothing is using a monocular camera as a single input, driving over 30 miles an hour, through a scene it has very little control through, very little capability to localize itself, it must act very quickly. In that scene, you're given a thousand runs to learn anything.


[User] Please transcribe this text

[Assistant] We'll discuss this in the coming weeks. This competition will result in four submissions.


[User] Please transcribe this text

[Assistant] That we evaluate everyone's in simulation, but the top four submissions we put head to head at the gym.


[User] Please transcribe this text

[Assistant] And until there is a winner declared, we'll keep slamming cars at 30 miles an hour.


[User] Please transcribe this text

[Assistant] Deep Crash. And also on the website, it's from last year, and on GitHub, there's Deep Tesla.


[User] Please transcribe this text

[Assistant] Which is using the large-scale naturalistic driving data set we have to train a neural network to do end-to-end steering. That takes in monocular video from the forward roadway and produces steering commands, uh, that, uh, steering commands for the car.


[User] Please transcribe this text

[Assistant] Lectures. Today we'll talk about deep learning. Tomorrow we'll talk about autonomous vehicles. Deep RLs on Wednesday.


[User] Please transcribe this text

[Assistant] Driving scene understanding, so segmentation, that's Thursday. On Friday, we have Sasha Arnoux, the Director of Engineering at Waymo.


[User] Please transcribe this text

[Assistant] Waymo is one of the companies that's truly taking huge strides in fully autonomous vehicles. They're taking the fully L4, L5 autonomous vehicle approach and it's fascinating to learn he's also the head of perception for them.


[User] Please transcribe this text

[Assistant] To learn from him what kind of problems they're facing and what kind of approach they're taking on. We have Emilio Frazoli, who one of last year's speakers, Sertesh Karaman, said Emilio is the smartest person he knows. So Emilio Frazoli is the CTO of Netonomy, an autonomous vehicle company that was just acquired by Delphi.


[User] Please transcribe this text

[Assistant] For a large sum of money and they're doing a lot of incredible work in Singapore and here in Boston.


[User] Please transcribe this text

[Assistant] Next Wednesday, we are going to talk about the topic of our research and my personal fascination is deep learning for driver state sensing, understanding the human, perceiving everything about the human being inside the car and outside the car.


[User] Please transcribe this text

[Assistant] One talk I'm really excited about is Oliver Cameron on Thursday.


[User] Please transcribe this text

[Assistant] He is now the CEO of Autonomous Vehicles Startup Voyage. He was previously the director of the self-driving car program for Udacity. He will talk about


[User] Please transcribe this text

[Assistant] How to start a self-driving car company. For those, you said that MIT folks and entrepreneurs, if you want to start one yourself, they'll tell you exactly how. That's super cool. And then Sterling Anderson.


[User] Please transcribe this text

[Assistant] Who was the director previously of Tesla Autopilot team and now is the co-founder of Aurora.


[User] Please transcribe this text

[Assistant] The, the self driving car startup that I mentioned that has now partnered with Nvidia and many others. So why self driving cars?


[User] Please transcribe this text

[Assistant] This class is about applying data-driven learning methods to the problem of autonomous vehicles. Why self-driving cars are fascinating and an interesting problem space.


[User] Please transcribe this text

[Assistant] In my opinion, this is the first wide reaching and profound integration of personal robots in society.


[User] Please transcribe this text

[Assistant] Wide reaching, because there's one billion cars on the road. Even a fraction of that will change.


[User] Please transcribe this text

[Assistant] The face of transportation and how we move about this world.


[User] Please transcribe this text

[Assistant] That takes that his or her life into the hands of an artificial intelligence system.


[User] Please transcribe this text

[Assistant] I showed a few quick clips here. You can Google first time with Tesla Autopilot on YouTube and watch people perform that transfer of control. There's something magical about a human and a robot working together.


[User] Please transcribe this text

[Assistant] That will transform what artificial intelligence is in the 21st century. And this particular autonomous system, AI system, self driving cars is on the scale


[User] Please transcribe this text

[Assistant] And the profound, the life-critical nature of it is profound in a way that it will truly test the capabilities of AI.


[User] Please transcribe this text

[Assistant] There is a personal connection that we argue throughout these lectures that we cannot escape considering the human being.


[User] Please transcribe this text

[Assistant] That autonomous vehicle must not only perceive and control its movement through the environment, it must also perceive everything about the human driver and the passenger, and interact, communicate, and build trust with that driver.


[User] Please transcribe this text

[Assistant] As I will argue throughout this course, an autonomous vehicle is more of a personal robot.


[User] Please transcribe this text

[Assistant] Then it is a perfect perception control system. Because perfect perception and control through this world full of humans.


[User] Please transcribe this text

[Assistant] Is extremely difficult and could be two, three, four decades away. Full autonomy.


[User] Please transcribe this text

[Assistant] Autonomous vehicles are going to be flawed. They're going to have flaws. And we have to design systems that are effectively caught, that effectively transfer control to human beings when they can't handle the situation. And that transfer of control


[User] Please transcribe this text

[Assistant] Is an, is a fascinating opportunity for AI.


[User] Please transcribe this text

[Assistant] Because the obstacle avoidance perception


[User] Please transcribe this text

[Assistant] of obstacles and obstacle avoidance is the easy problem.


[User] Please transcribe this text

[Assistant] It's the safe problem. Going 30 miles an hour navigating through streets of Boston is easy.


[User] Please transcribe this text

[Assistant] Our artificial intelligence systems can't escape human nature. They must work with it.


[User] Please transcribe this text

[Assistant] What's shown here is one of the algorithms we'll talk about next week for cognitive load.


[User] Please transcribe this text

[Assistant] Cognitive load, body pose estimation, drowsiness.


[User] Please transcribe this text

[Assistant] The, the, the movement towards full autonomy is so difficult, I would argue that it almost requires human level intelligence.


[User] Please transcribe this text

[Assistant] That the, as I said, two, three, four decade out journey for artificial intelligence researchers


[User] Please transcribe this text

[Assistant] To achieve full autonomy will require achieving, solving some of the problems, fundamental problems of creating intelligence.


[User] Please transcribe this text

[Assistant] And that's something we'll discuss in much more depth in a broader view in two weeks for the artificial general intelligence course.


[User] Please transcribe this text

[Assistant] But we have Andre Karpathy from Tesla, Ray Kurzweil, Mark Ryberg from Boston Dynamics.


[User] Please transcribe this text

[Assistant] Who asked for the dimensions of this room because he's bringing robots. Nothing else was told to me. It'll be a surprise.


[User] Please transcribe this text

[Assistant] So that is why I argue the human-centered artificial intelligence approach. In every algorithm of design, considers the human.


[User] Please transcribe this text

[Assistant] Seeing, understanding and the control problem, as we'll explore through the competitions and the assignments of this course, can handle 90 and increasing percent of the cases.


[User] Please transcribe this text

[Assistant] But it's the 10, 1, 0.1% of the cases, as we get better and better, that we have to, we're not able to handle through these methods. And that's where the human, perceiving the human is really important. This is the video from last year, of Arc de Triomphe. Thank you, I didn't know it last year, I know now.


[User] Please transcribe this text

[Assistant] Uh, that's, is one of millions of cases where human to human interaction is the, is the dominant driver, not the basic perception control problem.


[User] Please transcribe this text

[Assistant] So why deep learning in this space? Because deep learning is a set of methods that do well from a lot of data.


[User] Please transcribe this text

[Assistant] And to solve these problems where human life is at stake, we have to be able to have techniques that learn from data, learn from real world data. This is the fundamental reality of artificial intelligence systems that operate in the real world. They must learn from real world data.


[User] Please transcribe this text

[Assistant] Whether that's on the left for the perception, the control side, or on the right,


[User] Please transcribe this text

[Assistant] So the human, the perception and the communication interaction and collaboration with the human and the human robot interaction.


[User] Please transcribe this text

[Assistant] It's a set of techniques. If you allow me the definition of intelligence being the ability to accomplish complex goals, then I would argue the definition of understanding may be reasoning.


[User] Please transcribe this text

[Assistant] Is the ability to turn complex information into simple, useful, actionable information. And that is what deep learning does. Deep learning is representation learning. Or feature learning, if you will. It's able to take raw information.


[User] Please transcribe this text

[Assistant] Raw complicated information that's hard to do anything with and construct hierarchical representations of that information to be able to do something interesting with it.


[User] Please transcribe this text

[Assistant] It is the branch of artificial intelligence which is most capable and focused on this task.


[User] Please transcribe this text

[Assistant] Forming representations from data, whether it's supervised or unsupervised, whether it's with the help of humans or not, it's able to construct


[User] Please transcribe this text

[Assistant] Structure, find structure in the data such that you can


[User] Please transcribe this text

[Assistant] Extract simple, useful, actionable information. On the left from Ian Goodfellow's book,


[User] Please transcribe this text

[Assistant] Is the basic example of image classification. The input of the image.


[User] Please transcribe this text

[Assistant] And representing that place in the universe. Whether it's relative to earth or relative to the sun.


[User] Please transcribe this text

[Assistant] Representation matters because what's on the right is much more complicated than what's on the left.


[User] Please transcribe this text

[Assistant] You can think of, in a simple case here, when the task is to draw a line that separates green triangles and blue circles. In the Cartesian coordinate space, on the left, the task is much more difficult and possible to do well. On the right, it's trivial in polar coordinates.


[User] Please transcribe this text

[Assistant] This transformation is exactly what, which we need to learn. This is representation learning.


[User] Please transcribe this text

[Assistant] So you can take the same task of having to draw a line that separates the blue curve and the red curve on the left.


[User] Please transcribe this text

[Assistant] If we draw a straight line, it's going to be a high, there's no way to do it, uh, with zero error, with 100% accuracy. Shown on the right is our best attempt.


[User] Please transcribe this text

[Assistant] The topology, uh, the mapping of the space in such a way in the middle that allows for a straight line to be drawn that separates the blue curve and the red curve.


[User] Please transcribe this text

[Assistant] The learning of the function in the middle is what we're able to achieve with deep learning.


[User] Please transcribe this text

[Assistant] It's taking raw, complicated information and making it simple, actionable, useful.


[User] Please transcribe this text

[Assistant] And the point is that this kind of ability to learn from raw sensor information means that we can do a lot more with a lot more data.


[User] Please transcribe this text

[Assistant] So deep learning gets better with more data.


[User] Please transcribe this text

[Assistant] Where edge cases are everything. This is us driving.


[User] Please transcribe this text

[Assistant] With two perception control systems. One is a Tesla vehicle with autopilot


[User] Please transcribe this text

[Assistant] Version 1 system that's using a monocular camera to perceive the external environment and produce control decisions. And our own neural network running on the Jessin TX2 that's taking in the same with a monocular camera and producing control decisions.


[User] Please transcribe this text

[Assistant] And the two systems argue, and when they disagree, they raise up a flag to say that this is an edge case that needs human intervention.


[User] Please transcribe this text

[Assistant] There is, covering such edge cases using machine learning is the main problem of artificial intelligence in, in, when applied to the real world. It is the main problem to solve.


[User] Please transcribe this text

[Assistant] 476 million synapses. The full human brain is a lot more than that. 100 billion neurons, 1000 trillion synapses.


[User] Please transcribe this text

[Assistant] There's inspirational music with this one that I didn't realize was here. It should make you think.


[User] Please transcribe this text

[Assistant] Artificial Neural Networks. Okay, let's just let it play.


[User] Please transcribe this text

[Assistant] The, the human neural network is a hundred billion neurons, right? One thousand trillion synapses.


[User] Please transcribe this text

[Assistant] One of the state of the net, uh, state of the art neural networks is ResNet 152, which has 16 million synapses.


[User] Please transcribe this text

[Assistant] That's a difference of about a seven order of magnitude difference. The human brains have ten million times more synapses than artificial neural networks.


[User] Please transcribe this text

[Assistant] Plus or minus one order of magnitude, depending on the network.


[User] Please transcribe this text

[Assistant] So what's the difference between a biological neuron and an artificial neuron?


[User] Please transcribe this text

[Assistant] The topology of the human brain have no layers. Neural networks are stacked in layers. They're fixed, for the most part.


[User] Please transcribe this text

[Assistant] There is chaos. Very little structure in our human brain in terms of how neurons are connected. They're connected often to 10,000 plus other neurons. The number of synapses from individual neurons that are input into the neuron is huge.


[User] Please transcribe this text

[Assistant] They're asynchronous. The human brain works asynchronously. Artificial neural networks work synchronously.


[User] Please transcribe this text

[Assistant] The learning algorithm for artificial neural networks, the only one, the best one is backpropagation. And we don't know how human brains learn.


[User] Please transcribe this text

[Assistant] Processing speed. This is one of the...


[User] Please transcribe this text

[Assistant] The only benefits we have with artificial neurons networks is artificial neurons are faster. But they're also extremely power and efficient.


[User] Please transcribe this text

[Assistant] And, there is a division into stages of training and testing with neural networks. Uh, with uh, biological neural networks, as you're sitting here today, they're always learning.


[User] Please transcribe this text

[Assistant] The only profound similarity, the inspiring one, the captivating one, is that both are distributed computation at scale.


[User] Please transcribe this text

[Assistant] There is an emergent aspect to neural networks where the basic element of computation, a neuron, is simple, is extremely simple. But when connected together, beautiful, amazing, powerful approximators can be formed.


[User] Please transcribe this text

[Assistant] A neural network is built up with these computational units, where the inputs, there's a set of edges with weights on them.


[User] Please transcribe this text

[Assistant] The edges are, the weights are multiplied by this input signal. A bias is added with a non-linear function that determines whether the network gets activated or not. Well, the neuron gets activated or not. Visualized here.


[User] Please transcribe this text

[Assistant] And these neurons can be combined in a number of ways. They can form a feed forward neural network, or they can feed back into itself to form, to have state memory.


[User] Please transcribe this text

[Assistant] In recurrent neural networks. The ones on the left.


[User] Please transcribe this text

[Assistant] Are the ones that are most successful for most applications in computer vision.


[User] Please transcribe this text

[Assistant] The ones on the right are very popular and specific when temporal dynamics or dynamics time series of any kind are used. In fact, the ones on the right are much closer to the way our human brains are than the ones on the left. But that's why they're really hard to train.


[User] Please transcribe this text

[Assistant] One beautiful aspect of this emergent power from multiple neurons being connected together is the universal property that with a single hidden layer


[User] Please transcribe this text

[Assistant] These networks can learn any function, learn to approximate any function.


[User] Please transcribe this text

[Assistant] Which is an important property to be aware of. Because...


[User] Please transcribe this text

[Assistant] The limits here are not in the power of the networks. The limits is in the methods by which we construct them and train them.


[User] Please transcribe this text

[Assistant] Now uh, the approaches that essentially memorize patterns in the data.


[User] Please transcribe this text

[Assistant] And approaches that we can loosely say are beginning to reason.


[User] Please transcribe this text

[Assistant] To generalize over the data with minimal human input. On top, on the left, are the quote unquote teachers. It's how much human input in blue is needed to make the method successful. For supervised learning, which is what most of deep learning successes come from, where most of the data is annotated by human beings,


[User] Please transcribe this text

[Assistant] The human is at the core of the success. Most of the data that's part of the training needs to be annotated by human beings.


[User] Please transcribe this text

[Assistant] With some additional successes coming from augmentation methods that extend that.


[User] Please transcribe this text

[Assistant] And the semi-supervised reinforcement learning and unsupervised methods that we'll talk about later in the course, that's where the near term successes we hope are. And with the unsupervised learning approaches, that's where the true excitement about the possibilities of artificial intelligence lie.


[User] Please transcribe this text

[Assistant] Deep learning impact spaces. One is a special purpose intelligence. Is taking a problem, formalizing it, collecting enough data on it, and being able to, uh, solve a particular case that's, that provides value.


[User] Please transcribe this text

[Assistant] Of particular interest here is a network that estimates apartment costs in the Boston area. So you can take the number of bedrooms, the square feet in the neighborhood, and provide as output the estimated cost. On the right is the actual data.


[User] Please transcribe this text

[Assistant] And then there's general purpose intelligence, or something that feels like approaching general purpose intelligence, which is reinforcement and unsupervised learning.


[User] Please transcribe this text

[Assistant] Here with Andre, from Andre Karpathy's Pong the Pixels, a system that takes in 80 by 80 pixel image, and with no other information is able to beat, is able to win at this game. No information except a sequence of images, raw sensory information, the same way, the same kind of information that human beings take in from the visual, audio,


[User] Please transcribe this text

[Assistant] Touch sensory data, the very low level data, and be able to learn to win. In this very simplistic, in this very artificially constructed world, but nevertheless, a world where no feature learning is performed. Only raw sensory information is used to win. With very sparse, minimal human input. We'll talk about that on Wednesday, with Deep Reinforcement Learning.


[User] Please transcribe this text

[Assistant] So, but for now, we'll focus on supervised learning. Where there is input data, there is a network we're trying to train,


[User] Please transcribe this text

[Assistant] A learning system and there's a correct output that's labeled by human beings.


[User] Please transcribe this text

[Assistant] That's the general training process for a neural network. Input data, labels, and the training of that network, uh, that model. So that in a testing stage, a new input data that has never seen before, it's tasked with producing guesses and is evaluated based on that.


[User] Please transcribe this text

[Assistant] For autonomous vehicles, that means being released either in simulation or in the real world to operate.


[User] Please transcribe this text

[Assistant] And how they learn, how neural networks learn, is given the forward pass of taking the input data, whether it's from the training stage.


[User] Please transcribe this text

[Assistant] In the training stage, taking the input data, producing a prediction.


[User] Please transcribe this text

[Assistant] And then given that there's ground truth in the training stage, we can, we can have a measure of error based on a loss function that then punishes the, uh, the synapses, the connections, the parameters that were, uh, involved with making that, uh, that wrong prediction.


[User] Please transcribe this text

[Assistant] Really, you can think of input as being anything. It can be a number, a vector of numbers, a sequence of numbers, a sequence of vector of numbers. Anything you can think of from images to video to audio to text can be represented in this way. And the output can the same be a single number or it can be images, video, text, audio.


[User] Please transcribe this text

[Assistant] One to one mapping on the bottom, one to many, many to one, many to many, and many to many with different starting points for the data. Asynchronous.


[User] Please transcribe this text

[Assistant] Some quick terms that will come up. Deep learning is the same as neural networks.


[User] Please transcribe this text

[Assistant] It's really deep neural networks, large neural networks. It's a subset of machine learning that has been extremely successful in the past decade.


[User] Please transcribe this text

[Assistant] Multi-layer perceptron, deep neural network, recurrent neural network, long short-term memory network, LSTM, convolution neural network, and deep belief networks. All of these will come up through the slides.


[User] Please transcribe this text

[Assistant] And there is specific operations, layers within these networks of convolution, pooling, activation, and backpropagation. This concept that we'll discuss in this class. Activation functions, there's a lot of variance.


[User] Please transcribe this text

[Assistant] On the left is the activation function, the left column. On the x-axis is the input, on the y-axis is the output. The sigmoid function, the output.


[User] Please transcribe this text

[Assistant] If the font is too small, the output is not centered at zero.


[User] Please transcribe this text

[Assistant] For the tanh function, it's centered at zero, but it still suffers from vanishing gradients.


[User] Please transcribe this text

[Assistant] Vanishing gradients is when the value, the input is low or high.


[User] Please transcribe this text

[Assistant] The, uh, the output of the network, as you see in the right column there, the derivative of the function is very low. So the learning rate is very low. For Raylu,


[User] Please transcribe this text

[Assistant] Not, it's also not zero centered, but it does not suffer from vanishing gradients.


[User] Please transcribe this text

[Assistant] Using that loss function then back propagate through to construct a gradient to back propagate the error to the weights that were responsible for making either a correct or an incorrect decision.


[User] Please transcribe this text

[Assistant] So the sub desks of that, there's a forward pass, there's a backward pass, and a fraction of the weight's gradient subtracted from the weight. That's it.


[User] Please transcribe this text

[Assistant] That process is modular, so it's local to each individual neuron, which is why it's extremely, we're able to distribute it across, uh, multiple


[User] Please transcribe this text

[Assistant] Uh, across the GPU. Parallelize across the GPU.


[User] Please transcribe this text

[Assistant] So, learning for neural network.


[User] Please transcribe this text

[Assistant] And all that boils down to is essentially an optimization problem where the objective utility function is the loss function and the goal is to minimize it. And we have to update the parameters, the weights and the synapses and the biases to decrease that loss function.


[User] Please transcribe this text

[Assistant] Depending on the activation functions, different properties, different issues arise. There's vanishing gradients for sigmoid. Where the learning can be slow. There's dying relues.


[User] Please transcribe this text

[Assistant] Where the derivative is exactly zero for inputs less than zero.


[User] Please transcribe this text

[Assistant] There are solutions to this, like leaky railings and a bunch of details that you may discover when you try to win the deep traffic competition. But for the most part, these are the main activation functions. And it's the choice of the neural network designer which one works best.


[User] Please transcribe this text

[Assistant] And stochastic gradient descent without any kind of tricks to it can take a very long time to arrive at the minima.


[User] Please transcribe this text

[Assistant] One of the biggest problems in all of machine learning and certainly deep learning is overfitting.


[User] Please transcribe this text

[Assistant] That is a sine curve. Simple. Fits well.


[User] Please transcribe this text

[Assistant] That it has not seen yet that it has to fit, it's likely to produce a high error. So it's overfitting the training set.


[User] Please transcribe this text

[Assistant] This is a big problem for small datasets.


[User] Please transcribe this text

[Assistant] And so we have to fix that with regularization. Regularization is a set of methodologies that prevent overfitting.


[User] Please transcribe this text

[Assistant] Learning the training too well in order and then to not be able to generalize to the testing stage.


[User] Please transcribe this text

[Assistant] And overfitting, the main symptom, is the error decreases in training set, but increases in test set.


[User] Please transcribe this text

[Assistant] So there's a lot of techniques in traditional machine learning that deal with this and cross validation and so on, but because of the cost of training for neural networks,


[User] Please transcribe this text

[Assistant] It's traditional to use what's called a validation set. So you create a subset of the training.


[User] Please transcribe this text

[Assistant] They use keep away for which you have the ground truth.


[User] Please transcribe this text

[Assistant] And use that as a representative of the testing set. So you,


[User] Please transcribe this text

[Assistant] Perform early stoppage or more realistically just save a checkpoint often.


[User] Please transcribe this text

[Assistant] To see how as the training evolves, the performance changes on the validation set. And so you can stop when the performance in the validation set is getting a lot worse. It means you're overtraining on the training set.


[User] Please transcribe this text

[Assistant] In practice, of course, we run training much longer and see when, uh, what is the best performing, uh, what, what is the best performing snapshot checkpoint of the network.


[User] Please transcribe this text

[Assistant] Dropout is another very powerful regularization technique where we randomly remove part of the network, randomly remove some of the nodes in the network along with its incoming and outgoing edges.


[User] Please transcribe this text

[Assistant] So what that really looks like is a probability of keeping a node. And in many deep learning frameworks today,


[User] Please transcribe this text

[Assistant] It comes with a dropout layer, so it's essentially a probability that's usually greater than 0.5 that a node will be kept.


[User] Please transcribe this text

[Assistant] For the input layer, the probability should be much higher or more effectively what works well is just adding noise. What's the point here? You want to create enough diversity in the training data such that it is generalizable to the testing.


[User] Please transcribe this text

[Assistant] And as you'll see with deep traffic competition, there's L2 and L1 penalty, weight decay, weight penalty. Where...


[User] Please transcribe this text

[Assistant] There's a penalization on the weights that get too large. The L2 penalty keeps the weights small unless the air derivative is huge.


[User] Please transcribe this text

[Assistant] It produces a smoother model and prefers to distribute. When there is two similar inputs, it prefers to put half the weights on each. Distribute the weights as opposed to putting the weight on one of the edges. Makes the network more robust.


[User] Please transcribe this text

[Assistant] One penalty has the one benefit that for really large weights, they're allowed to be, to stay. So it allows for a few weights to remain very large.


[User] Please transcribe this text

[Assistant] These are the regularization techniques. And I wanted to mention them because they're useful to some of the competitions here in the course. And I recommend to go to Playground, to the TensorFlow Playground to play around with some of these parameters.


[User] Please transcribe this text

[Assistant] Where you get to online in the browser, play around with different inputs, different features, different number of layers and regularization techniques, and to build your intuition about classification regression problems given different input data sets.


[User] Please transcribe this text

[Assistant] So what changed? Why over the past many decades neural networks that have gone through two winters are now again dominating the artificial intelligence community? CPUs, GPUs,


[User] Please transcribe this text

[Assistant] ASICs, so computational power has skyrocketed. From Moore's Law to GPUs.


[User] Please transcribe this text

[Assistant] There is huge data set, including ImageNet and others.


[User] Please transcribe this text

[Assistant] There is research, bad propagation in the 80s, uh, uh, the convolutional neural networks, LSTMs. There's been a lot of interesting breakthroughs about how to design these architectures, how to build them such that they're trainable efficiently using GPUs.


[User] Please transcribe this text

[Assistant] There is the software infrastructure from being able to share the data with Git.


[User] Please transcribe this text

[Assistant] To being able to train networks and share code and effectively view neural networks as a stack of layers as opposed to having to implement stuff from scratch with TensorFlow, PyTorch and other, and other deep learning frameworks. And there's huge financial backing from Google, Facebook, and so on.


[User] Please transcribe this text

[Assistant] In order to understand why it works so well and where its limitations are, we need to understand where our own intuition comes from about what is hard and what is easy.


[User] Please transcribe this text

[Assistant] The important thing about computer vision, which is a lot of what this course is about, even as in deep reinforcement learning formulation,


[User] Please transcribe this text

[Assistant] Is that visual perception for us human beings was formed 540 million years ago. That's 540 million years worth of data.


[User] Please transcribe this text

[Assistant] An abstract thought is only formed about a hundred thousand years ago. That's several orders of magnitude less data.


[User] Please transcribe this text

[Assistant] So we can make with neural networks predictions that seem trivial.


[User] Please transcribe this text

[Assistant] The, uh, the trivial to us human beings, but completely


[User] Please transcribe this text

[Assistant] Challenging and wrong to neural networks. Here on the left showing a prediction of a dog with a little bit of distortion and noise added to the image producing the image on the right and neural network is confidently 99% plus accuracy predicting that it's an ostrich.


[User] Please transcribe this text

[Assistant] And there's all these problems to deal with, whether it's in computer vision data, whether it's in text data, audio, all of this variation arises. In vision, it's illumination variability.


[User] Please transcribe this text

[Assistant] The set of pixels and the numbers look completely different depending on the lighting conditions. It's the biggest problem in driving is lighting conditions, lighting variability, pose variation.


[User] Please transcribe this text

[Assistant] Objects need to be learned from every different perspective. I'll discuss that for when sensing the driver. Most of the deep learning work that's done on the face, on the human, is done on the frontal face, or semi-frontal face. There's very little work done on the full 360 pose variability that a human being can take on.


[User] Please transcribe this text

[Assistant] Interclass variability for the classification problem, for the detection problem, there is a lot of different kinds of objects for cats, dogs, cars, bicyclists, pedestrians.


[User] Please transcribe this text

[Assistant] So that brings us to object classification. And I'd like to take you through where deep learning has taken big strides for the past several years leading up to this year, to 2018.


[User] Please transcribe this text

[Assistant] So let's start at object classification. It's when you take a single image and you have to say one class that's most likely to belong in that image.


[User] Please transcribe this text

[Assistant] The most famous variant of that is the ImageNet competition, ImageNet Challenge. ImageNet data set is a data set of 14 million images with 21,000 categories.


[User] Please transcribe this text

[Assistant] And for say the category of fruit, there's a total of 188,000 images of fruit and there is 1200 images of Granny Smith apples. It gives you a sense of what we're talking about here.


[User] Please transcribe this text

[Assistant] So this has been the source of a lot of interesting breakthroughs in deep learning and a lot of the excitement in deep learning. It's first the big successful network, at least


[User] Please transcribe this text

[Assistant] One that became famous in deep learning is AlexNet in 2012 that took a leap of, a significant leap in performance on the ImageNet challenge.


[User] Please transcribe this text

[Assistant] So it was one of the first neural networks that was successfully trained on the GPU and achieved an incredible performance boost over the previous year on the ImageNet Challenge.


[User] Please transcribe this text

[Assistant] The challenge is, and I'll talk about some of these networks, is to, given a single image, give five guesses, and you have five guesses to guess, for one of them to be correct.


[User] Please transcribe this text

[Assistant] The human annotation is a question that often comes up. So how do you know the ground truth?


[User] Please transcribe this text

[Assistant] Human level performance is 5.1% accuracy on this task.


[User] Please transcribe this text

[Assistant] They're not tasked with performing the very high resolution semantic labeling of the image.


[User] Please transcribe this text

[Assistant] And the big transition in 2018 of the ImageNet Challenge leaving Stanford and going to Kaggle.


[User] Please transcribe this text

[Assistant] It's sort of a monumental step because in 2015 with the ResNet network was the first time that the human level performance was exceeded. And I think this is a very important


[User] Please transcribe this text

[Assistant] Map of where deep learning is. For a particular, what I would argue is a toy example despite the fact that it's 14 million images.


[User] Please transcribe this text

[Assistant] So we're developing state-of-the-art techniques here, and the next stage, as we are now exceeding human level performance in this task, is how to take these methods into the real world, to perform scene perception, to perform driver state perception.


[User] Please transcribe this text

[Assistant] In 2016 and 2017, CU Image and SCNET has a very unique new addition to the previous formulations that has achieved an accuracy of 2.2% error.


[User] Please transcribe this text

[Assistant] 2.25% error on the ImageNet Classification Challenge. It's an incredible result.


[User] Please transcribe this text

[Assistant] Okay, so you have this image classification, uh, architecture that takes in a single image and produces convolution, uh, and, uh, takes it through pooling, convolution, and at the end, fully connected layers and performs a classification task or regression task. And you can swap out that layer to perform any kind of, uh, other task.


[User] Please transcribe this text

[Assistant] Image is an input and produce an image is an output. But where the output image, in this case, is a segmentation.


[User] Please transcribe this text

[Assistant] Is where a color indicates what the object is, the category of the object. So it's pixel level segmentation. Every single pixel in the image is assigned a class, a category where that pixel belongs to.


[User] Please transcribe this text

[Assistant] This is the kind of task that's overlaid on top of other sensory information coming from the car in order to perceive the external environment.


[User] Please transcribe this text

[Assistant] You can continue to extract information from images in this way to produce image to image mapping. For example, to colorize images and take from grayscale images to color images.


[User] Please transcribe this text

[Assistant] Or you can use that kind of heatmap information to localize objects in the image.


[User] Please transcribe this text

[Assistant] So as opposed to just classifying that this is an image of a, of a cow, RCNN, fast, and faster RCNN, and a lot of other localization networks.


[User] Please transcribe this text

[Assistant] Allow you to propose different candidates for where exactly the cow is located in this image and thereby being able to perform object detection, not just object classification.


[User] Please transcribe this text

[Assistant] In 2017 there's been a lot of cool applications of these architectures. One of which is background removal.


[User] Please transcribe this text

[Assistant] Again, mapping from image to image, ability to remove background from selfies of humans or human-like


[User] Please transcribe this text

[Assistant] The references with some incredible animations are in the bottom of the slide and the slides are now available online.


[User] Please transcribe this text

[Assistant] There's been a lot of work in GANs and generative adversarial networks, in particular in driving.


[User] Please transcribe this text

[Assistant] Uh, GANs have been used to generate examples that, that generate examples from source data. Whether that's from raw data or in this case with Pix2Pix HD is taking


[User] Please transcribe this text

[Assistant] Images of the forward roadway. This is an exciting


[User] Please transcribe this text

[Assistant] Possibility for being able to generate a variety of cases for self-driving cars, for autonomous vehicles to be able to learn, to generate, to augment the data and be able to change the way different roads look, road conditions, to change the way vehicles look, cyclists, pedestrians.


[User] Please transcribe this text

[Assistant] Then we can move on to recurrent neural networks. Everything I've talked about was one-to-one mapping from image to image or image to number. Recurrent neural networks work with sequences. We can use sequences to generate handwriting,


[User] Please transcribe this text

[Assistant] And combining convolutional neural networks with recurrent neural networks, using convolutional neural networks to extract features frame to frame, and using those extracted features to input into the RNNs to then generate a labeling, a description of what's going on in the video.


[User] Please transcribe this text

[Assistant] A lot of exciting approaches for autonomous systems, especially in drones, where the time to make a decision


[User] Please transcribe this text

[Assistant] Uh, is short. Same with the RC car traveling 30 miles an hour. Attentional mechanisms for steering the attention of the network have been very popular for the localization task and for just saving how much interpretation of the image, how many pixels need to be considered in the classification task.


[User] Please transcribe this text

[Assistant] So we can steer, we can model the way a human being looks around an image to interpret it and use the network to do the same. And we can use that kind of steering to, uh, draw images as well.


[User] Please transcribe this text

[Assistant] Finally, the big breakthroughs in 2017 came from


[User] Please transcribe this text

[Assistant] This, the Pong to Pixels, the reinforcement learning, using sensory data, raw sensory data, and use reinforcement learning methods, deep RL methods, of which we'll talk about on Wednesday. I'm really excited about, uh, the underlying methodology of deep traffic and deep crash is using neural networks


[User] Please transcribe this text

[Assistant] As the approximators inside reinforcement learning approaches. So AlphaGo in 2016 have achieved a monumental task.


[User] Please transcribe this text

[Assistant] That when I first started in Artificial Intelligence was told to me it was impossible. We asked them to accomplish, which is to win at the game of Go against the top human player in the world.


[User] Please transcribe this text

[Assistant] However, that method was trained on human expert positions.


[User] Please transcribe this text

[Assistant] The AlphaGo system was trained on previous games played by human experts. In an incredible accomplishment,


[User] Please transcribe this text

[Assistant] So no knowledge of human experts, no games, no training data, very little human input.


[User] Please transcribe this text

[Assistant] And what more it was able to generate moves that were surprising to human experts.


[User] Please transcribe this text

[Assistant] I think it's Einstein that said that intelligence, that the key mark of intelligence is imagination.


[User] Please transcribe this text

[Assistant] I think it's beautiful to see an artificial intelligence system come up with something that surprises human experts. Truly surprises.


[User] Please transcribe this text

[Assistant] The gambling junkies DeepStack and a few other variants have been used in 2017 to win a heads up poker. Again, another incredible result. I was always told an artificial intelligence would be impossible for deep, uh, for any machine learning method to achieve.


[User] Please transcribe this text

[Assistant] And was able to beat a professional player and several competitors have come along since.


[User] Please transcribe this text

[Assistant] And the drawbacks, it's one of my favorite videos, I show it often, of coast runners. For these deep reinforcement learning approaches, the learning of the reward function, the definition of the reward function controls how the actual system behaves. And this will come


[User] Please transcribe this text

[Assistant] This would be extremely important for us with autonomous vehicles.


[User] Please transcribe this text

[Assistant] Here, the boat is tasked with, uh, gaining the highest number of points. And it figures out that it does not need to race, which is the whole point of the game, in order to gain points, but instead pick up green circles that regenerate themselves over and over. This is the, the counterintuitive behavior of a system that


[User] Please transcribe this text

[Assistant] Would not be expected when you first design the reward function. And this is a very formal, simple system. Nevertheless, it's extremely difficult to come up with a reward function that makes it operate in the way you expect it to operate.


[User] Please transcribe this text

[Assistant] Very applicable for autonomous vehicles. And of course on the perception side, as I mentioned with the ostrich and the dog, a little bit of noise


[User] Please transcribe this text

[Assistant] With 99.6% confidence, we can predict that the noise up top is a robin, a cheetah, a modillo, a lesser panda. These are outputs from actual state-of-the-art neural networks.


[User] Please transcribe this text

[Assistant] It should build our intuition to understand that we don't, that the visual characteristics, the visual, the spatial characteristics of an image do not necessarily convey the level of hierarchy necessary to function in this world.


[User] Please transcribe this text

[Assistant] In a similar way with a dog in an ostrich and everything in an ostrich, a network confidently with a little bit of noise can make the wrong prediction. Thinking a school bus is an ostrich and a speaker is an ostrich.


[User] Please transcribe this text

[Assistant] They're easily fooled. But not really.


[User] Please transcribe this text

[Assistant] Because they performed the task that they were trained to do well. So we have to make sure we keep our intuition.


[User] Please transcribe this text

[Assistant] Optimized to the way machines learn, not the way humans have learned, over the 540 million years of data that we've gained through developing the eye through evolution.


[User] Please transcribe this text

[Assistant] The current challenges we're taking on. First, transfer learning.


[User] Please transcribe this text

[Assistant] There's a lot of success in transfer learning between domains that are very close to each other. So image classification from one domain to the next.


[User] Please transcribe this text

[Assistant] There's a lot of value in forming representations of the way scenes look in order, natural scenes look in order to do scene segmentation in the driving case, for example. But we're not able to do any, any,


[User] Please transcribe this text

[Assistant] Bigger leaps in the way we perform transfer learning.


[User] Please transcribe this text

[Assistant] The biggest challenge for deep learning is to generalize. Generalize across domains.


[User] Please transcribe this text

[Assistant] It lacks the ability to reason in the way that we've defined understanding previously, which is the ability to turn complex information into simple, useful information.


[User] Please transcribe this text

[Assistant] Right now, neural networks are very inefficient. They require big data.


[User] Please transcribe this text

[Assistant] They're not fully automated despite the fact that the feature learning incredibly, the big breakthrough feature learning is performed automatically. You still have to do a lot of design of the actual architecture of the network and all the different hyper parameter tuning needs to be performed. Human input.


[User] Please transcribe this text

[Assistant] Perhaps a little bit more educated human input, in the form of PhD students, postdocs, faculty, is required to hi- tune these hyperparameters. But nevertheless, human input is still necessary. They cannot be left alone. For the most part.


[User] Please transcribe this text

[Assistant] The award, defining the award as we saw with Coastrun is extremely difficult for systems that operate in the real world. Transparency.


[User] Please transcribe this text

[Assistant] Quite possibly is not an important one, but neural networks currently are a black box, for the most part. They're not able to accept through a few successful visualization methods that visualize different aspects of the activations. They're not able to reveal to us humans why they work or where they fail.


[User] Please transcribe this text

[Assistant] And this is a philosophical question for autonomous vehicles. That we may not care as human beings if the system works well enough.


[User] Please transcribe this text

[Assistant] But I would argue that it would be a long time before systems work well enough where we don't care.


[User] Please transcribe this text

[Assistant] We'll care, and we'll have to work together with these systems. And that's where transparency, communication, collaboration is critical. And edge cases, it's all about edge cases. In robotics, in autonomous vehicles,


[User] Please transcribe this text

[Assistant] The 99.9% of driving is really boring. It's the same. Especially highway driving. Traffic driving.


[User] Please transcribe this text

[Assistant] It's the same. The obstacle avoidance, the car following, the lane centering, all of these problems are trivial. It's the edge cases. The trillions of edge cases that need to be generalized over on a very small amount of training data.


[User] Please transcribe this text

[Assistant] I mentioned a bunch of challenges, and this is an opportunity. This is an opportunity to come up with techniques that operate successfully in this world.


[User] Please transcribe this text

[Assistant] So I hope the competitions we present in this class, in the autonomous vehicle domain, will give you some insight and opportunity to apply, in some of these cases, open research problems.


[User] Please transcribe this text

[Assistant] With semantic segmentation of external perception. With control of the vehicle in deep traffic.


[User] Please transcribe this text

[Assistant] And with deep crash of control of the vehicle, uh, in underactuated high speed conditions and the driver's state perception.


[User] Please transcribe this text

[Assistant] So with that, I wanted to introduce deep learning to you today before we get to the fun tomorrow of autonomous vehicles. So we'd like to thank NVIDIA, Google, Autoliv, Toyota,


[User] Please transcribe this text

[Assistant] And, at the risk of setting off people's phones, Amazon Alexa Auto.


[User] Please transcribe this text

[Assistant] By the thousands of messages we received, by the attention, by the 18,000 competition entries, by the many people across the world, not just here at MIT, that are brilliant, that I've got a chance to interact with, and I hope we go bigger and do some impressive stuff in 2018. Thank you very much, and tomorrow is self-driving.


[User] Please transcribe this text

[Assistant] Well, do you think we oughta give some historical background?


[User] Please transcribe this text

[Assistant] But I think we oughta say something about the geographical location, cause not a lot of people know where the islands are.


[User] Please transcribe this text

[Assistant] Then we ought to give an overview of the whole education system.


[User] Please transcribe this text

[Assistant] Well, Dr. White said we shouldn't go into that sort of detail.


[User] Please transcribe this text

[Assistant] Look, let's think about that one later, shall we? Let's see how we're doing for time.


[User] Please transcribe this text

[Assistant] Of course. And then the role of English language.


[User] Please transcribe this text

[Assistant] Alright, so those are the topics we're gonna be, to be covering.


[User] Please transcribe this text

[Assistant] Oh yeaah, of course, the resources room. The islands are gonna be more of a problem.


[User] Please transcribe this text

[Assistant] Oh yeaah, that's right. We can just use the tourist brochure.


[User] Please transcribe this text

[Assistant] We can find out statistics on school places from them as well.


[User] Please transcribe this text

[Assistant] Look Julie, it's almost time for our tutorials. We can meet again on Monday, but we need to prepare some stuff before then.


[User] Please transcribe this text

[Assistant] If you think about most business ideas that turn into business startups, the majority of the ideas and the startups do not succeed. So as a baseline, uh,


[User] Please transcribe this text

[Assistant] You might say the odds are against you. But the flip side of that is...


[User] Please transcribe this text

[Assistant] Look at the world of opportunity that exists. There's always this question about


[User] Please transcribe this text

[Assistant] What's the best road to take? Uh, should I disrupt the category or should I create a new one? I would say, on balance, I generally don't want to be in the pioneering business.


[User] Please transcribe this text

[Assistant] And by that I mean, it's so hard to change consumer behavior.


[User] Please transcribe this text

[Assistant] It takes a long time, a lot of resources, and unless you have the most compelling story, idea, and form factor.


[User] Please transcribe this text

[Assistant] Uh, and I think are willing to run the long race of losing money, which is fine. Uh, I think the easier route is to disrupt a category that already exists. Uh, but there's an opportunity because the large companies feel as if they're entitled to it.


[User] Please transcribe this text

[Assistant] I love that opportunity where the entrepreneur can be disruptive, much more innovative than anyone else. So it's important to define the guardrails of growth. What, what, what size of a business are you contemplating starting or building?


[User] Please transcribe this text

[Assistant] Uhm, and the reason this is important is because not every product


[User] Please transcribe this text

[Assistant] or service or enterprise is a national opportunity.


[User] Please transcribe this text

[Assistant] I'm almost afraid to say this because I don't want a thousand people going out and doing this, but if I was starting a business today, uhm, I've always been fascinated by how much money and


[User] Please transcribe this text

[Assistant] Concern people have for their pets. And I have invested in a couple of pet businesses.


[User] Please transcribe this text

[Assistant] But I think there are unique opportunities in the niche market, which I think is a big market for replicating


[User] Please transcribe this text

[Assistant] What has happened to consumer products in terms of specialtiness and health as it relates to dog and cat food. And I can almost predict not that I'm going to do it.


[User] Please transcribe this text

[Assistant] I can almost predict that there's going to be much higher level concern.


[User] Please transcribe this text

[Assistant] And opportunities from the pet owner to the pet food company matching that category.


[User] Please transcribe this text

[Assistant] And, uh, I think it's better served by an entrepreneur or a small company than a large company that's already in the business. So if you think about specialty foods and what's happened in terms of organic and gluten free, I'm telling you, there's going to be an analog to this in pet food. In a big way. So, but don't go out and do it.


[User] Please transcribe this text

[Assistant] I'm just saying, I think it's an opportunity if it was really thought through. But then you look at, uh,


[User] Please transcribe this text

[Assistant] Another company that I've been fortunate enough to be involved in, uh, which is a shoe company. And the question is, is there any room really


[User] Please transcribe this text

[Assistant] And you also don't want to be in a category where it's massively


[User] Please transcribe this text

[Assistant] Uh, engage with large companies that have great resources and great innovation and you're going to be a single product company. So here's this company that comes along called Allbirds.


[User] Please transcribe this text

[Assistant] Uh, that we happen to be fortunate enough to invest in, in our venture fund called Maveron. And uhm,


[User] Please transcribe this text

[Assistant] The pricing of the shoe, the unique fabric, quality, integrity, and ethos of the company and the way they presented the product and the shoe.


[User] Please transcribe this text

[Assistant] How smart they were in terms of social media.


[User] Please transcribe this text

[Assistant] And, and the way they were able to emotionally engage with the consumer, they broke through. And, but the management team is so deeply, passionately committed to literally walking in the shoes of their customer.


[User] Please transcribe this text

[Assistant] That they have built a great startup, from startup to mid-size company, and I think they're on their way. But on balance, I think you would have probably said, they're gonna have a hard time.


[User] Please transcribe this text

[Assistant] But it does, I think, demonstrate there's always opportunity for a niche player that can look at a large market. I mean, shoes is a large market.


[User] Please transcribe this text

[Assistant] And say, we can create something that is completely, that will completely differentiate itself and will present ourselves in a different way.


[User] Please transcribe this text

[Assistant] And we will have the entrepreneurial nimbleness to do things that a large company cannot do. And we're going to create features and benefits of the product.


[User] Please transcribe this text

[Assistant] And a pricing strategy that will give us a competitive advantage.


[User] Please transcribe this text

[Assistant] And when you meet these people, uh, their love of what they do, their love of the product, love of their people, and their aspiration is infectious. And that's what you want to find. That combination.


[User] Please transcribe this text

[Assistant] I think there are a number of things that a young entrepreneur can do to try and do everything to mitigate and reduce a risk of the idea of failing. Uh, but you can't do everything. So what can you do?


[User] Please transcribe this text

[Assistant] Uh, first off, uh, I would be a sponge and talk to every peer that you can find that has started a business and both succeeded and failed. What happened? What were the surprises?


[User] Please transcribe this text

[Assistant] What should I expect? Because there will be things that you cannot prepare yourself for. And kind of build a reference guide.


[User] Please transcribe this text

[Assistant] And when I'm talking about being best student, you should have a guidebook.


[User] Please transcribe this text

[Assistant] On everything that company has done and what has succeeded, what has failed.


[User] Please transcribe this text

[Assistant] What is their resources? And the reason is you want to do everything you can to understand the mentality of, of those companies. And also, if your idea has merit, what can you prepare yourself for on the chessboard that is going to come as a response? Uh, because it's very easy for these companies to come after you.


[User] Please transcribe this text

[Assistant] How would you lose me? Come on, froggy.


[User] Please transcribe this text

[Assistant] You are a way better performer than Mammon ever was. And that's just facts. It's, it's hard, you know?


[User] Please transcribe this text

[Assistant] And I love you too, Fizzerali. And I would.


[User] Please transcribe this text

[Assistant] Next I'd like to invite Professor Hiroshiaka Yamakawa to give his presentation.


[User] Please transcribe this text

[Assistant] We are running late, so we can take about two questions from the audience.


[User] Please transcribe this text

[Assistant] The discussion is very down to earth, and you have raised good points. Uh, the title of our symposium is Superhumanity, but we have talked about labor and work a lot. And we have talked about how humanity can design itself.


[User] Please transcribe this text

[Assistant] Uh, we designed something for the future, not just referencing the past. With regard to labor, Professor Chin,


[User] Please transcribe this text

[Assistant] Gave us different definitions of labor. The relationship between labor and play.


[User] Please transcribe this text

[Assistant] And, uh, Swandon, uh, the labor that Swandon is mentioning and labor that is defined by Professor Chin is completely different.


[User] Please transcribe this text

[Assistant] We have inserted the word design in it, and for seminars like this, uh, I think we have to think about what the participants and audience would like to hear. We have, uh, people from, uh,


[User] Please transcribe this text

[Assistant] The art sector, we have philosophers here, and so I think most of the people here would like to know how this connects with fine arts, but you have been talking about very technical and specific subjects, so it was a bit hard to follow, I think, for most of us.


[User] Please transcribe this text

[Assistant] And, uh, the, I'm obsessed with the fourth industrial revolution and autonomous driving and autonomous cars. What are we going to do in the future?


[User] Please transcribe this text

[Assistant] Is what I'm obsessing about. And in the 1980s...


[User] Please transcribe this text

[Assistant] Uh, we talked about technology and we talked about man-machine systems for the first time. And, uh,


[User] Please transcribe this text

[Assistant] Experts, uh, in the academia, uh, in the humanities field, uh, they talked about, uh, making the system. But now everything has changed. The man-machine system that we thought about in the 1980s


[User] Please transcribe this text

[Assistant] Is now completely differently seen, uh, and we talk about automation technology and how to upgrade it.


[User] Please transcribe this text

[Assistant] And, uh, artificial intelligence, intelligence becoming automated. We never thought about that in the 1980s. It was the beginning of the computers, uh, then.


[User] Please transcribe this text

[Assistant] So I had a lot of fun following what you were discussing, but what I want to talk about and mention is design. I would like to talk about design. None of you have mentioned design when you talked, but it's something that's going to be there in the future.


[User] Please transcribe this text

[Assistant] Industrial Revolution, James Watt developed the steam engine starting the Industrial Revolution, and then 50 years later, Carnot, uh, announces the Carnot Cycle. It's a theory, and based on that theory, uh, we were able to develop, uh, different technologies.


[User] Please transcribe this text

[Assistant] I think, uh, it's important to talk about what existed in the past, because this is a philosophical debate, but, uh,


[User] Please transcribe this text

[Assistant] Uh, in the following seminars, I'm sure you'll talk about it more, but I'd like to, I'd like you to talk more about how we are going to design the future because, uh, the problem of labor is very important for Korea. I mean, uh, Korea could succeed or fail


[User] Please transcribe this text

[Assistant] Depending on what it does to its labor system. So I would like to ask you to give us more insights on the future, on how we can design the future, more than talking about the past. In any case, it was a lot of fun just listening to you. Thank you.


[User] Please transcribe this text

[Assistant] I'm sorry, but, uh, we have, uh, completely run out of time. So, uh, if you have questions, please ask them tomorrow. And, uh,


[User] Please transcribe this text

[Assistant] Uh, Professor Yuki talked about political engagement and, uh,


[User] Please transcribe this text

[Assistant] How humans can design themselves, uh, when asking this question, uh, let's try to answer this question tomorrow.


[User] Please transcribe this text

[Assistant] We will end this session now. Thank you very much.


[User] Please transcribe this text

[Assistant] I would like to thank all the speakers and, uh, I would like to thank, uh, Ms. Lee and, uh, our, uh, moderators.


[User] Please transcribe this text

[Assistant] And uh, we have ended our program today. I'd like to thank all of you for staying with us until the end.


[User] Please transcribe this text

[Assistant] The Supermarket. Written by Catalina Page. Illustrated by Nora Voutis.


[User] Please transcribe this text

[Assistant] Colorful Eggs. Written by Katie Anderson. Illustrated by Luciana Navarro Powell.


[User] Please transcribe this text

[Assistant] Today we have a new pupil, Peppa Pig. Now Peppa, run and join your friends.


[User] Please transcribe this text

[Assistant] Where are they going to use the Allali? Because Khan had this game off to a very rough start. Multiple times, he was the target, trying to get that Ornn into a more favorable matchup, because this is a very difficult matchup for Ornn. You can see that Khan continued to push let me in despite getting the kills.


[User] Please transcribe this text

[Assistant] And this is why you watch the LPL. We have the best mechanical players in the world. We will always grab to refined moments like that and they do not hesitate that when it comes down, when it's a clutch moment, they prove that they are the best in China and now the best in the world.


[User] Please transcribe this text

[Assistant] Yeah, hot take. I think the LPL will be the strongest region in the world in three years. I didn't think it would be today. I suspected that Kingzone might walk away with it, that LCK will still show such a formidable force at Worlds, but ultimately with the LDL, 20 teams underneath us in a Challenger scene,