[User] Please transcribe this text

[Assistant] Yes, he shouted back in his deepest voice. We're coming. We are?


[User] Please transcribe this text

[Assistant] Annie asked with surprise. A volcano, get it? said Jack. A volcano is a burning mountain.


[User] Please transcribe this text

[Assistant] Oh, right, said Annie. Like in the rhyme, the burning mountain got it. Quick, cover up your face, said Jack.


[User] Please transcribe this text

[Assistant] Jack and Annie covered their eyes with their goggles and pulled up their face masks. Then they started walking toward the bus. Try not to talk to anyone unless we have to, Jack said quickly. And if we do, talk in a really deep voice.


[User] Please transcribe this text

[Assistant] Uhm, maybe you shouldn't talk at all, said Jack. Hurry! Nancy called to them.


[User] Please transcribe this text

[Assistant] By the time Jack and Annie arrived at the bus everyone but Nancy had climbed aboard.


[User] Please transcribe this text

[Assistant] She bounded up the steep steps of the red bus and sat in the driver's seat.


[User] Please transcribe this text

[Assistant] Without a word, Jack and Annie climbed on after Nancy. Walking down the aisle Jack glanced at the others in the group. A couple of them nodded and he nodded back.


[User] Please transcribe this text

[Assistant] Everyone was hidden behind goggles, ski masks, and bulky parkas. Jack couldn't tell anyone's age, or even whether they were male or female.


[User] Please transcribe this text

[Assistant] As the bus's giant wheels moved slowly over the gravel road, Jack looked out the window. The sun shone brightly on the wide snowy fields. Glittering ice crystals blew in the wind. All the world around them seemed to be sparkling. How's everyone doing? Nancy called over her shoulder. Are you all happy campers? Everyone, including Jack and Annie, nodded.


[User] Please transcribe this text

[Assistant] Good, I like travelers who don't complain, Nancy joked. So far so good, Jack thought. No one seemed to suspect they were kids. We have a short ride, said Nancy.


[User] Please transcribe this text

[Assistant] I know you all come from different countries as researchers and journalists, said Nancy. Start from the front and tell me who you are. The woman in front pulled down her face mask. I'm Lucy Banks, she said.


[User] Please transcribe this text

[Assistant] I'm an American and I'm a space scientist. I'm writing a paper on the use of robots on the crater of Mount Erebus. Hopefully this will help our work on Mars someday. Oh brother, thought Jack.


[User] Please transcribe this text

[Assistant] What can we say? That we're Jack and Annie from Frog Creek and we've come to Antarctica to find the fourth secret of happiness to save Merlin the Magician and Camelot? Wonderful, Lucy, Nancy said.


[User] Please transcribe this text

[Assistant] Antarctica is as similar to Mars as any place you can find on Earth. Next?


[User] Please transcribe this text

[Assistant] Ali Khan, biologist from Turkey, said the man sitting behind Lucy Banks. I'm researching heat resistant bacteria in the crater of Mount Erebus. Quick, think, thought Jack. Very good, said Nancy. Next.


[User] Please transcribe this text

[Assistant] Tony Sars from Sydney, Australia, another man said. I'm a travel writer for the Sydney Morning Herald. He held up a notebook. Good, said Nancy.


[User] Please transcribe this text

[Assistant] Without taking off his mask, Jack shouted in a deep voice, Frog Creek Times USA. He held up his notebook. Story about Antarctica. She's, uh. Annie held up her camera. His photographer, she called in a deep voice.


[User] Please transcribe this text

[Assistant] Jack opened his notebook and got ready to write down the rules. Never rush, said Nancy. You should always think about where you're going and what you're doing. Jack scribbled, go slow.


[User] Please transcribe this text

[Assistant] Never walk on snow and ice feels alone, said Nancy. In many places beneath the snow there are deep hidden cracks in the ice.


[User] Please transcribe this text

[Assistant] Jack wrote, Stay with others, cracks and ice.


[User] Please transcribe this text

[Assistant] And remember, all of Antarctica is a nature preserve, said Nancy. Never, ever touch or disturb the wildlife. Oops, said Annie. Jack frowned. We really broke the rules with those penguins, he whispered. I know, but we won't do it again, Annie whispered back. Right, said Jack. He wrote down, never touch wildlife. Got all that? Nancy asked the group. Everyone nodded.


[User] Please transcribe this text

[Assistant] I look forward to sharing Antarctica with you today. I know you'll all find some great information and stories.


[User] Please transcribe this text

[Assistant] We're getting away with this, said Jack. He could hardly believe it. It reminds me of our last mission.


[User] Please transcribe this text

[Assistant] Annie whispered, on the ship with the ocean scientists. This is better, said Jack.


[User] Please transcribe this text

[Assistant] Here everyone's treating us like grownups, and I don't feel like throwing up.


[User] Please transcribe this text

[Assistant] He pulled the rhyme out of his pocket, and he and Annie read it silently. For the final secret you must go To a burning mountain of ice and snow, On wheels by air, then all fall down, Till you come to the cave with the ancient crown, Then speed to Camelot by close of day, Lest grief take Merlin forever away.


[User] Please transcribe this text

[Assistant] I know, but some of the rhyme fits, said Annie. Like you said, the burning mountain of ice and snow is the volcano, Mount Erebus. She pointed out the window of the bus, and there it is.


[User] Please transcribe this text

[Assistant] A white mountain loomed in the distance. Ice and snow covered its slopes and puffs of smoke rose from its peak, drifting into the blue sky. It's burning all right, said Jack. And we're on wheels, said Annie. Yup, said Jack. He looked at the rhyme again. Okay, burning mountain of ice and snow, on wheels,


[User] Please transcribe this text

[Assistant] Touching down onto a flat snowy field was an orange and white helicopter.


[User] Please transcribe this text

[Assistant] The helicopter had skis on the bottom so it could land on the ice and snow. The bus came to a stop at the edge of the field.


[User] Please transcribe this text

[Assistant] We must be going to the volcano in a helicopter, said Annie. By air just like the rhyme says, said Jack.


[User] Please transcribe this text

[Assistant] For those of you getting off the bus now, please remember, she said, the helicopter blades are extremely dangerous. Always wait for a signal from the pilot before approaching the chopper.


[User] Please transcribe this text

[Assistant] Everyone watched as the spinning blade slowly came to a stop. The pilot waved from the helicopter window. Okay, Pete says we can go, said Nancy.


[User] Please transcribe this text

[Assistant] Jack put their rhyme back into his pocket. He pulled on his backpack.


[User] Please transcribe this text

[Assistant] Jack and Annie followed the four other adults up the steps of the helicopter and squeezed into a small cabin. Sitting in two rows of seats behind the pilot, everyone buckled their seatbelts.


[User] Please transcribe this text

[Assistant] Nancy pulled the door shut and turned the latch. Then she sat next to Pete and put on a set of headphones.


[User] Please transcribe this text

[Assistant] Headphones on, everyone. They're under your seats, she said. They'll protect your ears from the chopper noise and also serve as a radio so I can talk to you.


[User] Please transcribe this text

[Assistant] Everyone reached under their seats and took out headphones Jack and Annie pulled off their hoods


[User] Please transcribe this text

[Assistant] Without taking off their goggles or face masks, they place the headphones over their ears. The thick pads muffle the sounds around them. Jack heard Nancy's voice over his headphones. Testing one, two, three, can everyone hear me? Everyone nodded. Okay, Pete, take us to Mount Erebus.


[User] Please transcribe this text

[Assistant] As they flew toward the burning mountain of ice and snow, he tried to remember what words came next. He slipped the rhyme out of his pocket and read, You must go to a burning mountain of ice and snow on wheels by air, then all fall down. All fall down? Wait a minute, what does that mean? thought Jack.


[User] Please transcribe this text

[Assistant] Does that mean the chopper falls down? Do we fall out of the chopper?


[User] Please transcribe this text

[Assistant] As these thoughts swirled through Jack's mind, Annie turned and gave him a thumbs up.


[User] Please transcribe this text

[Assistant] He watched anxiously out the window as the chopper approached a bright orange-red circle on top of Mount Erebus.


[User] Please transcribe this text

[Assistant] His temperature is over 1700 degrees Fahrenheit. Can you guys on this side see okay? Pete?


[User] Please transcribe this text

[Assistant] Pete tilted the helicopter to one side, and then the other. Everyone but Jack oohed and awed. Kim Lee and Annie took pictures. Go, thought Jack, before we all fall down.


[User] Please transcribe this text

[Assistant] Okay Pete, that's great, said Nancy. Let's land at the lower field camp now.


[User] Please transcribe this text

[Assistant] The helicopter tilted upright and began moving down the side of the volcanic mountain.


[User] Please transcribe this text

[Assistant] Stay seated till the blades come to a complete stop, said Nancy.


[User] Please transcribe this text

[Assistant] Everyone remained in their seats with their seatbelts and headphones on.


[User] Please transcribe this text

[Assistant] As you know, we'll be driving our snowmobiles up to the summit, said Nancy. Driving a snowmobile can be very dangerous on these steep icy slopes. Please remember everything you learned in your snowmobile training yesterday. As the others nodded, Annie nodded too.


[User] Please transcribe this text

[Assistant] Jack nudged her. They'd never had any training with snowmobiles. Another warning, said Nancy.


[User] Please transcribe this text

[Assistant] I know you've also been training this week to prevent altitude sickness, but still it can be very dangerous. So please let me know if you feel any symptoms. Altitude sickness? Jack wondered. He pulled off his glove again and opened up his backpack.


[User] Please transcribe this text

[Assistant] He took out the research book and looked up altitude sickness in the index. He turned to the right page and read


[User] Please transcribe this text

[Assistant] Altitude sickness, also known as mountain sickness, is caused by a lack of oxygen at great heights.


[User] Please transcribe this text

[Assistant] Symptoms include headaches, dizziness, and shortness of breath.


[User] Please transcribe this text

[Assistant] Okay, happy campers, said Nancy. All clear. Before we drive up to the top, we'll gather in the hut.


[User] Please transcribe this text

[Assistant] Nancy opened the helicopter door. Everyone took off their headphones, undid their seatbelts, and followed her down the steps of the chopper.


[User] Please transcribe this text

[Assistant] Jack was last as he struggled with putting the research book away, getting his glove back on, and then pulling his pack onto his back. What took you so long? Annie asked when he got out of the chopper. Jack just shook his head. Have a safe trip back to the station, Pete, called Nancy. See you later. Pete waved from the window. Then the chopper blade started rotating again. The chopper lifted off the ground and thundered away.


[User] Please transcribe this text

[Assistant] No we can't, said Jack. We can't use the wand just for ourselves, plus we haven't tried our hardest yet. This way gang, called Nancy.


[User] Please transcribe this text

[Assistant] She was ushering everyone into the small orange building. Jack and Annie followed the group inside.


[User] Please transcribe this text

[Assistant] The one-room hut had plastic chairs, a small heater, axes, jugs of water, and shelves with boxes of trail mix.


[User] Please transcribe this text

[Assistant] Have a seat and help yourself to some of the best water in the world. It comes from melted glacier ice.


[User] Please transcribe this text

[Assistant] Everyone sat on the plastic chairs. Lucy, Kim, Tony, and Ali poured water into 10 cups and lowered their mass to drink.


[User] Please transcribe this text

[Assistant] Before we take off, I want to warn you again about the snowmobiles, said Nancy.


[User] Please transcribe this text

[Assistant] Don't be afraid to go fast and don't lock up the brakes, said Nancy. Locking your brakes can spell disaster.


[User] Please transcribe this text

[Assistant] And watch out for lava bombs, said Ali, the biologist. Lava bombs, Jack piped. He quickly cleared his throat and deepened his voice. Excuse me, lava bombs? Lava that spits from the crater, said Ali. Spits, repeated Jack.


[User] Please transcribe this text

[Assistant] Like oatmeal that bubbles and spatters out of the pot, said Lucy, the space scientist.


[User] Please transcribe this text

[Assistant] Except those bombs aren't made of oatmeal, said Ali. They're blobs of fiery hot liquid rock. Some of them can be as big as a car. They burn deep holes in the ice and snow. If they hit you, said Tony chuckling, well, just think about it. Jack didn't want to think about it. Seriously, it's pretty neat, said Nancy.


[User] Please transcribe this text

[Assistant] For millions of years, hot gas and lava have carved deep hidden places beneath the surface ice of these slopes. Caverns and towers, no one knows all the secrets of Mount Erebus. Nancy took a last sip of water and put her cup down.


[User] Please transcribe this text

[Assistant] Okay guys, you have a few hours to do your experiments and get your stories, then Pete will come back for us. Let's go! As Jack stood up to go, he almost fell over. The room was spinning. He closed his eyes, but that only made it worse. After the others had filed out the door, Jack sat back down on his chair.


[User] Please transcribe this text

[Assistant] Try taking off your mask and goggles so you can breathe better. She helped Jack lift off his mask and goggles. Does that help? Yeah, a little. Jack took a deep breath. But we need lots of help. What do you mean? Said Annie.


[User] Please transcribe this text

[Assistant] Help getting over altitude sickness, said Jack.


[User] Please transcribe this text

[Assistant] Help driving snowmobiles. Help dodging lava bombs.


[User] Please transcribe this text

[Assistant] Help finding an ancient crown. And what is that anyway? An ancient crown? Nancy poked her head back into the hut. What's up, guys? You coming? Yikes, said Annie.


[User] Please transcribe this text

[Assistant] Annie, give it up, said Jack. We're caught. Oh, okay, said Annie. She pushed up her goggles and pulled down her mask.


[User] Please transcribe this text

[Assistant] I'm Annie, and he's Jack. We're sister and brother. I'm going to have a heart attack, said Nancy. What are you kids doing here? We were looking for... Annie stopped.


[User] Please transcribe this text

[Assistant] I've got to get you two back to the station. Your parents must be frantic. I can't believe this.


[User] Please transcribe this text

[Assistant] No, no, it's our fault. No one will blame you, said Jack. But Nancy was pulling out a small radio. Come in, Pete, come in. The radio crackled and popped. Then Pete's voice came through. Hear ya, Nancy, he said.


[User] Please transcribe this text

[Assistant] Pete, you need to get the chopper back here quick and pick up two of my group. They're just little kids. Not so little, thought Jack. Repeat that, Nancy, said Pete.


[User] Please transcribe this text

[Assistant] Two little kids came with a group. Nancy shouted. I didn't know they were kids. Oh, it's too hard to explain. Can you come back right away, please?


[User] Please transcribe this text

[Assistant] Sure thing, said Pete. You head up with the others, I'll pick up the kids.


[User] Please transcribe this text

[Assistant] Thanks, they'll be here at the hut waiting for you to take them back, said Nancy. Over and out.


[User] Please transcribe this text

[Assistant] Nancy put away her radio and looked at Jack and Annie. I don't know how you two got past me. We're sorry, said Annie. This is unbelievable, said Nancy. Jack couldn't believe it either. How did they mess up so badly? I'm so sorry I brought you here, said Nancy.


[User] Please transcribe this text

[Assistant] I've got to lead the group up a safe route to the crater or they'll be in trouble, but Pete should be back here in just a few minutes. Will you be okay by yourselves till then? We'll be fine, don't worry, said Annie. Good, said Nancy. Here, sweeties.


[User] Please transcribe this text

[Assistant] While they drank the water, Nancy spread a blanket on the floor and turned on the small heater. Lie down here, she said. Just rest. She patted the blanket.


[User] Please transcribe this text

[Assistant] Nancy said with a big sigh, You kids nearly gave me a heart attack. She repeated half to herself as she left the hut. Sorry, said Jack. But Nancy was gone.


[User] Please transcribe this text

[Assistant] Jack pulled the rhyme out of his pocket and handed it to Annie. Okay, said Annie. She read aloud. Then all fall down till you come to the cave of the ancient crown. I wonder if this counts as falling down, asked Annie. She put the rhyme into her pocket. I don't think so, said Jack. I don't know what that means.


[User] Please transcribe this text

[Assistant] And there's no ancient crown in Antarctica. It's all science and research and rules and helicopters and snowmobiles. It's the real world. His voice trailed off.


[User] Please transcribe this text

[Assistant] Well, I know one thing. I don't want to waste time lying around here, said Annie. She threw off the blanket and stood up. At least I can take a few pictures while we wait for Pete. You really feel like doing that, said Jack. Not really, but I'm going to try, said Annie. I don't think you should, said Jack.


[User] Please transcribe this text

[Assistant] Don't worry, I'll be back soon, said Annie. Maybe I'll see an ancient crown. Yeah, sure, said Jack.


[User] Please transcribe this text

[Assistant] Annie put on her goggles and ski mask and headed outside. Jack reached into his pack and pulled out their book. He took off his glove and looked up ancient crown in the index. He wasn't surprised to find it wasn't there. Jack put the book back in his pack and took out his notebook. He read over his notes. Go slow. Stay with others. Cracks and ice. Never touch wildlife. Jack's hand was cold so he put his glove back on.


[User] Please transcribe this text

[Assistant] He put away his notebook and then laid his head back down and closed his eyes. He just wanted to sleep. The warmth from the small heater felt good. The sound of the snowmobiles was fading into the distance. As he started to fall asleep, the words of his notes ran through his mind. Go slow, stay with others, cracks and ice. No, thought Jack. He sat straight up. He tossed off the blanket. He threw on his pack and rushed out of the hut.


[User] Please transcribe this text

[Assistant] Jack caught sight of her. She was aiming her camera up the slope at the smoking crater of the mountain.


[User] Please transcribe this text

[Assistant] You have to come back now. He shouted, walking toward her. You shouldn't be walking around by yourself.


[User] Please transcribe this text

[Assistant] Okay, okay. Annie put her camera in her pocket. Come on, said Jack.


[User] Please transcribe this text

[Assistant] He took Annie's hand. They held onto each other and walked through the blowing snow toward the hut.


[User] Please transcribe this text

[Assistant] Remember Nancy's rules, said Jack. There are deep cracks and ahhh.


[User] Please transcribe this text

[Assistant] Before Jack could finish, the ground beneath him gave way, and he and Annie crashed through a thin layer of snow, hiding a deep crack in the ice.


[User] Please transcribe this text

[Assistant] Jack and Annie landed on an icy ledge. Clumps of snow fell on top of them. Silence filled the air.


[User] Please transcribe this text

[Assistant] A thin shaft of light came from the opening they had fallen through. It was at least ten feet above them. You okay? Jack said. I think so, said Annie. They both sat up slowly. Annie peered over the edge of the ledge. Uh oh, she said. Look. Jack looked. He and Annie were on the ledge of a ravine.


[User] Please transcribe this text

[Assistant] That plunged thousands of feet down into darkness.


[User] Please transcribe this text

[Assistant] This must be one of those hidden places in the mountain Nancy talked about, said Jack. The ones made by the lava and hot gases. It's incredible, said Annie. She reached into her pocket for her camera. As soon as Annie moved, Jack heard the ice crack. Don't move, he said. Annie froze. Forget pictures, said Jack. We're facing serious danger here.


[User] Please transcribe this text

[Assistant] If we move, the ice might break under us and we'll fall thousands of feet. Got it, said Annie. She took a deep breath. Maybe we should use the wand. We can't, said Jack. The wand won't work. We can only use it for the good of others, not just ourselves. Darn, said Annie. They were both still for a moment, listening to the immense silence around them. Okay, said Annie. The way I see it,


[User] Please transcribe this text

[Assistant] If we don't use the wand, we'll be stuck here forever. Soon we'll make the wrong move and fall. Right, said Jack. So we'll never find the secret of happiness from Merlin, said Annie. Merlin will fade away completely from sorrow and Camelot will lose his magic forever. Right, said Jack.


[User] Please transcribe this text

[Assistant] So maybe in this case, rescuing ourselves isn't just our good, said Annie. Our good is also the good of others, like Merlin. Good thinking, said Jack. Let's try it. He carefully twisted around and took off his backpack. Then he very slowly reached inside and pulled out the wand of Dianthus. Okay, five words, Jack whispered. I guess I'll just wish for it to save you and me and Merlin.


[User] Please transcribe this text

[Assistant] Said Jack. He turned to put the wand away. I guess the rules must crack. The ice broke. The ledge gave way. Ahhhhhhh.


[User] Please transcribe this text

[Assistant] Call Jack and Annie as they fell through the twilight, down through darkness, down, down, down, down into blackness.


[User] Please transcribe this text

[Assistant] Clutching the wand, Jack lay in the pitch dark. He pushed up his goggles, but he still couldn't see anything. You there? It was Annie's voice. Yes, said Jack. You okay? Yes, but we're really in trouble now, said Jack. We fell way down into a dark hole and the wand doesn't work. He struggled to sit up. Well, maybe we should try again, said Annie. What's the point?


[User] Please transcribe this text

[Assistant] The eyes under them was gliding silently and smoothly through the darkness. What's happening, said Jack. Maybe we're not in a hole, said Annie. Look, there's light ahead. In the distance, they saw a glimmer of light. As they kept moving, the light grew brighter.


[User] Please transcribe this text

[Assistant] It grew brighter and brighter until they slipped out of a dark tunnel into dazzling light. Now Jack could see they were lying on a slab of ice, floating down a narrow river. We're on an ice raft, said Annie. What's happening, said Jack again. I don't know, said Annie, but I think maybe the wand did work.


[User] Please transcribe this text

[Assistant] The ice raft floated through light and shadow, past tall frozen cliffs, then it glided toward an archway in one of the cliff walls. Where are we going? asked Jack.


[User] Please transcribe this text

[Assistant] The raft floated through the archway into a gigantic cavern. The cavern was like a huge, icy cathedral, its walls shown as if they were covered with silver. Oh, whispered Annie. What is this place? Said Jack. I don't know, but now I'm sure the wand worked, said Annie.


[User] Please transcribe this text

[Assistant] They looked just like the Emperor penguins Jack and Annie had seen earlier. Jack and Annie stood up so they could see better. As their raft floated toward the ledge, the penguin in front stepped back into the cave. The other didn't move. Man, said Jack. The penguin wore a glittering crown. The ancient crown, Jack whispered, we found it.


[User] Please transcribe this text

[Assistant] Annie stepped off the ice raft onto the ledge and followed the Emperor into the cave. Jack put the Wand of Dianthus into his pack.


[User] Please transcribe this text

[Assistant] He leapt from the raft onto the ledge, then hurried into the cave. The cave of the ancient crown was sparkling. Icicles hung everywhere, pillars of ice glimmered with blue light.


[User] Please transcribe this text

[Assistant] Strange music, unlike any Jack had ever heard before.


[User] Please transcribe this text

[Assistant] It sounded like the music of a thousand icicle wind chimes. The emperor led Jack and Annie around the pillars.


[User] Please transcribe this text

[Assistant] Penguin couples were dancing to the strange music on a glittering ice rink.


[User] Please transcribe this text

[Assistant] Pink and blue lights spilled over the dancers as they gracefully glided and turned.


[User] Please transcribe this text

[Assistant] Some penguins danced silently with their eyes closed. Others touched beaks.


[User] Please transcribe this text

[Assistant] A group of baby penguins stands together, hopping and sliding on the ice. How, how could this be happening? stammered Jack. Don't ask, whispered Annie. It's magic.


[User] Please transcribe this text

[Assistant] Though the strange music kept playing, all the dancers came to a stop. None of the penguins seemed alarmed though.


[User] Please transcribe this text

[Assistant] They all looked at their visitors with calm, friendly faces.


[User] Please transcribe this text

[Assistant] He'd forgotten all about rescuing the small penguin from the petrol. You may stay with us for as long as you wish, said the emperor. You are honored members of our tribe now. Thank you, said Annie, but we can't stay long.


[User] Please transcribe this text

[Assistant] We came to Antarctica to look for a secret of happiness. To save one of our friends, said Jack. It felt natural to tell the emperor and his tribe the truth. They lived in a world as magical as the world of Camelot. His name is Merlin. He's the magician of Camelot, and he's really sad. The penguins began murmuring and whispering to one another. Jack couldn't understand what they were saying. From the back of the crowd, a baby penguin waddled forward.


[User] Please transcribe this text

[Assistant] It was even smaller than the young penguin they had saved from the petrol. The little penguin waddled up to the emperor. Peep, peep, she said. The emperor looked down at her. Peep, peep. She says she wishes to go with you, said the emperor. She wants to help your friend. But she's so little, said Annie. What about her mom and dad? She is an orphan, said the emperor.


[User] Please transcribe this text

[Assistant] Her parents were lost in a terrible storm, but she is very brave and full of joy.


[User] Please transcribe this text

[Assistant] Then she leaned over and patted the penguin's tiny head. Oh, touch her Jack, she's so soft. Jack patted the baby on the head too.


[User] Please transcribe this text

[Assistant] As he touched her soft little feathers, she tilted her head and stared at him with her big eyes.


[User] Please transcribe this text

[Assistant] Jack felt a wave of tenderness for the baby penguin. He couldn't believe she was an orphan.


[User] Please transcribe this text

[Assistant] Jack couldn't believe he'd just named the little penguin Penny. Usually Annie was the one who named the animals. Peep, said Penny. She would like you to pick her up, said the Emperor. Oh, okay, said Jack. He leaned over and held out his arms. Penny snuggled close to his parka. Peep, Jack laughed and picked Penny up. He held the little penguin tightly. The Emperor turned to the crowd.


[User] Please transcribe this text

[Assistant] The icicle chime music grew softer and softer as Jack and Annie followed the emperor through the cave. The tall penguin led them outside onto the ledge. Thank you for your help today, he said. Thank you for letting Penny come with us, said Annie.


[User] Please transcribe this text

[Assistant] We promised to take good care of her. We'll take her to Merlin, said Jack.


[User] Please transcribe this text

[Assistant] The emperor touched the orphan's downy head with the tip of his wing.


[User] Please transcribe this text

[Assistant] Silently he bowed to them, and they bowed back.


[User] Please transcribe this text

[Assistant] Then the emperor penguin turned and walked back into the cave of the ancient crown.


[User] Please transcribe this text

[Assistant] Jack sighed. For a moment he hated to leave the enchanted world of the penguins. Oh, she's cold, Jack, said Annie. Jack looked down. Penny was shivering in his arms. Put her inside your parka, said Annie.


[User] Please transcribe this text

[Assistant] Jack unzipped his parka and carefully placed the little penguin against his sweater. He pulled the zipper back up. He held Penny close to him.


[User] Please transcribe this text

[Assistant] Perfect, said Annie. Just make sure she can breathe okay.


[User] Please transcribe this text

[Assistant] Don't worry, said Jack padding his jacket. I'll take good care of her. Do you think we can use the wand now to get back? asked Annie. Jack nodded. I think so, he said. I can't think of any other way.


[User] Please transcribe this text

[Assistant] Should we tell it to take us to the treehouse? asked Annie. Or Frog Creek or Camelot.


[User] Please transcribe this text

[Assistant] No, we have to go back to Mount Erebus first, said Jack. If Nancy finds out that Pete never picked us up, she really will have a heart attack.


[User] Please transcribe this text

[Assistant] Oh right, said Annie. Then we better put on our masks and goggles again.


[User] Please transcribe this text

[Assistant] Jack pulled up his face mask and pulled down his goggles. Get the wand out of my pack, he said.


[User] Please transcribe this text

[Assistant] Annie reached into Jack's backpack and pulled out the wand of Dianthus. Okay, ready? She said. Jack patted his parka, comforting the baby penguin. Here we go Penny, he said, on a big adventure.


[User] Please transcribe this text

[Assistant] Annie held up the wand, then took a deep breath and said, take us back to Nancy. In less than an instant, Jack and Annie and Penny were on the slope of Mount Erebus.


[User] Please transcribe this text

[Assistant] The snowmobiles and the helicopter were both parked on the slope.


[User] Please transcribe this text

[Assistant] He turned around. Nancy had caught sight of them. She was waving her arms and running toward them.


[User] Please transcribe this text

[Assistant] Hey, you too, Nancy called. She dashed across the snow and threw her arms around Annie, then Jack. Jack held his breath, hoping Penny wouldn't be squashed. Peep, Jack pulled away from Nancy and pretended to cough. He made his coughing end in a sort of peep. The chopper just got here, said Nancy.


[User] Please transcribe this text

[Assistant] Pete told me he was delayed by a snow squall. I freaked out. Where have you been? Are you okay? Don't worry, we're fine, said Annie. Perfect, said Jack.


[User] Please transcribe this text

[Assistant] We don't feel a bit sick or anything. We were just getting some fresh air. We're ready to go now.


[User] Please transcribe this text

[Assistant] Annie grabbed Nancy's arm and walked with her. Did you have fun on the mountain? Annie asked trying to keep Nancy's attention off Jack. What happened? What did you see?


[User] Please transcribe this text

[Assistant] We saw lots of things, but I was worried about you guys the whole time, said Nancy. Your parents must be frantic back at the station. They had their own expedition today, said Annie. They study penguins. Peep, what was that? asked Nancy. Jack did his weird cough again.


[User] Please transcribe this text

[Assistant] Are you sure you're okay, Jack? Nancy called after him. I'm perfect, he said.


[User] Please transcribe this text

[Assistant] So glad you're safe, young man, said Ali, the biologist. He slapped Jack on the back. Penny peeped and Jack coughed.


[User] Please transcribe this text

[Assistant] Sorry you couldn't make it up to the top, said Kim, the photographer.


[User] Please transcribe this text

[Assistant] That's okay, said Annie. We still got a good story. Did you know, said Lucy the space scientist.


[User] Please transcribe this text

[Assistant] Excellent, but you mustn't tell a soul, said Tony the journalist, or one of us will steal it. Tony laughed and the others laughed with him. Okay, we'll keep it a secret, Annie smiled. Pete opened the door of the chopper. Brave little kids first, said Nancy. Oh brother, thought Jack.


[User] Please transcribe this text

[Assistant] I love late spring evenings in Antarctica, Nancy said to the group.


[User] Please transcribe this text

[Assistant] Jack looked out the window. The evening sky was lavender with streaks of pink


[User] Please transcribe this text

[Assistant] This light always reminds me that we're in a different world from the world back home, said Nancy.


[User] Please transcribe this text

[Assistant] Shack and Annie smiled at each other. If only the others knew how many different worlds there really were.


[User] Please transcribe this text

[Assistant] The chopper swept up through the soft light of the cold sky, up the slope of the burning mountain, past the orange-red lake of boiling lava, over white fields of ice and snow, until finally it landed at the heliport where the red bus was waiting.


[User] Please transcribe this text

[Assistant] The chopper blades stopped spinning. Pete gave the signal. Then Jack and Annie followed Nancy and the others out of the helicopter.


[User] Please transcribe this text

[Assistant] Jack held Penny in place under his parka as he boarded the red bus. He sat with Annie near the back. Nancy took the driver seat and started the engine.


[User] Please transcribe this text

[Assistant] As the bus rolled along, Jack peeked inside his parka.


[User] Please transcribe this text

[Assistant] Penny looked up at him. She blinked a few times, as if she was a little worried.


[User] Please transcribe this text

[Assistant] Jack patted her gently until she closed her eyes and fell asleep. Jack kept patting the front of his parka to comfort Penny.


[User] Please transcribe this text

[Assistant] As Penny snuggled close to Jack, none of his worries of the day mattered anymore. His fear of falling into the lava lake, his dread of altitude sickness, his embarrassment at being caught by Nancy, all the cares and confusions of the day were wiped away by his feelings for the orphan penguin.


[User] Please transcribe this text

[Assistant] When the red bus stopped at the station, Jack and Annie followed everyone down the aisle and climbed off. As the others stood talking in the group, Jack and Annie started walking away. We're leaving now, bye Nancy, said Annie. Bye everyone. Thanks for everything, called Jack.


[User] Please transcribe this text

[Assistant] Oh no you don't, said Nancy. She grabbed them both by the sleeves of their parkas. I'm not letting you two out of my sight again. Not until I hand deliver you to your parents. But...


[User] Please transcribe this text

[Assistant] But our parents are still on their penguin expedition, said Annie.


[User] Please transcribe this text

[Assistant] Nancy led Jack and Annie to a building at the edge of the compound. Well, here you are, home safe and sound. Thanks, said Annie.


[User] Please transcribe this text

[Assistant] Bye, said Jack. He was desperate to get away with Penny. Wait, said Nancy. Oh no, what now, thought Jack.


[User] Please transcribe this text

[Assistant] I'm still worried about you guys, said Nancy.


[User] Please transcribe this text

[Assistant] Are your parents really here to study penguins? I want you to tell me the true story now. Annie heaved a sigh. Okay.


[User] Please transcribe this text

[Assistant] The true story is that Jack and I came alone to Antarctica in a magic tree house. Annie, said Jack. But Annie kept talking. It belongs to Morgan Le Fay of Camelot.


[User] Please transcribe this text

[Assistant] Morgan wants us to find the fourth secret of happiness for Merlin the Magician.


[User] Please transcribe this text

[Assistant] See, he's very sad. And as soon as we leave Antarctica, we're headed for Camelot to cheer him up. Nancy just stared at Annie.


[User] Please transcribe this text

[Assistant] Jack held his breath afraid Nancy would finally have her heart attack. But Nancy burst out laughing and shook her head.


[User] Please transcribe this text

[Assistant] They didn't hear us, said Annie. We better go, they'll wonder where we are. Bye Nancy, thanks for everything.


[User] Please transcribe this text

[Assistant] Nancy, you coming with us for coffee? Tony yells, standing by the bus. You should go, Nancy, said Annie. We'll be fine.


[User] Please transcribe this text

[Assistant] Okay, said Nancy sighing. Bye guys, run and catch up with your folks.


[User] Please transcribe this text

[Assistant] Jack coughed his funny cough. And take care of that cough of yours, Jack, said Nancy. Don't worry, I will, said Jack.


[User] Please transcribe this text

[Assistant] Then Jack and Annie took off. They ran behind the building. They stopped and peeked back around the corner. They watched Nancy walk off with Tony and the others. Let's go, said Jack.


[User] Please transcribe this text

[Assistant] Jack put his arms around Penny as they crossed the icy slope and ran to the cliff near the seashore. The treehouse was still there, tucked under the overhang.


[User] Please transcribe this text

[Assistant] Annie climbed in through the window. Jack followed her, careful not to let Penny slip out of his parka.


[User] Please transcribe this text

[Assistant] Annie pulled Morgan's rhyme out of her pocket and read the last part. Then speed to Camelot by close of day, let's grief take Merlin forever away. Let's go, said Jack, speed to Camelot. Annie pointed to the word Camelot and said loudly and clearly, I wish we could go there. A blast of light, a roar of wind, a rumble of thunder, and of course they were there.


[User] Please transcribe this text

[Assistant] Peep! Penny's head was sticking out of Jack's jacket.


[User] Please transcribe this text

[Assistant] And held her up to the window so she could see the grounds of King Arthur's castle.


[User] Please transcribe this text

[Assistant] The tree house had landed in an apple tree in an orchard.


[User] Please transcribe this text

[Assistant] In the near distance, castle towers rose into a late afternoon sky. Several knights on horses were riding away from the castle. Jack! Annie!


[User] Please transcribe this text

[Assistant] Remember, Leonardo da Vinci told us the secret of happiness has to be available to everyone. Not many people can have their own baby penguin. All right, said Annie. Jack, Annie, come down. Teddy and Kathleen call from below. We're coming, said Annie.


[User] Please transcribe this text

[Assistant] Jack put Penny inside his jacket, then carefully climbed down the rope ladder after Annie.


[User] Please transcribe this text

[Assistant] Oh, what did you bring back with you, said Kathleen. Penny was peeking out of Jack's jacket again. A baby penguin for Merlin, said Annie.


[User] Please transcribe this text

[Assistant] Look at her, said Kathleen. She's beautiful.


[User] Please transcribe this text

[Assistant] Then let us hurry to him at once, said Teddy. Come, follow us.


[User] Please transcribe this text

[Assistant] Kathleen and Teddy led the way between the trees to a small wooden cottage at the edge of the orchard. Merlin's in there? asked Jack. It is a garden house he always loved as a child, said Teddy. Morgan thought he would find comfort there, but alas he seems to have found nothing but more grief.


[User] Please transcribe this text

[Assistant] Teddy opened the door to the cottage and ushered Annie and Jack inside. Morgan was sitting by Merlin's bed.


[User] Please transcribe this text

[Assistant] Morgan turned around. She too looked tired, but her face brightened when she saw Jack and Annie. Thank goodness you have come, she said. Annie crossed the room and hugged Morgan. Jack stepped forward with Penny. Look what we brought for Merlin, he whispered. Oh, said Morgan.


[User] Please transcribe this text

[Assistant] The enchantress gently touched the little penguin. She is lovely indeed, she whispered. Thank you for bringing her. Morgan turned back to Merlin. Merlin, she said.


[User] Please transcribe this text

[Assistant] Jack and Annie from Frog Creek are here. They want to talk with you.


[User] Please transcribe this text

[Assistant] Annie stepped forward while Jack stood in the shadows with Penny. Hi Merlin, said Annie. How are you doing?


[User] Please transcribe this text

[Assistant] The magician didn't open his eyes, but he nodded to let Annie know he heard her.


[User] Please transcribe this text

[Assistant] We have four secrets of happiness to share with you, said Annie.


[User] Please transcribe this text

[Assistant] She reached into Jack's backpack. She pulled out the poem that the poet Basho had given them on their trip to old Japan. Listen, here's a poem from a man named Basho, said Annie. She read the short poem. An old pond, a frog jumps in, the sound of water.


[User] Please transcribe this text

[Assistant] The secret of happiness for Basho was that he paid attention to small things in nature, explained Annie. Merlin nodded slowly. Nature, he said in a raspy voice. That's right, said Annie. And we brought you this too. She pulled out the angel picture Leonardo da Vinci had drawn. She held it up to show Merlin.


[User] Please transcribe this text

[Assistant] This is an angel drawn by a great genius named Leonardo da Vinci, said Annie. Merlin opened his eyes and peered at the drawing. Isn't she beautiful? said Annie.


[User] Please transcribe this text

[Assistant] The secret of Leonardo's happiness was that he had great curiosity about everything. Angels, art, noses, feathers, flowers, flying. Every day he felt happy when he learned something new. Merlin nodded as he stared at the angel sketch. Curiosity, he breathed. Yes, and here's the third secret, said Annie.


[User] Please transcribe this text

[Assistant] She reached into Jack's pack and pulled out the Nautilus shell they'd been given on their trip to the deep ocean. A sea creature once lived inside this shell, said Annie.


[User] Please transcribe this text

[Assistant] We learned from an ocean scientist that a secret of happiness is having compassion for all living things, from a tiny shell creature to a giant octopus. Merlin took the shell from Annie.


[User] Please transcribe this text

[Assistant] We don't actually know what the fourth secret of happiness is, but we want to give you something else. Merlin looked at Jack. Jack held up the tiny penguin. Her name is Penny, he said. Merlin just stared at Penny. He looked confused. She's an orphan, said Jack. Her parents were lost in a terrible storm. Merlin frowned. She is very small, he said hoarsely.


[User] Please transcribe this text

[Assistant] The emperor of the cave of the ancient crown wants you to take care of her, said Jack. He says she is very brave and full of joy. Jack set Penny on the floor. Standing alone, the penguin looked tiny and fragile. Go to Merlin, Jack urged Penny.


[User] Please transcribe this text

[Assistant] Penny waddled across the floor to Merlin. Her little body rocked stiffly from side to side as she held at her wings to keep her balance.


[User] Please transcribe this text

[Assistant] At first his laughter seemed more like coughing than laughing, as if he hadn't laughed in a long time. But then his laughter grew louder.


[User] Please transcribe this text

[Assistant] The Emperor of the Cave of the Ancient Crown has sent her to you. He is very wise. Merlin nodded.


[User] Please transcribe this text

[Assistant] The air smells like ripe apples and wood smoke today, he said.


[User] Please transcribe this text

[Assistant] Yes, my old friend, it does, said Morgan. She wiped tears from her eyes. Merlin turned back and looked at Jack and Annie. Thank you for bringing. What did you say her name was? Penny, said Annie. Yes, Penny, to me, said Merlin. And thank you for your other gifts as well. I will never forget the secrets you have shared with me. You're welcome, said Jack and Annie.


[User] Please transcribe this text

[Assistant] Peep. Yes, yes. Merlin murmured to the baby penguin. You will stay with me and we will have a happy time together. Let us go into the orchard and I will show you the rising moon.


[User] Please transcribe this text

[Assistant] The baby penguin took tiny quick steps alongside the magician as they walked into the orchard. The two strolled together between the trees.


[User] Please transcribe this text

[Assistant] Well done, Teddy said softly. Indeed, said Kathleen, smiling at Jack and Annie. Morgan smiled at them too. Thank you for your four secrets, she said. You're welcome, said Annie. But we're not exactly sure what the fourth secret is.


[User] Please transcribe this text

[Assistant] I told Annie I don't think it can be a baby penguin, said Jack, because in our time people aren't allowed to have baby penguins. That is true, said Morgan.


[User] Please transcribe this text

[Assistant] Taking loving care of another can make one very happy she said.


[User] Please transcribe this text

[Assistant] Like the other three secrets, it helps us look outside ourselves. Then we can better see all the gifts the world has to offer.


[User] Please transcribe this text

[Assistant] Yeah, taking care of Penny really took my attention off myself, Jack said. I forgot about a million things I was worried about. I know you will miss her, said Morgan, but I imagine you will see her again someday. So are we going on another mission soon? asked Annie.


[User] Please transcribe this text

[Assistant] Return home and rest first, and we will send for you again, said Morgan.


[User] Please transcribe this text

[Assistant] Can you try to make it really soon? Said Annie. We don't like long rests. Morgan laughed. We will see, she said.


[User] Please transcribe this text

[Assistant] Oh, before we leave, I want to take a picture of you three guys, said Annie. It's for my family project at school. You seem like a family to me. Look this way, please. Annie aimed her camera at Morgan, Teddy, and Kathleen. Smile, she said. What are you doing? asked Teddy. What are you holding?


[User] Please transcribe this text

[Assistant] It's a camera, said Annie. Just smile, say cheese. Cheese? Why cheese? said Teddy. Click. Flash. Got it. Annie put her camera back into her pocket. What was that? What did you do? asked Teddy.


[User] Please transcribe this text

[Assistant] It's hard to explain, said Jack. It's like magic. From our time.


[User] Please transcribe this text

[Assistant] Goodbye now, said Morgan smiling. Have a safe journey home to your family. Thanks, said Jack.


[User] Please transcribe this text

[Assistant] See all you guys again soon, real soon we hope, said Annie. We hope so too, said Teddy. Goodbye, said Kathleen.


[User] Please transcribe this text

[Assistant] Jack and Annie left the garden cottage and hurried back to the tree house. They climbed up the rope ladder. Jack grabbed the Pennsylvania book and opened it to a picture of the Frog Creek woods. Wait! I see Merlin and Penny! Look!


[User] Please transcribe this text

[Assistant] Annie pointed toward Merlin in the distance. He was still walking through the apple orchard with Penny at his side. They look like a little family now too, said Jack.


[User] Please transcribe this text

[Assistant] Yeah, they do. I should take their picture, said Annie.


[User] Please transcribe this text

[Assistant] The wind started to blow, the treehouse started to spin. It spun faster and faster.


[User] Please transcribe this text

[Assistant] Then everything was still. Absolutely still.


[User] Please transcribe this text

[Assistant] We're home, Annie said. That was a great trip. She and Jack were back in the Frog Creek woods.


[User] Please transcribe this text

[Assistant] Yeaah, I hope we see Penny on our next mission, said Jack.


[User] Please transcribe this text

[Assistant] At least we have a picture of them and a picture of Teddy, Kathleen, and Morgan.


[User] Please transcribe this text

[Assistant] All my pictures are gone, said Annie. No Antarctica, no Merlin or Morgan, no penguins. Really? said Jack. Maybe you can't keep the pictures you take on a magical trip. I think you're right, said Annie.


[User] Please transcribe this text

[Assistant] I only have one picture. I took it right before we left home. She held up her camera to show Jack. It was a picture of him. He had a grin on his face.


[User] Please transcribe this text

[Assistant] That's when I saw the streak of light over the woods, said Jack, and I knew that Treehouse was back.


[User] Please transcribe this text

[Assistant] Jack put the Antarctica book on the floor of the treehouse and pulled on his pack. Then he climbed down the rope ladder and he followed. As they started tramping over the fallen leaves, the woods were growing dark. Jack felt a little cold and hungry.


[User] Please transcribe this text

[Assistant] So the fourth secret is take care of someone who needs you, said Annie.


[User] Please transcribe this text

[Assistant] I guess that could mean lots of things, like take care of a sad person, a baby, a puppy, or a new kid in school. Jack nodded. Yep, he said. And maybe it works the other way too.


[User] Please transcribe this text

[Assistant] I think sometimes you can make other people happy by letting them take care of you, Jack said. Oh right, said Annie.


[User] Please transcribe this text

[Assistant] It seems to make mom and dad happy to take care of us. Like when they tell us to wear scarves and gloves, said Jack. And make us dinner, said Annie. And tell us to be home before dark, said Jack. We better hurry, said Annie.


[User] Please transcribe this text

[Assistant] Yep, let's go make mom and dad happy, said Jack laughing. By letting them take care of us, said Annie. The wind shook the tree limbs and leaves fluttered to the ground.


[User] Please transcribe this text

[Assistant] We hope you have enjoyed this unabridged production of Magic Treehouse Merlin Missions, book number 12, Eve of the Emperor Penguin. This program was directed by Lewis Milgram, executive producer Orly Moskowitz.


[User] Please transcribe this text

[Assistant] Text Copyright 2008 by Mary Pope Osborne Production Copyright 2008 Penguin Random House LLC All Rights Reserved


[User] Please transcribe this text

[Assistant] Thank you. Yes, I have found your reservation. Okay, this is your room key. Your room is on the second floor.


[User] Please transcribe this text

[Assistant] When you get off the lift, turn left. Your room is at the end of the corridor on the right side.


[User] Please transcribe this text

[Assistant] Let's say one of the most ancient optimization methods much simpler than in fact the more advanced methods you have already seen in class.


[User] Please transcribe this text

[Assistant] And interestingly, this more ancient method remains the method for training large-scale machine learning systems. And so there's a little bit of history around that. I'm not gonna go too much into the history.


[User] Please transcribe this text

[Assistant] The bottom line which probably, uh, Gil has also mentioned to you in class, that at least for large data science problems, in the end, stuff reduces to solving an optimization problem.


[User] Please transcribe this text

[Assistant] Actually started liking stuff like gradient descent, which was invented by Cauchy back in the day. And this is how I'm writing the abstract problem.


[User] Please transcribe this text

[Assistant] And what I want to see is, okay, is it fitting on the page? This is my implementation in MATLAB of gradient descent.


[User] Please transcribe this text

[Assistant] What really changes in this implementation of gradient descent today is this part. That's it. You've seen gradient descent.


[User] Please transcribe this text

[Assistant] I will become very concrete pretty soon, but abstractly what I want you to look at is the kinds of optimization problems we are solving in machine learning.


[User] Please transcribe this text

[Assistant] And I'll give you very concrete examples of these optimization problems so that you can relate to them better.


[User] Please transcribe this text

[Assistant] But I'm just writing this as the key topic that all the optimization problems that I'm going to talk about today.


[User] Please transcribe this text

[Assistant] They look like that. We're trying to find an x.


[User] Please transcribe this text

[Assistant] Over a cost function, where the cost function can be written as a sum.


[User] Please transcribe this text

[Assistant] In modern day machine learning parlance, these are also called finite sum problems in case you run into that term. And they're just called finite because n is finite here.


[User] Please transcribe this text

[Assistant] In pure optimization theory parlance, n can actually go to infinity and then they are called stochastic optimization problems. Just for terminology, if while searching the internet you run into some such terminology, so you kind of know what it means.


[User] Please transcribe this text

[Assistant] Here is our setup in machine learning. We have a bunch of training data.


[User] Please transcribe this text

[Assistant] Okay, on this slide, I'm calling x1 through xn. These are the training data, the raw features. Later, actually, I stopped writing x for them. I'll write them with the letter a, but hopefully that's okay.


[User] Please transcribe this text

[Assistant] So x1 through xn, these could be just raw images, for instance, in ImageNet or some other image data set. There could be text documents. There could be anything. Y1 through Yn, in classical machine learning, think of them as plus minus one labels, cat, not cat, or in a regression setup, as some real number. So that's our training data. We have d-dimensional raw vectors, n of those,


[User] Please transcribe this text

[Assistant] Either plus minus 1 in a classification setting or a real number in a regression setting.


[User] Please transcribe this text

[Assistant] It's kind of immaterial for my lecture right now. So that's the input.


[User] Please transcribe this text

[Assistant] Whenever anybody says large-scale machine learning, what do we really mean?


[User] Please transcribe this text

[Assistant] So, what does that mean in words? The n is the number of training data points.


[User] Please transcribe this text

[Assistant] So n could be these days, what, million, 10 million, 100 million, depends on how big computers and data sets you got. So n can be huge. D.


[User] Please transcribe this text

[Assistant] That can also be pretty large. Think if X is an image. If it's a megapixel image, wow, D is like a million already.


[User] Please transcribe this text

[Assistant] Somebody like Criteo or Facebook or Google and you're serving web advertisements. D, these are the features.


[User] Please transcribe this text

[Assistant] Could be like in several hundred million, even a billion.


[User] Please transcribe this text

[Assistant] We are interested in thinking of optimization methods for large-scale machine learning that can handle such big DNS.


[User] Please transcribe this text

[Assistant] And this is driving a lot of research also in theoretical computer science, including the search for sub-linear time algorithms and all sorts of data structures and hashing tricks just to deal with these two quantities.


[User] Please transcribe this text

[Assistant] So here is an example, super toy example. And I hope really, you know, that I can squeeze in a little bit of a


[User] Please transcribe this text

[Assistant] Proof later on towards the end. I'll take a vote here in class to see if you are interested. Let's look at the most classic question, least squares regression.


[User] Please transcribe this text

[Assistant] A is a matrix of observations, or sorry, measurements. B are the observations. We're trying to solve AX minus B whole square. Of course, a linear system of equations, the most classical problem in linear algebra can also be written like that, let's say.


[User] Please transcribe this text

[Assistant] Can be expanded, hopefully you are comfortable with this norm.


[User] Please transcribe this text

[Assistant] So x2 square, this is just defined as, that's the definition of that notation, but I'll just write it only once now. Uh, I hope you are fully familiar with that.


[User] Please transcribe this text

[Assistant] So, by expanding that, I managed to write least squares problem in terms of what I call the finite sum, right.


[User] Please transcribe this text

[Assistant] So it's just going over all the rows in A, the end rows, let's say. So that's the least squares, classical least squares problem. It assumes this finite sum form that we care about.


[User] Please transcribe this text

[Assistant] Another random example, something called Lasso. Maybe you have, if anybody of you has played with machine learning or statistics toolkits, you may have seen something called Lasso. Lasso is essentially least squares, but there's another


[User] Please transcribe this text

[Assistant] Uh, simple term at the end. That again looks like f of i.


[User] Please transcribe this text

[Assistant] Support vector machines wants a workhorse of, and there's still a workhorse of people who work with small to medium sized data.


[User] Please transcribe this text

[Assistant] Deep learning stuff requires huge amount of data. If you have small to medium amount of data, logistic regression, support vector machines, trees, etc., these will be your first go-to methods. They're still very widely used.


[User] Please transcribe this text

[Assistant] These problems are again written in terms of a loss over training data. So this again has this awesome format which I'll just now record here.


[User] Please transcribe this text

[Assistant] I may not even need to repeat it. Sometimes I write it with a normalization.


[User] Please transcribe this text

[Assistant] You may wonder at some point why, as that finite sum problem.


[User] Please transcribe this text

[Assistant] And maybe the example that you wanted to see is.


[User] Please transcribe this text

[Assistant] Something like that. So deep neural networks that are very popular these days, they are just yet another example of this finite sum problem. How are they an example of that? So you have n training data points.


[User] Please transcribe this text

[Assistant] There's a neural network loss, like cross entropy or what have you, squared loss, cross entropy, any kind of loss. YIs are the labels, cat, not cat, or whatever, or they may be a multi-class.


[User] Please transcribe this text

[Assistant] Then you have a transfer function called a deep neural network which takes raw images as input and generates a prediction whether this is a dog or not. That whole thing I'm just calling DNN.


[User] Please transcribe this text

[Assistant] So it's a function of AIs, which are the training data. X.


[User] Please transcribe this text

[Assistant] But mathematically it's still just one particular instance of this finite sum problem.


[User] Please transcribe this text

[Assistant] And in people who do a lot of statistics, maximum likelihood estimation.


[User] Please transcribe this text

[Assistant] This is log likelihood over n observations. We want to maximize log likelihood. Once again, just a finite sum.


[User] Please transcribe this text

[Assistant] So pretty much most of the problems that we are interested in machine learning and statistics, when I write them, write them down as an optimization problem, they look like these finite sum problems.


[User] Please transcribe this text

[Assistant] And that's the reason to develop specialized optimization procedures to solve such finite sum problems. And that's where SGD comes in.


[User] Please transcribe this text

[Assistant] Hopefully this iteration is familiar to you, gradient descent, right? Okay.


[User] Please transcribe this text

[Assistant] F of x refers to that entire summation. F sub i of x refers to a single component, okay. So,


[User] Please transcribe this text

[Assistant] If you were to try to solve, that is to minimize this cost function, neural network SVM what have you, using gradient descent, that's what one iteration would look like.


[User] Please transcribe this text

[Assistant] Because it's a finite sum, gradients are linear operators, you know, gradient of the sum is the sum of the gradients, so that's gradient descent for you.


[User] Please transcribe this text

[Assistant] If you put yourself in the shoes of, you know, algorithm designers,


[User] Please transcribe this text

[Assistant] Some things that you may want to think about. What may you not like about this iteration given that


[User] Please transcribe this text

[Assistant] Big N Big D story that I told you. So anybody have any?


[User] Please transcribe this text

[Assistant] Reservations or about drawbacks of this iteration? Any comments?


[User] Please transcribe this text

[Assistant] And there, that is the prime drawback for large scale. That n can be huge. There can be a variety of other drawbacks. Some of those you may have seen previously when people compare, whether to do gradient or to do Newton, et cetera. But for the purpose of today, for finite sums, the big drawback is computing gradient at a single point. There's a subscript x k missing there.


[User] Please transcribe this text

[Assistant] Involves computing the gradient of that entire sum. That sum is huge.


[User] Please transcribe this text

[Assistant] So getting a single gradient to do a single step of gradient descent for a large data set could take you hours or days.


[User] Please transcribe this text

[Assistant] But then, okay, if you identify that drawback, anybody have any ideas how to


[User] Please transcribe this text

[Assistant] Counter that drawback, at least say, purely from an engineering perspective.


[User] Please transcribe this text

[Assistant] Random sample of the full n. So these are all excellent ideas, and hence,


[User] Please transcribe this text

[Assistant] You folks in the class have discovered the most important method for optimizing machine learning problems sitting here in a few moments. Isn't that great?


[User] Please transcribe this text

[Assistant] So the part that is missing is of course to make sense of does this idea work? Why does it work?


[User] Please transcribe this text

[Assistant] So this idea, uh, is really at the heart of stochastic gradient descent.


[User] Please transcribe this text

[Assistant] I, maybe I can show you an example actually that I, uh,


[User] Please transcribe this text

[Assistant] I'll show you a simulation I found on somebody's nice webpage about that.


[User] Please transcribe this text

[Assistant] So exactly your idea, just put in slight mathematical notation that what if at each iteration,


[User] Please transcribe this text

[Assistant] We randomly pick some integer, i, k, out of the n training data points, and we instead just perform this update, right?


[User] Please transcribe this text

[Assistant] So you, instead of using the full gradient,


[User] Please transcribe this text

[Assistant] One iteration is now N times faster. If N were a million or a billion, wow, that's super fast. But why should this work, right?


[User] Please transcribe this text

[Assistant] I mean, I could have done many other things. I could have not done any update and just output the zero vector that would take even lesser time. It's also an idea. It's a bad idea, but it's an idea in a similar league. I could have done a variety of other things. Why would you think that just replacing that sum with just one random example may work?


[User] Please transcribe this text

[Assistant] Let's see a little bit more about that. So of course it's n times faster.


[User] Please transcribe this text

[Assistant] And the key question for us here right now, the scientific question is, does this make sense?


[User] Please transcribe this text

[Assistant] It makes great engineering sense. Does it make algorithmic or mathematical sense?


[User] Please transcribe this text

[Assistant] So this idea of doing stuff in the stochastic manner was actually originally proposed by Robbins and Monroe somewhere I think around 1951. And that's the most advanced method that we are essentially using currently.


[User] Please transcribe this text

[Assistant] So I'll show you that this idea makes sense, but maybe let's first just look at a comparison of SGD with gradient descent in this guy's simulation.


[User] Please transcribe this text

[Assistant] So this is that MATLAB code of gradient descent and


[User] Please transcribe this text

[Assistant] This is just a simulation of gradient descent as you pick a different step size, that gamma in there, you move towards the optimum. If the step size is small,


[User] Please transcribe this text

[Assistant] You make many small steps and you keep making slow progress and you reach there. That's for a well-conditioned problem and an ill-conditioned problem.


[User] Please transcribe this text

[Assistant] Let's hope that this loads correctly. Oh, there's even a picture of Robbins who was a co-discoverer of the stochastic gradient method. There's a nice simulation that instead of making that kind of deterministic descent,


[User] Please transcribe this text

[Assistant] After all gradient descent is called gradient descent. At every step,


[User] Please transcribe this text

[Assistant] It kind of stochastically still seems to be making progress towards the optimum.


[User] Please transcribe this text

[Assistant] Stochastic gradient descent, because it's not using exact gradients, just working with these random examples, it actually is much more sensitive to step sizes.


[User] Please transcribe this text

[Assistant] And you can see as I increase the step size, its behavior, this is like actually full simulation for a toy problem.


[User] Please transcribe this text

[Assistant] So initially, so what I want you to notice is let me go through this, uh, you know, a few times. Keep looking at.


[User] Please transcribe this text

[Assistant] This slider that I'm shifting is just the step size. So let me just remind you in case you forgot the iteration. We are running xk plus one is


[User] Please transcribe this text

[Assistant] Xk minus some eta k, okay, it's called alpha there, times some randomly chosen data point. You compute its gradient. This is SGD, that's what we are running.


[User] Please transcribe this text

[Assistant] And we threw away tons of information. We didn't use the full gradient, we were just using this crude.


[User] Please transcribe this text

[Assistant] There's a zoomed in version also of this later here.


[User] Please transcribe this text

[Assistant] Again, I'll repeat again and then I'll ask you for your observations if you notice some patterns.


[User] Please transcribe this text

[Assistant] There's one more interesting thing happening here, which is a very, very typical thing for SGD and one of the reasons why people love SGD. Let me do that once again briefly.


[User] Please transcribe this text

[Assistant] Okay, this is tiny step size, almost zero, close to zero, it's not exactly zero.


[User] Please transcribe this text

[Assistant] It doesn't look that stochastic, right? But that's kind of obvious from there. If eta k is very tiny, you'll hardly make any move, so things will look very stable.


[User] Please transcribe this text

[Assistant] And in fact, the speed at which stochastic gradient converges is, that's why extremely sensitive to how you pick the step size.


[User] Please transcribe this text

[Assistant] It's still an open research problem to come up with the best way to pick step sizes. Okay, so it's, even though it's simple, it doesn't mean it's trivial.


[User] Please transcribe this text

[Assistant] And as I vary the step size, okay, make some progress.


[User] Please transcribe this text

[Assistant] A more stable progress in the beginning and when it


[User] Please transcribe this text

[Assistant] The amount of fluctuation near the solution is wilder as he noticed back there.


[User] Please transcribe this text

[Assistant] But one very interesting thing is more or less.


[User] Please transcribe this text

[Assistant] Constant, there is more fluctuation also on the outside, but you see that the initial part still seems to be making pretty good progress. And as you come close to the solution, it fluctuates more.


[User] Please transcribe this text

[Assistant] And that is a very principally typical behavior of stochastic gradient descent that in the beginning,


[User] Please transcribe this text

[Assistant] It makes rapid strides, so you may see your training loss decrease, whoop, super fast and then kind of, you know, peter out.


[User] Please transcribe this text

[Assistant] And it's this particular behavior which got people super excited that hey, in machine learning, we are working with all sorts of big data. I just want a quick and dirty progress on my training. I don't care about getting to the best optimum.


[User] Please transcribe this text

[Assistant] Because in machine learning, you don't just care about solving the optimization problem, you actually care about


[User] Please transcribe this text

[Assistant] Finding solutions that work well on unseen data. So that means you don't want to over fit and solve the optimization problem supremely well. So it's great to make rapid initial progress and if after that progress peers out, it's okay. This intuitionistic statements that I'm making


[User] Please transcribe this text

[Assistant] In some nice cases like convex optimization problems, one can mathematically fully quantify these. One can prove theorems to quantify each thing that I said in terms of how close, how fast and so on. We will see a little bit of that.


[User] Please transcribe this text

[Assistant] And this is what really happens to SGD, you know, it makes great initial progress and


[User] Please transcribe this text

[Assistant] Regardless of uh, how you use step sizes,


[User] Please transcribe this text

[Assistant] Close to the optimum it can either get stuck or enter some kind of chaos dynamics or just behave like crazy. So that's typical of SGD.


[User] Please transcribe this text

[Assistant] And let's look at now slight mathematical insight into, uh, roughly why this behavior may happen.


[User] Please transcribe this text

[Assistant] A trivial one-dimensional optimization problem, but it conveys the crux of why this behavior is displayed by stochastic gradient methods that works really well in the beginning.


[User] Please transcribe this text

[Assistant] Okay, so let's look at a simple one dimensional optimization problem.


[User] Please transcribe this text

[Assistant] I'll kind of draw it out maybe on the other side so that people on this side are not disadvantaged. So I'll just draw out a least squares problem.


[User] Please transcribe this text

[Assistant] X is one dimensional. Previously I had AI transpose X. Now AI is also a scalar. So it's just 1D stuff. Everything is 1D.


[User] Please transcribe this text

[Assistant] These are quadratic functions, right? So they look like.


[User] Please transcribe this text

[Assistant] This, corresponding to different i's, there's like some different functions sitting


[User] Please transcribe this text

[Assistant] We know we can actually explicitly compute the solution of that problem, right? You set, so, you set the derivative of f of x to zero, so, uh, you set the gradient of f of x to zero, hopefully, uh, that's easy for you to do.


[User] Please transcribe this text

[Assistant] So if you do that differentiation, we'll get gradient of f of x will be just given by


[User] Please transcribe this text

[Assistant] A i is equal to zero, and you solve that for x, you get x star, the optimum of this least squares problem, right? So we actually know how to solve it pretty easily.


[User] Please transcribe this text

[Assistant] It's a really cool example actually. I got that from textbook by Professor Dimitri Batsekas. Now, a very interesting thing.


[User] Please transcribe this text

[Assistant] We are not going to use the full gradient. We are only going to use the gradients of individual components, right? So what does the minimum of an individual component look like?


[User] Please transcribe this text

[Assistant] Well the minimum of an individual component is attained when we can set this thing to zero.


[User] Please transcribe this text

[Assistant] And that thing becomes zero if you just pick x equal to bi divided by ai, right?


[User] Please transcribe this text

[Assistant] So a single component can be minimized by that choice.


[User] Please transcribe this text

[Assistant] You can do a little bit of arithmetic mean geometric mean type inequalities to draw this picture.


[User] Please transcribe this text

[Assistant] So over all i from 1 through n, this is the minimum value of this ratio, ai by bi. And let's say this is the max value.


[User] Please transcribe this text

[Assistant] So you can verify with some algebra that that solution will lie in this interval.


[User] Please transcribe this text

[Assistant] So let's, so you, you may want to, this is a tiny exercise for you.


[User] Please transcribe this text

[Assistant] Hopefully some of you love inequalities like me, so this is hopefully not such a bad exercise. But you can verify that within this range of the individual means and max is where the combined solution lies. So of course intuitively you would have physics-style thinking, you would have guessed that right away.


[User] Please transcribe this text

[Assistant] So, that means when you're outside, where the individual solutions, let's call this the far out zone.


[User] Please transcribe this text

[Assistant] And also this side is the far out zone. And this region within which the true minimum can lie, you can say, okay, that's the region of confusion.


[User] Please transcribe this text

[Assistant] Why I'm calling it the region of confusion? Because there, by minimizing an individual f i, you're not going to be able to tell what is the combined x star, that's all.


[User] Please transcribe this text

[Assistant] And a very interesting thing happens now, just to gain some mathematical insight into that simulation that I showed you, that if you have a scalar x,


[User] Please transcribe this text

[Assistant] That is outside this region of confusion, which states that if you are far from the region within which an optimum can lie. So you're far away. You've just started out your progress. You made a random initialization.


[User] Please transcribe this text

[Assistant] What happens when you're in that far out region? So if you're in that far out region, you use a stochastic gradient of some i-th component.


[User] Please transcribe this text

[Assistant] So the full gradient will look like that. A stochastic gradient looks like just one component.


[User] Please transcribe this text

[Assistant] And when you're far out outside that min and max regime,


[User] Please transcribe this text

[Assistant] You can check by just, by just looking at it.


[User] Please transcribe this text

[Assistant] A stochastic gradient in the far away regime has exactly the same sign as the full gradient.


[User] Please transcribe this text

[Assistant] What does gradient descent do? It says well walk in the direction of the negative gradient.


[User] Please transcribe this text

[Assistant] From the optimum outside the region of confusion, your stochastic gradient


[User] Please transcribe this text

[Assistant] Has the same sign as the true gradient, maybe in more linear algebra terms.


[User] Please transcribe this text

[Assistant] It makes, you know, it makes an acute angle with your gradient. That means even though a stochastic gradient is not exactly the full gradient,


[User] Please transcribe this text

[Assistant] It has some component in the direction of the true gradient. This is 1D here it is exactly the same sign in multiple dimensions.


[User] Please transcribe this text

[Assistant] This is the idea that it'll have some component in the direction of the true gradient when you're far away.


[User] Please transcribe this text

[Assistant] Which means, if you then use that direction to make an update in that style, you will end up making solid progress. And the beauty is, in the time it would have taken you to do one single iteration of batch gradient descent,


[User] Please transcribe this text

[Assistant] Again, this is, in the 1D case, this is explicit mathematically. In the high D case, this is more intuitive, okay.


[User] Please transcribe this text

[Assistant] Without further assumptions about angles, etc, one can't.


[User] Please transcribe this text

[Assistant] Make such a broad claim, but intuitively this is what's happening on why you see this awesome initial speed.


[User] Please transcribe this text

[Assistant] Some stochastic gradient may have the same sign as the full gradient, some may not. And that's why you can get that crazy fluctuations.


[User] Please transcribe this text

[Assistant] So this simple 1D example kind of exactly shows you what we saw in that picture.


[User] Please transcribe this text

[Assistant] And people really love this initial progress because often we also do early stopping. You know, you train for some time and then you say, okay, I'm done.


[User] Please transcribe this text

[Assistant] So importantly, if you are purely an optimization person, not thinking so much in terms of machine learning, then please keep in mind that stochastic gradient descent or stochastic gradient method is not such a great optimization method.


[User] Please transcribe this text

[Assistant] Because once in the region of confusion, it can just fluctuate all over forever.


[User] Please transcribe this text

[Assistant] And in machine learning, you say, oh, the region of confusion, that's fine. It'll make my method robust. It'll make my neural network training more robust. It'll generalize better, et cetera, et cetera. We like that. So it depends on which, uh, frame of mind you're in. Okay.


[User] Please transcribe this text

[Assistant] So that's, that's the awesome thing about the stochastic gradient method.


[User] Please transcribe this text

[Assistant] I will give you now the key mathematical ideas behind the success of SJD. This was like a little illustration.


[User] Please transcribe this text

[Assistant] This is an idea that records throughout machine learning and throughout theoretical computer science and statistics. Anytime you're faced with the need to compute an expensive quantity, resort to randomization to speed up the computation. SGD is one example. The true gradient was expensive to compute, so we


[User] Please transcribe this text

[Assistant] Create a randomized estimate of the true gradient, and the randomized estimate is much faster to compute.


[User] Please transcribe this text

[Assistant] And mathematically what will start happening is.


[User] Please transcribe this text

[Assistant] Depending on how good your randomized estimate is, your method may or may not converge to the right answer.


[User] Please transcribe this text

[Assistant] So of course one has to be careful about what particular randomized estimate one makes.


[User] Please transcribe this text

[Assistant] So, but really abstractly, even if I hadn't shown you the main idea, this idea you can apply in many other settings.


[User] Please transcribe this text

[Assistant] If you have a difficult quantity, come up with a randomized estimate.


[User] Please transcribe this text

[Assistant] Stochastic gradient descent. It uses stochastic gradients. Stochastic is here just used very loosely, you know, it just means there's some randomization. That's all it means.


[User] Please transcribe this text

[Assistant] And the property, the key property that we have is in expectation, the expectation is over whatever randomness you used. So if you picked, you know, some


[User] Please transcribe this text

[Assistant] Instead of the true gradient, in expectation actually it is the true gradient. So in statistics language this is called


[User] Please transcribe this text

[Assistant] The stochastic gradient that we use is an unbiased estimate of the true gradient.


[User] Please transcribe this text

[Assistant] And this is a very important property in the mathematical analysis of stochastic gradient descent, that it is an unbiased estimate.


[User] Please transcribe this text

[Assistant] And intuitively speaking, anytime you did any proof in class or in the book or lecture notes, wherever, where you were using true gradients,


[User] Please transcribe this text

[Assistant] More or less, you can do those same proofs, more or less, not always, using stochastic gradients by replace, by


[User] Please transcribe this text

[Assistant] So the unbiasedness is great, so it means I can kind of plug in these stochastic gradients in place of the true gradient and I'm still doing something meaningful. So this is answering that earlier question, you know.


[User] Please transcribe this text

[Assistant] Why this random stuff, why should we think it may work? But there's another very important aspect to it, why it works beyond this, uh, unbiasedness.


[User] Please transcribe this text

[Assistant] That the amount of noise, or like the amount of uh, yeaah, the amount of stochasticity is controlled. So just because it is an unbiased estimate,


[User] Please transcribe this text

[Assistant] Doesn't mean that it's gonna work that well. Why? Because it could still fluctuate hugely, right?


[User] Please transcribe this text

[Assistant] Essentially plus infinity here, minus infinity here, you take an average, you get zero, so.


[User] Please transcribe this text

[Assistant] That is essentially unbiased but the fluctuation is gigantic. So whenever talking about estimates, what's the other key quantity we need to care about beyond expectation?


[User] Please transcribe this text

[Assistant] Just this simple statistical point, in fact, is at the heart of a sequence of research progress in the past five years in the field of stochastic gradient where people have worked really hard to come up with newer and newer, fancier and fancier versions of stochastic gradient which


[User] Please transcribe this text

[Assistant] Have the unbiasedness property, but have smaller and smaller variance. And the smaller the variance you have, the better your stochastic gradient is as a replacement of the true gradient.


[User] Please transcribe this text

[Assistant] And of course the better is the replacement of the true gradient, then you truly get that, you know, end time speed up.


[User] Please transcribe this text

[Assistant] Okay, so the speed of convergence depends on how noisy the stochastic gradients are. It seems like I'm going too slow. I won't be able to do a proof. It sucks, but I'll...


[User] Please transcribe this text

[Assistant] Let's, let me actually tell you then about, uh, rather than the, the proof.


[User] Please transcribe this text

[Assistant] I think I'll share the proof with Gil, because the proof that I wanted to actually show you,


[User] Please transcribe this text

[Assistant] Talk- gives a proof of stochastic gradient is well behaved on both convex and non-convex problems. And the proof I wanted to show was for the non-convex case because it applies to neural networks. So you may be curious about that proof.


[User] Please transcribe this text

[Assistant] And remarkably that proof is much simpler than the case of convex problems.


[User] Please transcribe this text

[Assistant] So let me just mention some very important points about stochastic gradient. So even though this method has been around since 1951, every deep learning toolkit has it and we are studying it in class, uh, there are still gaps between what we can say theoretically and what happens in practice.


[User] Please transcribe this text

[Assistant] And I will show you those gaps already and encourage you to think about those if you wish. So, let us look back at our problem and tell you about two variants.


[User] Please transcribe this text

[Assistant] So here are the two variants. Uh, I'm gonna ask if any of you is familiar with these variants, uh, in some way or the other. So, let's, I just called it feasible. Here there are no constraints, so start with any random vector of your choice. In deep network training you have to work harder. And then, this is the iteration you run, right? Option one and option two.


[User] Please transcribe this text

[Assistant] So option one says, that was the idea we had in class, randomly pick some training data point, use its stochastic gradient.


[User] Please transcribe this text

[Assistant] The moment you use the word random, you have to define


[User] Please transcribe this text

[Assistant] You know, just uniformly at random each time you draw a number from 1 through n, use that stochastic gradient, move on. Which means the same point can easily be picked twice also.


[User] Please transcribe this text

[Assistant] And without replacement means if you've picked up point number three, you're not gonna pick it again until you've gone through the entire training data set. Those are two types of randomness.


[User] Please transcribe this text

[Assistant] There is no right or wrong answer to this. I'm just taking a poll. What would you use? Think that you're writing a program for this. And maybe think really pragmatically, practically. That's enough of a hint. Which version would you use? I'm just curious.


[User] Please transcribe this text

[Assistant] Who, and the, uh, the exclusion, the compliment thereof, or I don't know, maybe some people are undecided. Who would use two? Very few people. Ooh, okay.


[User] Please transcribe this text

[Assistant] How many of you use neural network training toolkits like TensorFlow, PyTorch, what not? Which version are they using?


[User] Please transcribe this text

[Assistant] Actually every person in the real world is using version 2. Are you really gonna randomly go through your RAM each time?


[User] Please transcribe this text

[Assistant] Use a pre-shuffle operation and then just whoop, stream through the data. What does streaming through the data mean? Without replacement.


[User] Please transcribe this text

[Assistant] So all the toolkits actually are using the without replacement version.


[User] Please transcribe this text

[Assistant] Even though intuitively the just uniform random feels much nicer.


[User] Please transcribe this text

[Assistant] So, even for this method, everybody studies it, there are million papers on it.


[User] Please transcribe this text

[Assistant] The version that is used in practice is not the version we know how to analyze.


[User] Please transcribe this text

[Assistant] Without replacement means non-IID probability theory and non-IID probability theory is not so easy. That's the answer.


[User] Please transcribe this text

[Assistant] Okay, so the other version is this minibatch idea, which you mentioned really early on, is that


[User] Please transcribe this text

[Assistant] Rather than pick one random point, I'll pick a mini-batch.


[User] Please transcribe this text

[Assistant] So I had a million points, each time instead of picking one, maybe I'll pick ten or hundred or thousand or what have you.


[User] Please transcribe this text

[Assistant] Averages things, averaging things reduces the variance, so this is actually a good thing. Because the more quantities you average, the less noise you have.


[User] Please transcribe this text

[Assistant] That's kind of what happens in probability, right? So we pick a mini-batch.


[User] Please transcribe this text

[Assistant] And the stochastic estimate, now is this, you know, not just a single gradient but averaged over a mini-batch.


[User] Please transcribe this text

[Assistant] So minibatch of size 1 is the pure vanilla SGD. Minibatch of size N is nothing other than pure gradient descent. Something in between is what people actually use.


[User] Please transcribe this text

[Assistant] And again, the theoretical analysis only exists if the mini-batch is picked with replacement, not without replacement. So one of the reasons actually, a very important thing, in theory you don't gain too much in terms of


[User] Please transcribe this text

[Assistant] Or each core or sub core or small chip or what have you depending on your hardware.


[User] Please transcribe this text

[Assistant] Can be working with one stochastic gradient. So mini batches, the larger the mini batch, the more things you can do in parallel.


[User] Please transcribe this text

[Assistant] So mini-batches are greatly exploited by people to, uh, give you a cheap version of parallelism.


[User] Please transcribe this text

[Assistant] Computes a stochastic gradient. So the hard part is not adding these things up and


[User] Please transcribe this text

[Assistant] Making the update to x, the hard part is computing a stochastic gradient. So if you can compute.


[User] Please transcribe this text

[Assistant] Well, you'd love to use very large mini-batches so that you can fully max out on the parallelism available to you, right? Maybe you have a multi-GPU system if you're, I don't know, friends with Nvidia or Google. I only have like two GPUs. But it depends on how many GPUs you have.


[User] Please transcribe this text

[Assistant] You'd like to really max out on parallelism so that you can really crunch through big data sets as fast as possible. But you know what happens with very large mini-batches?


[User] Please transcribe this text

[Assistant] So if you have very large mini-batches, stochastic gradient starts looking more like


[User] Please transcribe this text

[Assistant] That's not a bad thing, that's awesome for optimization.


[User] Please transcribe this text

[Assistant] There is a weird conundrum that happens in training deep neural networks. This type of problem we wouldn't have for convex optimization, but in deep neural networks this really disturbing thing happens that


[User] Please transcribe this text

[Assistant] If you use this very large mini batches, your method starts resembling gradient descent. That means it decreases noise so much so that this region of confusion shrinks so much


[User] Please transcribe this text

[Assistant] Which all sounds good, but it ends up being really bad for machine learning.


[User] Please transcribe this text

[Assistant] That's what I said that in machine learning you want some region of uncertainty and what it means actually is


[User] Please transcribe this text

[Assistant] A lot of people are, have been working on this, including at big companies, that


[User] Please transcribe this text

[Assistant] If you reduce that region of uncertainty too much, you end up overfitting your neural network and then


[User] Please transcribe this text

[Assistant] So parallelism programming optimization theory, big mini-batch is awesome. Unfortunately, there's a price to be paid. It hurts your test error performance. And there are all sorts of methods people are trying to cook up, including


[User] Please transcribe this text

[Assistant] Even though these ideas are simple, you see that every simple idea leads to an entire sub-area of SGD.


[User] Please transcribe this text

[Assistant] So here are practical challenges. People have various heuristics for solving these challenges. You can cook up your own, but it's not that one idea always works. So if you look at SGD, what are the moving parts? The moving parts in SGD?


[User] Please transcribe this text

[Assistant] The gradients, stochastic gradients, the step size, the mini batch. So, how should I pick step sizes? Very non-trivial problem.


[User] Please transcribe this text

[Assistant] Different deep learning toolkits may have different ways of automating that tuning, but it's one of the painful things.


[User] Please transcribe this text

[Assistant] Which mini-batch to use, with replacement without replacement I already showed you. But which mini-batch should I use, how large that should be? Again, not an easy question to answer. How to compute stochastic gradients?


[User] Please transcribe this text

[Assistant] Does anybody know how stochastic gradients are computed for deep network training? Anybody know?


[User] Please transcribe this text

[Assistant] There is a very famous algorithm called backpropagation.


[User] Please transcribe this text

[Assistant] Some, some kind of algorithm which computes for you a single stochastic gradient.


[User] Please transcribe this text

[Assistant] And hence, you know, this uh, TensorFlow, et cetera, these toolkits, they come up with all sorts of ways to.


[User] Please transcribe this text

[Assistant] Automate the computation of a gradient because really that's the main thing.


[User] Please transcribe this text

[Assistant] And then other ideas like gradient clipping and momentum, et cetera. There's a bunch of other ideas. And the theoretical challenges I mentioned to you already.


[User] Please transcribe this text

[Assistant] Proving that it works, that it actually solves what it set out to do. Unfortunately, I was too slow. I couldn't show you the awesome five-line proof that I have that SGD works for neural networks.


[User] Please transcribe this text

[Assistant] And theoretical analysis, as I said, it's really lagging. My proof also uses the with replacement and the without replacement version, which is the one that is actually implemented.


[User] Please transcribe this text

[Assistant] There's very little progress on that. There is some progress. There's a bunch of papers, including from our colleagues at MIT, but it's quite unsolved. And the biggest question which,


[User] Please transcribe this text

[Assistant] Stuff like, why does SGD work so well for neural networks? We use this crappy optimization method.


[User] Please transcribe this text

[Assistant] Very rapidly does some fitting. Data is large, neural network is large.


[User] Please transcribe this text

[Assistant] And then this neural network ends up having great classification performance. Why is that happening? It's called trying to explain, build a theory of generalization. Why does an SGD-trained neural network work better than neural networks trained with more fancy optimization methods?


[User] Please transcribe this text

[Assistant] It's a mystery and most of the people who take interest in theoretical machine learning and statistics, that is one of the mysteries they're trying to understand. So I think that's my story of SGD and this is the part we skip, but it's okay. The intuition behind SGD is much more important than this. So I think we can close.


[User] Please transcribe this text

[Assistant] The following content is provided under a Creative Commons license. Your support will help MIT OpenCourseWare continue to offer high quality educational resources for free.


[User] Please transcribe this text

[Assistant] To make a donation or to view additional materials from hundreds of MIT courses.


[User] Please transcribe this text

[Assistant] Visit MIT OpenCourseWare at ocw.mit.edu.


[User] Please transcribe this text

[Assistant] Where does the Green Knight live? Oh, not far from here.


[User] Please transcribe this text

[Assistant] It's my wedding anniversary. Nick sent me these flowers to celebrate.


[User] Please transcribe this text

[Assistant] We've been married for 20 years, but we've been together for almost 25 years.


[User] Please transcribe this text

[Assistant] Oh yes. We have had many fights over the years.


[User] Please transcribe this text

[Assistant] When you find the right person, you will make it work, Maria.


[User] Please transcribe this text

[Assistant] I think you and Carolyn are wise to get to know each other well. Before tying the knot and walking down the aisle.


[User] Please transcribe this text

[Assistant] I'm so sorry, Maria. It will get better. You deserve to date someone who respects you.


[User] Please transcribe this text

[Assistant] Would you like me to set you up with someone? I have a nephew who would be just your type. I think he would be a perfect match for you. He's cute, funny, and intelligent.


[User] Please transcribe this text

[Assistant] And this person was very forceful and said, look, that's the only way you'll ever do it. So I said, okay, if I have to do it, I had a sabbatical. I said, I'll write about my research if there's no other way, and then I'll get to do my cookbook. So I wrote a...


[User] Please transcribe this text

[Assistant] Yellow. Turns out they're identical. Can anybody see them as identical? Very, very hard. I can cover the rest of the cube up, and if I cover the rest of the cube, you can see that they're identical.