Buuuuugs iiiiin Spaaaaace!
CC Attribution - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Fulton, Colin
Space is really cool. From precision electronics, to giant rockets, spacecraft can represent the best of engineering. But sometimes things go wrong. Terribly wrong. What do exploding Soviet rockets have to do with Agile development? How did LISP and Forth hacking save part of the Galileo probe? What about that time astronauts added life-saving monkey patches... while orbiting the moon! Exactly how much damage can a little dead code do? (Hint: a lot!) This talk is for anyone whose love of a good space story is rivaled only by their passion for incredibly resilient software.
 I think it's about time to get started. If people want to keep filing in there are plenty of seats, go ahead and sit down wherever. No need to say in the back. So my name is Colin Fulton. I'm a front-end developer with Duo Security. If you want to reach me on Twitter, you can hit me up at Peter Kweins. I don't really use Twitter, so if you actually want to reach me personally, just email me at justcolon at gmail.com. If I don't respond, just send an email again. Sometimes I forget to respond to things. And all the slides, as well as a number of other resources, are going to be up on github.com. They're all available now with my github account, Just Colin. And you can go browse those at your wish. Alrighty. So many of you may remember the time that NASA sent a rocket to Mars, and it crashed into Mars. And the reason why it crashed is because they had one piece of software that used pounds per second, while the rest of their software used newtons per second. And this was very embarrassing for NASA, but how many of you have ever done a multi-corporation project which takes years, if not decades, to complete with many people working on specs that are hundreds of pages long and not written a single bug in it? In this case, the bug was pretty bad because they lost a multi-million-dollar satellite, but still, I think we all can relate to this kind of problem. All of you may also remember when we initially launched Hubble into space, the images that it took initially were really blurry. They couldn't do anything with them. And that's because they had actually gotten the geometry of the mirror wrong. So they had to send up a number of astronauts in the space shuttle to go install, effectively, a pair of glasses to correct it. What a lot of people don't know is that they were actually able to measure how blurry the images were, and the glasses not only got them just as good as it would have been if they had initially gotten it right, they actually were able to correct for a number of other things and get sharper images than they would have gotten if they hadn't had this bug. This is an expensive way to discover a way to get a sharper image, but still, it worked. So today I want to tell you four stories about bugs in space programs, because there's a lot that we can learn from bugs, and space programs are really cool, so the bugs are more exciting. I'm going to tell you right up front. I'm not some guru on top of a mountain who can pass out sage advice. I'm going to be editorializing a little bit, but I'm not going to be able to give you grand lessons in this talk. I'm just a workaday developer, like all of you. In fact, many of you here, if not most of you, are probably smarter than me and know more about some of these subjects than me. But I'm going to tell you these stories so that you can kind of reflect on your own experience and what you've done and see what you can get out of them. And I'll provide a couple of little editorializing bits to kind of give you pointers and where I think that's interesting to think about these stories. So many of you have probably heard the phrase, we can send a man to the moon, but, which is usually said to be like, we can send a man to the moon, but we can't get a printer to work well. I don't know if any of you have ever done anything with printer firmware, but it's actually more complicated than you think it would be. And going to the moon was not something that we did perfectly. There were a number of bugs along the way, and this is just one of them. Specifically Apollo 14. And no, this isn't a typo. This isn't Apollo 13, the famous one that had the explosion and the exciting movies. No, no, this was a much more subtle, but still almost dangerous bug. Now, in Apollo 14, they managed to get to the moon and they were orbiting the moon. And it was less than four hours before they were about to land. And the crew on the ground noticed that the abort bit was set in the computer. That's the bit that gets set when you push the abort button, which the computer then checks and says, oh, God, the abort bit is set. We have to abort whatever we're doing and get back into lunar orbit. Now, luckily, they weren't in one of the procedures that would actually check the abort bit yet, and so was ignoring it. But what they ended up discovering was that there was probably a little bit of metal or a little bit of solder inside the abort button that was just floating around and occasionally randomly shorted, which meant the abort button was acting as if it was randomly being pushed every so often. This is fine while you're orbiting the moon, but while you're landing, it means that there is a certain percentage chance that you're just going to have the rocket thruster turn on, whip you around the other way, and send you back into orbit. And you only have enough fuel to land on the moon once because it's really expensive to get all that fuel into space. So this was a major problem, a really serious problem. So they contacted the guys who knew about the Apollo Guidance Computer, the people who had actually written the code for it and said, how can you help us? And so they had about three to four hours to figure out a way to patch the solution using the Apollo Guidance Computer so that it would ignore this particular abort button. They also needed to make sure that there was another way to do an abort. They only had three to four hours to create that patch, test it, send it over to Houston where they would then test it again, and then relay that patch up to the astronauts to manually punch in. Now the problem is the Apollo Guidance Computer really didn't want to ignore that abort button for a number of different reasons. So this is some Apollo Guidance Computer source code. If you've never looked at assembly, this may look a little bit weird to you, but it's actually very, very similar to the assembly programs that are written today. In the first column, optionally, there's a label which you can jump to. So you can think of this as just like a name that you can put there either as documentation or to say elsewhere in your code, hey, once you get to this point, jump over to this other part of the code. Then the next bit is what's called the operator, or sometimes these are known as mnemonics. These are little tiny short strings which represent different commands. In this case, the CA there means clear register and then add on to it. Finally, we have the operator. This is the thing that you're going to operate on. So in this case, we are going to clear register and then add in this particular flag to it which is stored. Then finally, there are comments which, hey, look, they use hash comments just like Ruby. That's pretty cool. So what does this line of code do? The first line clears out a register and then sets it to a word that's full of a bunch of different flag bits. Next they use the mask command to mask out all but the let abort bit, the bit that actually gets used. So now all the other bits will be zero except if that let abort bit is set, that'll be one. Then there's a little bit of magic. They didn't have enough space to get all the commands in so they have to call this magic extend command but we don't have time to go into what that does. Then finally, there's a branch if zero command. So what that'll do is it'll check, is the number that's currently in the register zero? If so, it will go ahead and jump to the land disk there which is another label in the program and if not, it'll continue on to do the abort sequence. So if the bit is set, it'll go ahead and continue with the abort sequence. If the bit isn't set, then they'll go ahead and do it. And so there's this let abort flag stored in memory. So let's just set the let abort flag. This'll be easy. Let's go ahead and set it and then if we set it to zero, then this code will run and an abort isn't going to happen. The problem is it isn't that simple. There are a variety of programs that get run that also change that let abort flag because they want to set that flag during the landing to say, hey, we're in the middle of a landing and that means we may want to abort. And so they would have to set the let abort flag immediately after the computer set it and that would mean that there was a gap of time between when the computer set it and when they manually set it that theoretically that abort button could short out and it would be a problem. So if you also may be thinking that maybe we can just send up a patch for the code that will ignore the abort button, but in this case the code itself is woven with a bunch of metal wires by a bunch of women guided by a computer in Massachusetts. So if we want to patch the code, we would have to actually weave a bunch of wires together, package it up in epoxy, then send it up in a rocket and send it back to the moon and you can't really do that in four hours. So that's why we can't edit this code. It's very, very much read only, more so than we have to deal with today. So the solution that they came up with has to do with the next line of code that we have here. You'll see that immediately after we check the abort flag, we check that modreg there and that checks if we are in program 71. Program 71 is one of the two abort programs. The other is program 70. If the computer senses that it's already an abort sequence, it's not going to go into another abort sequence because why would you abort an abort? So instead of setting the abort flag, we can set modreg. If we do that, then we're going to go ahead and trick the computer that we're already in an abort, but it'll continue doing whatever it's doing. This will solve the problem, except it's a little bit more complicated than that because there are other programs that check this particular register to see what program they're in in order to figure out what to do during the landing. So the sequence that they came up with was this. First you set the modreg to program 71. That tells the computer that you're already in an abort sequence, so it's going to ignore abort after you set that. Then you have to wait until the ignition starts, which is going to automatically set the let-abort flag, but we don't care about that because we already changed the modreg register. Then you're going to push the throttle up to full because the computer isn't going to do that because it thinks it's in the middle of an abort sequence. Then you're going to set the zoom flag, which lets the computer know that, hey, in the future, you can go ahead and throttle up the rocket because, again, it's checking these flags and we're kind of really screwing with the way the program is supposed to work. After that, you can reset the let-abort flag now that it's safe to go and set it back to zero because it isn't going to be set again. That'll, again, cause it to ignore the abort button. Then we can go ahead and set the mode register back to program 63, the program that it thinks that it really should be in, and that way it'll go ahead and check the zoom flag and throttle up the engine as it should for landing. Then they can go ahead and turn the throttle back down and let the computer take control. If you're confused, don't worry. It's a little bit confusing and convoluted, but all this put together meant that they were able to safely ignore that abort button. In future flights, they actually set up additional routines to allow them to do this in a slightly less convoluted way in case this happened again in the future. Not about you, but I think monkey patching is really great when you need it. You can fix amazing problems like this. If they didn't have the ability to manually go in and edit all the registers in memory while the program is running, they would have been screwed. They may have aborted midway through the flight. However, monkey patching will save you in situations like this, but you only should really use it when you need it, which is almost never. I don't know about you, but I'm not landing people on the moon terribly often. Most of the time I have more than four hours to fix a bug, so take it as you will. The next thing I want to talk about is how much damage one line of code can do. The line of code was in the Ariane 5 rocket, which is one of three heavy lifting rockets in the world. These are rockets which can lift tens of thousands of pounds into space. They are big, they are expensive, they are complicated, and we don't have that many of them because it's not that often that we're throwing things that big into space, but every so often they're needed. Flight 501 was to be the first flight of the rocket. The European space union had spent pretty much about a decade developing this rocket in total. They had spent a lot of time carefully testing all their systems and making sure that this would work because this project in total cost them about $7 billion. So if something went wrong, that's a big loss. These rockets are disposable, they don't reuse them every single time, but their first launch was going to be an actual mission launching satellites into space. So if this mission failed, not only would they lose the rocket, they would also lose all the satellites that they were trying to launch into space. So Flight 501, initially the launch is going okay. Everything is relatively nominal. There are a couple little hiccups that happen, but this is normal with a new rocket. Now less than a minute into the launch, the thrusters on the rocket suddenly gimbled to one direction and that caused the rocket to turn really suddenly. Now rockets have a pointy end and you want to keep that pointy end going into the wind because that's good because then the wind will just go around it. If the wind starts beating against to the side, the rocket is going to fall apart. And so the rocket did start to disintegrate and there's a video of this. The rocket you can see bits of the rocket coming off from all the wind hitting it. And at this point the rocket exploded, which generally is a bad thing. However, in this case, it's a good thing because when a rocket like this is in a dangerous situation, it has a self-destruct sequence. So it can just blow up. In this case, the rocket detected that it was falling apart and so it intentionally exploded to prevent even worse things from happening. It's bad when a bunch of bits of rockets fall to the ground from an exploded rocket. It's even worse when a rocket that is flying flies into the ground. Now the line of code that failed is this one. Now there are a couple problems here. I don't know Ada code and this is written in Ada and even though this is relatively easy to read, they didn't use the most intuitive variable or function names. I assume that if you worked on this rocket, this makes perfect sense to you. But the line that went wrong is this bit right here. And in this bit, they are taking a 64-bit floating point number and converting it into a 16-bit integer. 64 is bigger than 16. And they weren't checking whether or not overflowed. This isn't like Ruby where you can just add arbitrary numbers together and it will magically just figure out how to get those big numbers for you. Now luckily, they had actually detected all of the possible overflows in their system because they are rocket engineers. They are doing actual rocket science so they knew that overflows were possible. And they had fixed almost all of the overflows. And the reason why they couldn't fix all of them is because they had made a rule that their computer couldn't work harder than 80% of its capability because this is a relatively resource constrained system so if they work it too hard, it's going to cause problems and then crash. Now the reason why they didn't put the checks around this bit of code is pretty interesting. They actually tested it with real data, which is a good thing, except they tested it with real data from the Arian 4 rocket, which was a different rocket, which had a different flight profile. So in the Arian 4 rocket, this particular bit of code would never overflow, except in the Arian 5 rocket, the horizontal velocity, which is what this particular number represents, was much larger and thus it overflowed. Now luckily for them, they had two copies of the computer running this and it was only the backup that failed first with this overflow. The problem is they're both getting the same data, so immediately after, just milliseconds after the backup failed, the other system failed and so they had both redundant systems go down. Now a team was formed to investigate exactly what happened and what went wrong and they got a number of different conclusions out of this. One is that the computer itself was assuming, or I'm sorry, one is that they were protecting against random faults, that's why you have two computers. If one happens to randomly fail due to a bad capacitor or a bad solder joint, you still have the other computer and it can take over. They weren't protecting against design faults, they weren't protecting against the crazy idea that maybe there's a bug in their code and something went wrong. The committee that was looking at it instructed that they should really just assume that all faults where is soft, is faulty until it is correct. If you do test-driven development, this is what you do. You assume that your code is bad until you have a test that can prove that it's right. The first thing you do is create a test that can prove that that code is right. Then as you write it, you can check that. The other issue is that the computer had assumed that everything would work and thus when it had this error, it just stopped working and sent out an error message. That error message was sent to the rocket Gimbal's, which interpreted it as a movement command and thus it went over and blew up. The next thing that they said is that you should focus not only on developing good code but good documentation because a lot of the investigation that they did and actually figure out all these bugs and exactly what happened wouldn't be possible if it wasn't for all the documentation about their process that they provided. The final recommendation was to remove dead code. This is kind of the zinger with the story. That code was to calibrate the gyroscopes inside the rocket to make sure that they knew which direction was up. That program would run when the rocket was sitting on the launch pad so it was constantly calibrating those gyroscopes so when it launched, it knew what direction up was. It was still running after the launch happened. This doesn't quite make sense because after the launch happens, the rocket is going to be turning so you don't want to calibrate which way is up while you're actually turning the rocket. The reason why they kept it running in the Ariane 4 rocket, the previous rocket, was a very clever hack. They wanted to make sure that if the launch was extended for some reason, they had to delay when the launch was, they were still running that calibration sequence. They could get it up and running faster. In this case, this dead code caused a rocket to blow up. They said, okay, clever hacks are all fine and good but remove dead code. It isn't doing anything. Your clever hack is very nice but it caused a rocket to explode. There's a little post-script. The first launch of the Ariane 5 rocket was obviously not so good but a lot of the subsequent launches were quite good. In fact, even through to this year, they are currently at a record of 81 successful launches in a row. They've had about four launches total in a history of almost 100 launches. This is a really successful rocket and they did a really good job with it. And after doing this investigation, they fixed a lot of the problems in their systems. The next thing I want to talk to you about is agile development and agile development in the Soviet Union. We all know that the US won the space race and we know the US won the space race, won because we're Americans, at least most of us are, and we know that America is the best. We put the first people on the moon and we have the first people orbit the moon before that. Therefore, we won the space race. The problem with this is that it ignores the rest of the space race. We did accomplish that, we definitely did do that. The USSR did a couple of things before us. They actually did all of the things before us. They beat us at every step of the space race, which is why we didn't feel so good throughout it. So the question is, why is it that the USSR never managed to send a person to the moon? We got to the moon and they beat us every step of the way, so surely they were better than us. The answer is that getting to the moon is hard, really hard. One of the hard problems is you need to launch a tremendously large rocket because not only do you need to get into space, you need to get to the moon. So you need a rocket big enough to get from low Earth orbit to the moon. And you need to get that rocket into space, which means you need the largest rocket that humanity has ever constructed. Now the engine powering that rocket for the US was the F1 engine. This thing is an amazing feat of engineering. To give you an idea of how big and powerful this rocket was, it took in fuel at a rate such that the pumps that it used could drain an average size bathtub in about 200 milliseconds. That's a lot of fuel. And not only did they have this engine that was sucking in all this fuel, they had five of them on the first stage. Five of these engines. Look at how big these things are. That's actually a relatively tall man standing next to them. And look at just how ginormous this thing is. This is an amazing feat of engineering. And the USSR lacked the industrial capabilities of the United States. They weren't able to produce engines that big and that powerful. They just didn't have the industry capable of doing it. So they decided to go a different tact. They designed the NK33 engine. Now they knew they couldn't go big, so instead they went efficient. They decided to design a rocket engine which had as good thrust to weight ratio as possible. And because it was lighter but still delivered a lot of thrust, they could use more of them. It wasn't as powerful, that's why they had to use more of them. And specifically, they used 33 of them. That's a lot of rocket engines to put on the bottom of your rocket. And all of those have to work and work every single time. So not only did they have to make a very efficient engine, they also had to make a relatively simple and reliable engine. Now when the US was designing their F1 engine, they did a lot of design work. They were very careful in their designs that when they launched the rocket, they were relatively sure that it was going to work. It didn't work all of the time, but they had to be really cautious because the press was constantly watching them. And any failure was another example of how the US was losing the space race. The Soviet Union on the other hand decided to take a more hardware-centered approach. They'd still do lots of design work, of course, but they were more willing to launch a rocket and just see what happens. If it blew up, you could learn from that explosion, improve the rocket, and iterate, much like we do with agile software development. And so they blew up a lot of rockets. What's okay about this from the Soviet Union's perspective is that they didn't have the free press of the United States, so a lot of their explosions were hidden. And because they were doing them out in the middle of, I believe, Kazakhstan, there weren't really any people watching. And so all those explosions could be much better hidden. We didn't find out about a lot of them until much later. Now the NK-33 engine was designed to replace the previous engines used on the N1 rocket, the equivalent of the Saturn V rocket, the rocket that was going to take the USSR to the moon. The first four launches blew up, which is okay because they designed the NK-33 engine to be a more efficient, better engine, and it was going to be used on the fifth launch in order to get them to the moon. The problem is the fifth launch was canceled because at this point the US had already gotten to the moon. And so the Union wasn't interested in going to the moon anymore. And so the Soviet government ordered that all the NK-33 engines, which were tested but never actually used on a rocket, should be destroyed. The engineers didn't do that, though, because these were really good engines. They knew they were good engines, that they had something special here. So they just hid them away in a warehouse, and they didn't let the government know about this. And it wasn't until the 1990s that it was revealed that they had the stockpile of old engines from the 1960s. And so they got in contact with some American rocket engineers from a company and said, hey, we have this really cool rocket that we think you might be interested in. And when they started to tell the US engineers about the numbers, the US engineers didn't believe it, because this was an engine that had a thrust to weight ratio significantly better than anything that they had managed to design in all the subsequent decades. Not only that, it used something called a closed cycle technology. Now for those of you who aren't rocket nerds, I can't really go into the details of what this is, but it's an engineering feat so impressive that the US had deemed it impossible, because it involves powering those pumps that pump all the fuel, powering them by effectively the same explosion that is running your rocket itself, which means you have really, really hot rocket fuel going through a turbine. Now you can cool it using the liquid oxygen, which you have, which is really cold. The problem is your turbine is made out of metal. And oxygen corrodes metal, and heat accelerates that. So you have crazy amounts of heat that can melt through about three inch thick of metal, like it's a candle, and you're also filling it with a highly corrosive substance. That's why the US deemed this impossible. The US got their hand on one of these rocket engines, they tested out and found that, yes, in fact, in the 1960s, the Soviet Union did manage to design this rocket. Imagine if someone came to you today and said, I have a computer, I have a whole stock pile of them. They're the best Soviet servers you've ever seen. Not only are they more powerful than some of the servers that you have, they're also more efficient and use less power. We're also using a bunch of quantum technology that your physicists thought was impossible. Would you like to buy some? You would probably think they're crazy, but this actually happened. And they managed to sell them for $1.1 million a piece to the US. They use them throughout the 90s, and the descendants of this engine are still used today. There have only been about two or three rocket engines ever flown or designed that are more efficient than this particular engine. And yet the Soviet Union never made it to the moon. They had all this amazing engineering that could beat us every step of the way, partially because of the sort of agile process that they did and being willing to fail things. But whether it's due to lack of political will or a number of political problems, because there were some political issues within the USSR, as well as the number of things, they never made it. And that's really interesting, that despite all this really good engineering, they never managed to make it to the moon. Now, the last story I want to tell you about is about one bad byte. We've seen how much damage one line of code can do, but what can happen when just one little byte goes wrong? Specifically, one byte in the Galileo space probe. There was a space probe which was designed and built in the 1980s, launched in the 90s, and in I believe the late 90s reached the Jovian system and studied Jupiter and its moons and produced some really beautiful images and really expanded our knowledge of Jupiter and its moons. Now, Galileo is great. It did a lot of really amazing science, but there were a number of problems along the way. If you look up illustrations of Galileo, the early ones show this big antenna dish that's folded out on the top. And then if you look at later illustrations, you'll see it's just a little thing that looks like an umbrella that's just half folded. That's because the antenna dish didn't manage to open all the way. That's another story for another day. What I'm going to talk about is the magnetometer. That's the thing that measured the magnetic field of Jupiter. That magnetometer broke on Galileo because there was one byte, just one little tiny byte, in the source code which went bad. Space and especially Jupiter is a really punishing environment. There is a lot of radiation going on there. And so probably a gamma ray or something hit one of those bytes and just caused it to go bad and fried that little bit of the memory. So let's put yourself in JPL shoes. We're the people who design this particular spacecraft. So you have the code. You have the code that goes with the magnetometer, the code that has the bad biten. This is a good thing. So you know where that byte is and you also have all the source code so you can figure out what that code did and so you can work around it. Now that code was written in fourth. I don't know about you, but I wasn't familiar with fourth not that long ago. And it's a relatively obscure programming language, though it's not uncommon to see it being used in the space program. We'll get back to fourth later. Now it was developed using a development environment for the Apple II. This is a relatively old system. And the reason why I know that is because by the time this bug happened JPL had gotten rid of all their Apple IIs. So okay, we have to fix this code and we have the code luckily, but we don't have the computer that the code was developed on. Nor do we have the actual development environment that was used for generating that code and compiling it for that particular computer. How would you feel upon hearing this? You need to fix a bug in this code. Don't worry we have the code, but you don't have a compiler. Also we don't have any development systems. Here are the bytes, good luck. Enter Ron Garrett. Well the engineers on the magnetometer deem had deemed that this would be too expensive and too complicated to be able to fix just because of how much time it would take for them to recreate all the development tools required to fix this. Ron Garrett thought a little bit differently. He thought that he'd be able to do it in a lot less time and save JPL money or potentially save the magnetometer because they were going to give up on this thing. He had a couple advantages on the side. One was that the magnetometer used an 1802 processor. This is a relatively simple processor and it's very orthogonal. It's very simple in the way that it works and everything kind of works like everything else. It's very easy to figure out based off how one command works, how all the other commands work. Unlike modern day x86 processors which are really powerful but they're really complicated and inconsistent in how they're implemented. So the processor is relatively simple so writing a compiler for it isn't going to be the hardest task in the world. Another advantage of the ad is fourth. I want to take you a moment to show the fourth programming language. This is a valid line of fourth. I don't know about you but to me this looks like someone just took code and just sorted it so all the numbers are over here and then all of the other things are over here. The reason why it looks weird is because fourth is one of the very, very few programming languages to use what's known as post-fix notation. The reason why it does that is because of the run time that actually runs it. It is a stack-based programming language. Everything that you do is based off of a stack sort of like an array that you can only pop and push on to. You can't access into the middle of. And so if we run this line of fourth code we first get into the number four. The number four isn't just the number four, a value, it's also a command. It's a command that puts a four on the top of the stack. We then get to the number five which puts a five on the top of the stack, then a two which puts the two on the top of the stack pushing all the things before it down. Next we get to the plus command. The plus command takes the top two items on the stack, pops them off, it adds them together and puts them back on to the top of the stack. Then we have the multiplication command which takes the top two items off the stack, pops them off, multiplies them together and then puts them back on the top of the stack. Finally the period command takes whatever is on the top of the stack and prints it out to standard out or whatever your equivalent output is. Now this way that I wrote it is a little bit confusing and that's intentional to kind of show what postfix notation looks like. More often you might see fourth code that looks like this which makes a little bit sense as long as you just kind of flip the way your brain works. So we're going to take a five and a two, we're going to add them, we're going to take the result of that and we're going to multiply it by four and then we're going to output that. Now what about functions? Here is a function in fourth, specifically a function which squares a number. The colon there is just fourth syntax to say hey we have a function definition coming up then we have the word square, the name of the function. Next we have dup. Dup will take whatever happens to be on the stack when you are running this particular function and duplicate whatever is on the top of the stack. It'll just copy it and push another one on the stack. Then we take a multiply command, multiply those two numbers together, put them back on the top of the stack and the colon says they're at the end of the function. Now what's really cool about this and what some of you may have noticed is here we have a definition of a function and it doesn't have any parameters, it doesn't have any return value. In fourth you don't have any of that, you don't have variables, there are direct memory accesses but you don't have variables because you don't need that. Everything just operates on the stack. Not only does this make fourth very small, it also makes it very easy to write a fourth interpreter because you don't have to deal with all the complicated things with objects and arrays and having variables in your program or return arguments or lexical scope. None of those things are actually needed because of the way that fourth works. This makes fourth very easy to implement on very simple, small space constrained and also processor constrained microcomputers. So Ron Garrett had these two advantages for him that this was going to be relatively simple to do. He also had another secret which was that he was a Lisp programmer. Now Lisp programmers are often people who lurk in shadows and have long beards and speak in hush tones in parentheses. I don't have time to go into how wonderful Lisp is but it is a really amazing language and Ruby has actually pulled some stuff out of it. But one of the really great things about Lisp and specifically common Lisp which he used is it's a fantastic language for just hacking something together because the syntax of the language and the way it's run is relatively simple. In the case of Lisp you can construct programs that construct other programs or modify programs relatively easily. Meta programming. Also in the case of common Lisp the parser that parses your code before it evaluates it is actually something that you have control over at run time. So in common Lisp you can write some Lisp code that changes the way that the Lisp parser works so that midway through your code you can start writing everything in Ruby syntax and it will totally understand that. I don't know of almost any other language that has that ability and makes it so simple to do that. You should probably never do that but in the case of this you don't have that much time and you need to hack something together. And so Ron Garrett worked really hard on this. He did a lot of Lisp work and he managed to come up with a solution and I believe about a little more than a month and he did get some help from the magnetometer team at JPL. They helped him validate the program was correct and validate that his bug fix was correct. But in the end he managed to come up with a patch that could save the Galileo probe in not that much time working on his own despite all of the issues. Here is the actual patch that he wrote. I am not a fourth programmer and I don't know 1802 byte code. If any of you do know anything about 1802 byte code please come up to me afterwards because you're probably an interesting person to talk to. What I can tell you is that up there first we have a function definition, then we have the compiled version of that function, then we have another function definition which is another one that you have to patch and then a compiled version of that. I believe looking at the code what he had to do was fill out the space that was in that bad bit of memory with some code that would never get executed and then take another function and clean it up so it's a little more efficient and space compact so that they wouldn't take over the extra space. And all this code as well as the tools that he used to do it are available on GitHub for you to look at if you want to look at 1990's Lisp code. I actually contacted Ron Garrett not that long ago to ask him about this. Unfortunately he didn't remember too many things because this was 25 years ago and he worked on a lot of really cool interesting projects at JPL. But I did ask him for lessons and he did say that there was one lesson that he could take away from this project and that might be worth sharing with you. And that's that Lisp is really awesome. I mean it's really cool that you're able to do that. I would like to add the amendment that Forth is really awesome and that you are probably now the largest audience at any RubyConf to know what Forth is so congratulations. Now I've told you these four stories. This is where I normally would give you some takeaways but I don't really want to give you too much about what I think about these stories. I've already told you most of what I think. I want you to come up with sort of what can you learn about your own programming styles your own development, what you do at work, what you can take out of this. But there is one thing that I want to mention. That's that history is really awesome. We have documentation of a lot of these bugs and what happens because of NASA being a public organization and they have to share a lot of this stuff. But there's a lot that we can learn from in history. If we take the time to learn from other people's bugs to learn from things that were studied 10, 20, even 40 years ago there are really cool things to learn about computer programming. Things that we still haven't explored yet as programmers. We're doing ourselves a disservice if we don't explore history more. So if you'd like to learn more about this stuff, I have a couple of recommendations. One is to watch the talk, the remote agent experiment which was a Google tech talk that Ron Garrett gave. He goes into more detail about some of the other list programming that he did at JPL as well as a lot of organizational problems that happened. I'll just leave it at that. I'd also recommend reading the book, Digital Apollo. But if you're not that nerdy about space, this is a program that is about the Apollo guidance computer and the design for it. There's a lot of really interesting information there about UI design work and also the politics and philosophy of designing a UI when you're dealing with a situation where you have a bunch of people who really want to control the system directly and a bunch of engineers who say, no, no, we want a computer to do it. It's a really fascinating history and it's a really grabbing read. The Apollo 11 source code which is what I actually pulled from is up on GitHub if you want to look at it. You can actually go through and I believe it actually has proper code coloring on it too which is quite impressive. You can go through and read it. It's really fun to look at all the little Easter eggs that are in there. If you're really nerdy and you really want to understand it, I would recommend the book, The Apollo Guidance Computer. Well, Digital Apollo is really accessible. The Apollo Guidance Computer is a relatively readable book but it's not very accessible. It's hundreds of pages explaining how the Apollo Guidance Computer worked, how all the bits were arranged inside of it, all of the commands. It was an invaluable resource in putting together this talk and at the very, very end of it, he tells some stories about various things that went wrong and various bug fixes they did in the Apollo Guidance Computer but he doesn't tell it from the perspective of the people who worked on it. He tells it from the perspective of all the nitty gritty about the bits that actually were moved about to do these fixes. Finally, if you don't want to read or don't want to do a bunch of talks, I would recommend looking up the Moon Machines documentary series. They have a number of documentaries and they're available online, whether or not it's legal, I don't know, but they're available on YouTube if you want to go ahead and learn more about the F1 engines and the Sarin V rocket. They have another one, The Apollo Guidance Computer, which is really interesting if you're a computer programmer. Also I'd recommend watching the one on space suits. You wouldn't think space suits are interesting but I won't spoil it but you'd be surprised who actually makes the space suits that we still use in the US to go into space. A couple of special thanks. Thank you to Ron Garrett for helping me and sharing the fact that he had put up this code on GitHub. Also, thanks to my friend Rachel McGuffin. She's the one who did all the illustrations that you did in this presentation. I think that this talk wouldn't have been as good if it wasn't for Rachel McGuffin. I'd like you to take a moment to please give a round of applause for Ron Garrett for this amazing bug fix that he did because I don't think anyone's ever given him a really thorough round of applause for all that work he did and Rachel McGuffin for all the work that she did putting together these illustrations. So the Merlin heavy lift, sorry this isn't going to get really nerdy for a moment. He asked how did the engines compare to, that I talked about to some of the ones on the spacecraft that Elon Musk's SpaceX has been developing. I don't know if the F1 engine is more powerful. The F1 engine actually isn't the most powerful rocket engine ever developed. It's the most powerful single nozzle liquid propellant rocket engine ever developed. The reason for that is you see those two big white tubes on the sides of the space shuttle whenever it launched. Those rocket engines are stupidly powerful, like orders of magnitude more powerful than the one that I showed you, but those are solid rocket fuel engines. They're quite different. Specifically to your question, the NK33 engine, the only ones more efficient than this are some theoretical engines that were never built, some derivatives of it, and other engines designed by the Soviet Union, and then the Merlin engine, which SpaceX developed. I don't know about some of the details about the specific engines that you developed. I think they're not necessarily more efficient in terms of thrust to weight ratio, but they have a lot of other really interesting characteristics. And because they haven't flown yet, they don't really necessarily count as real rocket engines until they get on a rocket and fly. Any other questions? All righty, well, it's time to go. If you have any other questions or you want to just come up and see me, I love talking about this stuff, and thank you again.
 I think it's about time to get started.  If people want to keep filing in there are plenty of seats, go ahead and sit down wherever.  No need to say in the back.  So my name is Colin Fulton.  I'm a front-end developer with Duo Security.  If you want to reach me on Twitter, you can hit me up at Peter Kweins.  I don't really use Twitter, so if you actually want to reach me personally, just email me  at justcolon at gmail.com.  If I don't respond, just send an email again.  Sometimes I forget to respond to things.  And all the slides, as well as a number of other resources, are going to be up on github.com.  They're all available now with my github account, Just Colin.  And you can go browse those at your wish.  Alrighty.  So many of you may remember the time that NASA sent a rocket to Mars, and it crashed  into Mars.  And the reason why it crashed is because they had one piece of software that used pounds  per second, while the rest of their software used newtons per second.  And this was very embarrassing for NASA, but how many of you have ever done a multi-corporation  project which takes years, if not decades, to complete with many people working on specs  that are hundreds of pages long and not written a single bug in it?  In this case, the bug was pretty bad because they lost a multi-million-dollar satellite,  but still, I think we all can relate to this kind of problem.  All of you may also remember when we initially launched Hubble into space, the images that  it took initially were really blurry.  They couldn't do anything with them.  And that's because they had actually gotten the geometry of the mirror wrong.  So they had to send up a number of astronauts in the space shuttle to go install, effectively,  a pair of glasses to correct it.  What a lot of people don't know is that they were actually able to measure how blurry the  images were, and the glasses not only got them just as good as it would have been if  they had initially gotten it right, they actually were able to correct for a number of other  things and get sharper images than they would have gotten if they hadn't had this bug.  This is an expensive way to discover a way to get a sharper image, but still, it worked.  So today I want to tell you four stories about bugs in space programs, because there's a  lot that we can learn from bugs, and space programs are really cool, so the bugs are  more exciting.  I'm going to tell you right up front.  I'm not some guru on top of a mountain who can pass out sage advice.  I'm going to be editorializing a little bit, but I'm not going to be able to give you grand  lessons in this talk.  I'm just a workaday developer, like all of you.  In fact, many of you here, if not most of you, are probably smarter than me and know  more about some of these subjects than me.  But I'm going to tell you these stories so that you can kind of reflect on your own  experience and what you've done and see what you can get out of them.  And I'll provide a couple of little editorializing bits to kind of give you pointers and where  I think that's interesting to think about these stories.  So many of you have probably heard the phrase, we can send a man to the moon, but, which  is usually said to be like, we can send a man to the moon, but we can't get a printer  to work well.  I don't know if any of you have ever done anything with printer firmware, but it's  actually more complicated than you think it would be.  And going to the moon was not something that we did perfectly.  There were a number of bugs along the way, and this is just one of them.  Specifically Apollo 14.  And no, this isn't a typo.  This isn't Apollo 13, the famous one that had the explosion and the exciting movies.  No, no, this was a much more subtle, but still almost dangerous bug.  Now, in Apollo 14, they managed to get to the moon and they were orbiting the moon.  And it was less than four hours before they were about to land.  And the crew on the ground noticed that the abort bit was set in the computer.  That's the bit that gets set when you push the abort button, which the computer then  checks and says, oh, God, the abort bit is set.  We have to abort whatever we're doing and get back into lunar orbit.  Now, luckily, they weren't in one of the procedures that would actually check the abort  bit yet, and so was ignoring it.  But what they ended up discovering was that there was probably a little bit of metal or  a little bit of solder inside the abort button that was just floating around and occasionally  randomly shorted, which meant the abort button was acting as if it was randomly being pushed  every so often.  This is fine while you're orbiting the moon, but while you're landing, it means that there  is a certain percentage chance that you're just going to have the rocket thruster turn  on, whip you around the other way, and send you back into orbit.  And you only have enough fuel to land on the moon once because it's really expensive to  get all that fuel into space.  So this was a major problem, a really serious problem.  So they contacted the guys who knew about the Apollo Guidance Computer, the people  who had actually written the code for it and said, how can you help us?  And so they had about three to four hours to figure out a way to patch the solution  using the Apollo Guidance Computer so that it would ignore this particular abort button.  They also needed to make sure that there was another way to do an abort.  They only had three to four hours to create that patch, test it, send it over to Houston  where they would then test it again, and then relay that patch up to the astronauts to manually  punch in.  Now the problem is the Apollo Guidance Computer really didn't want to ignore that abort button  for a number of different reasons.  So this is some Apollo Guidance Computer source code.  If you've never looked at assembly, this may look a little bit weird to you, but it's actually  very, very similar to the assembly programs that are written today.  In the first column, optionally, there's a label which you can jump to.  So you can think of this as just like a name that you can put there either as documentation  or to say elsewhere in your code, hey, once you get to this point, jump over to this other  part of the code.  Then the next bit is what's called the operator, or sometimes these are known as mnemonics.  These are little tiny short strings which represent different commands.  In this case, the CA there means clear register and then add on to it.  Finally, we have the operator.  This is the thing that you're going to operate on.  So in this case, we are going to clear register and then add in this particular flag to it  which is stored.  Then finally, there are comments which, hey, look, they use hash comments just like Ruby.  That's pretty cool.  So what does this line of code do?  The first line clears out a register and then sets it to a word that's full of a bunch of  different flag bits.  Next they use the mask command to mask out all but the let abort bit, the bit that actually  gets used.  So now all the other bits will be zero except if that let abort bit is set, that'll be  one.  Then there's a little bit of magic.  They didn't have enough space to get all the commands in so they have to call this  magic extend command but we don't have time to go into what that does.  Then finally, there's a branch if zero command.  So what that'll do is it'll check, is the number that's currently in the register zero?  If so, it will go ahead and jump to the land disk there which is another label in the program  and if not, it'll continue on to do the abort sequence.  So if the bit is set, it'll go ahead and continue with the abort sequence.  If the bit isn't set, then they'll go ahead and do it.  And so there's this let abort flag stored in memory.  So let's just set the let abort flag.  This'll be easy.  Let's go ahead and set it and then if we set it to zero, then this code will run and an  abort isn't going to happen.  The problem is it isn't that simple.  There are a variety of programs that get run that also change that let abort flag because  they want to set that flag during the landing to say, hey, we're in the middle of a landing  and that means we may want to abort.  And so they would have to set the let abort flag immediately after the computer set it  and that would mean that there was a gap of time between when the computer set it and  when they manually set it that theoretically that abort button could short out and it would  be a problem.  So if you also may be thinking that maybe we can just send up a patch for the code that  will ignore the abort button, but in this case the code itself is woven with a bunch  of metal wires by a bunch of women guided by a computer in Massachusetts.  So if we want to patch the code, we would have to actually weave a bunch of wires together,  package it up in epoxy, then send it up in a rocket and send it back to the moon and  you can't really do that in four hours.  So that's why we can't edit this code.  It's very, very much read only, more so than we have to deal with today.  So the solution that they came up with has to do with the next line of code that we have  here.  You'll see that immediately after we check the abort flag, we check that modreg there  and that checks if we are in program 71.  Program 71 is one of the two abort programs.  The other is program 70.  If the computer senses that it's already an abort sequence, it's not going to go into  another abort sequence because why would you abort an abort?  So instead of setting the abort flag, we can set modreg.  If we do that, then we're going to go ahead and trick the computer that we're already  in an abort, but it'll continue doing whatever it's doing.  This will solve the problem, except it's a little bit more complicated than that because  there are other programs that check this particular register to see what program they're in in  order to figure out what to do during the landing.  So the sequence that they came up with was this.  First you set the modreg to program 71.  That tells the computer that you're already in an abort sequence, so it's going to ignore  abort after you set that.  Then you have to wait until the ignition starts, which is going to automatically set the let-abort  flag, but we don't care about that because we already changed the modreg register.  Then you're going to push the throttle up to full because the computer isn't going to  do that because it thinks it's in the middle of an abort sequence.  Then you're going to set the zoom flag, which lets the computer know that, hey, in the future,  you can go ahead and throttle up the rocket because, again, it's checking these flags  and we're kind of really screwing with the way the program is supposed to work.  After that, you can reset the let-abort flag now that it's safe to go and set it back to  zero because it isn't going to be set again.  That'll, again, cause it to ignore the abort button.  Then we can go ahead and set the mode register back to program 63, the program that it thinks  that it really should be in, and that way it'll go ahead and check the zoom flag and  throttle up the engine as it should for landing.  Then they can go ahead and turn the throttle back down and let the computer take control.  If you're confused, don't worry.  It's a little bit confusing and convoluted, but all this put together meant that they  were able to safely ignore that abort button.  In future flights, they actually set up additional routines to allow them to do this in a slightly  less convoluted way in case this happened again in the future.  Not about you, but I think monkey patching is really great when you need it.  You can fix amazing problems like this.  If they didn't have the ability to manually go in and edit all the registers in memory  while the program is running, they would have been screwed.  They may have aborted midway through the flight.  However, monkey patching will save you in situations like this, but you only should really use  it when you need it, which is almost never.  I don't know about you, but I'm not landing people on the moon terribly often.  Most of the time I have more than four hours to fix a bug, so take it as you will.  The next thing I want to talk about is how much damage one line of code can do.  The line of code was in the Ariane 5 rocket, which is one of three heavy lifting rockets  in the world.  These are rockets which can lift tens of thousands of pounds into space.  They are big, they are expensive, they are complicated, and we don't have that many of  them because it's not that often that we're throwing things that big into space, but every  so often they're needed.  Flight 501 was to be the first flight of the rocket.  The European space union had spent pretty much about a decade developing this rocket  in total.  They had spent a lot of time carefully testing all their systems and making sure that this  would work because this project in total cost them about $7 billion.  So if something went wrong, that's a big loss.  These rockets are disposable, they don't reuse them every single time, but their first launch  was going to be an actual mission launching satellites into space.  So if this mission failed, not only would they lose the rocket, they would also lose  all the satellites that they were trying to launch into space.  So Flight 501, initially the launch is going okay.  Everything is relatively nominal.  There are a couple little hiccups that happen, but this is normal with a new rocket.  Now less than a minute into the launch, the thrusters on the rocket suddenly gimbled to  one direction and that caused the rocket to turn really suddenly.  Now rockets have a pointy end and you want to keep that pointy end going into the wind  because that's good because then the wind will just go around it.  If the wind starts beating against to the side, the rocket is going to fall apart.  And so the rocket did start to disintegrate and there's a video of this.  The rocket you can see bits of the rocket coming off from all the wind hitting it.  And at this point the rocket exploded, which generally is a bad thing.  However, in this case, it's a good thing because when a rocket like this is in a dangerous  situation, it has a self-destruct sequence.  So it can just blow up.  In this case, the rocket detected that it was falling apart and so it intentionally exploded  to prevent even worse things from happening.  It's bad when a bunch of bits of rockets fall to the ground from an exploded rocket.  It's even worse when a rocket that is flying flies into the ground.  Now the line of code that failed is this one.  Now there are a couple problems here.  I don't know Ada code and this is written in Ada and even though this is relatively  easy to read, they didn't use the most intuitive variable or function names.  I assume that if you worked on this rocket, this makes perfect sense to you.  But the line that went wrong is this bit right here.  And in this bit, they are taking a 64-bit floating point number and converting it into  a 16-bit integer.  64 is bigger than 16.  And they weren't checking whether or not overflowed.  This isn't like Ruby where you can just add arbitrary numbers together and it will magically  just figure out how to get those big numbers for you.  Now luckily, they had actually detected all of the possible overflows in their system because  they are rocket engineers.  They are doing actual rocket science so they knew that overflows were possible.  And they had fixed almost all of the overflows.  And the reason why they couldn't fix all of them is because they had made a rule that  their computer couldn't work harder than 80% of its capability because this is a relatively  resource constrained system so if they work it too hard, it's going to cause problems  and then crash.  Now the reason why they didn't put the checks around this bit of code is pretty interesting.  They actually tested it with real data, which is a good thing, except they tested it with  real data from the Arian 4 rocket, which was a different rocket, which had a different  flight profile.  So in the Arian 4 rocket, this particular bit of code would never overflow, except  in the Arian 5 rocket, the horizontal velocity, which is what this particular number represents,  was much larger and thus it overflowed.  Now luckily for them, they had two copies of the computer running this and it was only  the backup that failed first with this overflow.  The problem is they're both getting the same data, so immediately after, just milliseconds  after the backup failed, the other system failed and so they had both redundant systems  go down.  Now a team was formed to investigate exactly what happened and what went wrong and they  got a number of different conclusions out of this.  One is that the computer itself was assuming, or I'm sorry, one is that they were protecting  against random faults, that's why you have two computers.  If one happens to randomly fail due to a bad capacitor or a bad solder joint, you still  have the other computer and it can take over.  They weren't protecting against design faults, they weren't protecting against the crazy  idea that maybe there's a bug in their code and something went wrong.  The committee that was looking at it instructed that they should really just assume that all  faults where is soft, is faulty until it is correct.  If you do test-driven development, this is what you do.  You assume that your code is bad until you have a test that can prove that it's right.  The first thing you do is create a test that can prove that that code is right.  Then as you write it, you can check that.  The other issue is that the computer had assumed that everything would work and thus when it  had this error, it just stopped working and sent out an error message.  That error message was sent to the rocket Gimbal's, which interpreted it as a movement  command and thus it went over and blew up.  The next thing that they said is that you should focus not only on developing good code  but good documentation because a lot of the investigation that they did and actually figure  out all these bugs and exactly what happened wouldn't be possible if it wasn't for all  the documentation about their process that they provided.  The final recommendation was to remove dead code.  This is kind of the zinger with the story.  That code was to calibrate the gyroscopes inside the rocket to make sure that they knew  which direction was up.  That program would run when the rocket was sitting on the launch pad so it was constantly  calibrating those gyroscopes so when it launched, it knew what direction up was.  It was still running after the launch happened.  This doesn't quite make sense because after the launch happens, the rocket is going to  be turning so you don't want to calibrate which way is up while you're actually turning  the rocket.  The reason why they kept it running in the Ariane 4 rocket, the previous rocket, was  a very clever hack.  They wanted to make sure that if the launch was extended for some reason, they had to  delay when the launch was, they were still running that calibration sequence.  They could get it up and running faster.  In this case, this dead code caused a rocket to blow up.  They said, okay, clever hacks are all fine and good but remove dead code.  It isn't doing anything.  Your clever hack is very nice but it caused a rocket to explode.  There's a little post-script.  The first launch of the Ariane 5 rocket was obviously not so good but a lot of the subsequent  launches were quite good.  In fact, even through to this year, they are currently at a record of 81 successful launches  in a row.  They've had about four launches total in a history of almost 100 launches.  This is a really successful rocket and they did a really good job with it.  And after doing this investigation, they fixed a lot of the problems in their systems.  The next thing I want to talk to you about is agile development and agile development  in the Soviet Union.  We all know that the US won the space race and we know the US won the space race, won  because we're Americans, at least most of us are, and we know that America is the best.  We put the first people on the moon and we have the first people orbit the moon before  that.  Therefore, we won the space race.  The problem with this is that it ignores the rest of the space race.  We did accomplish that, we definitely did do that.  The USSR did a couple of things before us.  They actually did all of the things before us.  They beat us at every step of the space race, which is why we didn't feel so good throughout  it.  So the question is, why is it that the USSR never managed to send a person to the moon?  We got to the moon and they beat us every step of the way, so surely they were better  than us.  The answer is that getting to the moon is hard, really hard.  One of the hard problems is you need to launch a tremendously large rocket because not only  do you need to get into space, you need to get to the moon.  So you need a rocket big enough to get from low Earth orbit to the moon.  And you need to get that rocket into space, which means you need the largest rocket that  humanity has ever constructed.  Now the engine powering that rocket for the US was the F1 engine.  This thing is an amazing feat of engineering.  To give you an idea of how big and powerful this rocket was, it took in fuel at a rate  such that the pumps that it used could drain an average size bathtub in about 200 milliseconds.  That's a lot of fuel.  And not only did they have this engine that was sucking in all this fuel, they had five  of them on the first stage.  Five of these engines.  Look at how big these things are.  That's actually a relatively tall man standing next to them.  And look at just how ginormous this thing is.  This is an amazing feat of engineering.  And the USSR lacked the industrial capabilities of the United States.  They weren't able to produce engines that big and that powerful.  They just didn't have the industry capable of doing it.  So they decided to go a different tact.  They designed the NK33 engine.  Now they knew they couldn't go big, so instead they went efficient.  They decided to design a rocket engine which had as good thrust to weight ratio as possible.  And because it was lighter but still delivered a lot of thrust, they could use more of them.  It wasn't as powerful, that's why they had to use more of them.  And specifically, they used 33 of them.  That's a lot of rocket engines to put on the bottom of your rocket.  And all of those have to work and work every single time.  So not only did they have to make a very efficient engine, they also had to make a relatively  simple and reliable engine.  Now when the US was designing their F1 engine, they did a lot of design work.  They were very careful in their designs that when they launched the rocket, they were relatively  sure that it was going to work.  It didn't work all of the time, but they had to be really cautious because the press was  constantly watching them.  And any failure was another example of how the US was losing the space race.  The Soviet Union on the other hand decided to take a more hardware-centered approach.  They'd still do lots of design work, of course, but they were more willing to launch a rocket  and just see what happens.  If it blew up, you could learn from that explosion, improve the rocket, and iterate, much like  we do with agile software development.  And so they blew up a lot of rockets.  What's okay about this from the Soviet Union's perspective is that they didn't have the free  press of the United States, so a lot of their explosions were hidden.  And because they were doing them out in the middle of, I believe, Kazakhstan, there weren't  really any people watching.  And so all those explosions could be much better hidden.  We didn't find out about a lot of them until much later.  Now the NK-33 engine was designed to replace the previous engines used on the N1 rocket,  the equivalent of the Saturn V rocket, the rocket that was going to take the USSR to the  moon.  The first four launches blew up, which is okay because they designed the NK-33 engine  to be a more efficient, better engine, and it was going to be used on the fifth launch  in order to get them to the moon.  The problem is the fifth launch was canceled because at this point the US had already gotten  to the moon.  And so the Union wasn't interested in going to the moon anymore.  And so the Soviet government ordered that all the NK-33 engines, which were tested but  never actually used on a rocket, should be destroyed.  The engineers didn't do that, though, because these were really good engines.  They knew they were good engines, that they had something special here.  So they just hid them away in a warehouse, and they didn't let the government know about  this.  And it wasn't until the 1990s that it was revealed that they had the stockpile of old  engines from the 1960s.  And so they got in contact with some American rocket engineers from a company and said,  hey, we have this really cool rocket that we think you might be interested in.  And when they started to tell the US engineers about the numbers, the US engineers didn't  believe it, because this was an engine that had a thrust to weight ratio significantly  better than anything that they had managed to design in all the subsequent decades.  Not only that, it used something called a closed cycle technology.  Now for those of you who aren't rocket nerds, I can't really go into the details of what  this is, but it's an engineering feat so impressive that the US had deemed it impossible, because  it involves powering those pumps that pump all the fuel, powering them by effectively  the same explosion that is running your rocket itself, which means you have really, really  hot rocket fuel going through a turbine.  Now you can cool it using the liquid oxygen, which you have, which is really cold.  The problem is your turbine is made out of metal.  And oxygen corrodes metal, and heat accelerates that.  So you have crazy amounts of heat that can melt through about three inch thick of metal,  like it's a candle, and you're also filling it with a highly corrosive substance.  That's why the US deemed this impossible.  The US got their hand on one of these rocket engines, they tested out and found that, yes,  in fact, in the 1960s, the Soviet Union did manage to design this rocket.  Imagine if someone came to you today and said, I have a computer, I have a whole stock pile  of them.  They're the best Soviet servers you've ever seen.  Not only are they more powerful than some of the servers that you have, they're also  more efficient and use less power.  We're also using a bunch of quantum technology that your physicists thought was impossible.  Would you like to buy some?  You would probably think they're crazy, but this actually happened.  And they managed to sell them for $1.1 million a piece to the US.  They use them throughout the 90s, and the descendants of this engine are still used  today.  There have only been about two or three rocket engines ever flown or designed that are more  efficient than this particular engine.  And yet the Soviet Union never made it to the moon.  They had all this amazing engineering that could beat us every step of the way, partially  because of the sort of agile process that they did and being willing to fail things.  But whether it's due to lack of political will or a number of political problems, because  there were some political issues within the USSR, as well as the number of things, they  never made it.  And that's really interesting, that despite all this really good engineering, they never  managed to make it to the moon.  Now, the last story I want to tell you about is about one bad byte.  We've seen how much damage one line of code can do, but what can happen when just one  little byte goes wrong?  Specifically, one byte in the Galileo space probe.  There was a space probe which was designed and built in the 1980s, launched in the 90s,  and in I believe the late 90s reached the Jovian system and studied Jupiter and its moons  and produced some really beautiful images and really expanded our knowledge of Jupiter  and its moons.  Now, Galileo is great.  It did a lot of really amazing science, but there were a number of problems along the  way.  If you look up illustrations of Galileo, the early ones show this big antenna dish that's  folded out on the top.  And then if you look at later illustrations, you'll see it's just a little thing that looks  like an umbrella that's just half folded.  That's because the antenna dish didn't manage to open all the way.  That's another story for another day.  What I'm going to talk about is the magnetometer.  That's the thing that measured the magnetic field of Jupiter.  That magnetometer broke on Galileo because there was one byte, just one little tiny byte,  in the source code which went bad.  Space and especially Jupiter is a really punishing environment.  There is a lot of radiation going on there.  And so probably a gamma ray or something hit one of those bytes and just caused it to go  bad and fried that little bit of the memory.  So let's put yourself in JPL shoes.  We're the people who design this particular spacecraft.  So you have the code.  You have the code that goes with the magnetometer, the code that has the bad biten.  This is a good thing.  So you know where that byte is and you also have all the source code so you can figure  out what that code did and so you can work around it.  Now that code was written in fourth.  I don't know about you, but I wasn't familiar with fourth not that long ago.  And it's a relatively obscure programming language, though it's not uncommon to see  it being used in the space program.  We'll get back to fourth later.  Now it was developed using a development environment for the Apple II.  This is a relatively old system.  And the reason why I know that is because by the time this bug happened JPL had gotten  rid of all their Apple IIs.  So okay, we have to fix this code and we have the code luckily, but we don't have the computer  that the code was developed on.  Nor do we have the actual development environment that was used for generating that code and  compiling it for that particular computer.  How would you feel upon hearing this?  You need to fix a bug in this code.  Don't worry we have the code, but you don't have a compiler.  Also we don't have any development systems.  Here are the bytes, good luck.  Enter Ron Garrett.  Well the engineers on the magnetometer deem had deemed that this would be too expensive  and too complicated to be able to fix just because of how much time it would take for  them to recreate all the development tools required to fix this.  Ron Garrett thought a little bit differently.  He thought that he'd be able to do it in a lot less time and save JPL money or potentially  save the magnetometer because they were going to give up on this thing.  He had a couple advantages on the side.  One was that the magnetometer used an 1802 processor.  This is a relatively simple processor and it's very orthogonal.  It's very simple in the way that it works and everything kind of works like everything  else.  It's very easy to figure out based off how one command works, how all the other commands  work.  Unlike modern day x86 processors which are really powerful but they're really complicated  and inconsistent in how they're implemented.  So the processor is relatively simple so writing a compiler for it isn't going to be the hardest  task in the world.  Another advantage of the ad is fourth.  I want to take you a moment to show the fourth programming language.  This is a valid line of fourth.  I don't know about you but to me this looks like someone just took code and just sorted  it so all the numbers are over here and then all of the other things are over here.  The reason why it looks weird is because fourth is one of the very, very few programming  languages to use what's known as post-fix notation.  The reason why it does that is because of the run time that actually runs it.  It is a stack-based programming language.  Everything that you do is based off of a stack sort of like an array that you can only pop  and push on to.  You can't access into the middle of.  And so if we run this line of fourth code we first get into the number four.  The number four isn't just the number four, a value, it's also a command.  It's a command that puts a four on the top of the stack.  We then get to the number five which puts a five on the top of the stack, then a two  which puts the two on the top of the stack pushing all the things before it down.  Next we get to the plus command.  The plus command takes the top two items on the stack, pops them off, it adds them together  and puts them back on to the top of the stack.  Then we have the multiplication command which takes the top two items off the stack, pops  them off, multiplies them together and then puts them back on the top of the stack.  Finally the period command takes whatever is on the top of the stack and prints it out  to standard out or whatever your equivalent output is.  Now this way that I wrote it is a little bit confusing and that's intentional to kind  of show what postfix notation looks like.  More often you might see fourth code that looks like this which makes a little bit sense  as long as you just kind of flip the way your brain works.  So we're going to take a five and a two, we're going to add them, we're going to take the  result of that and we're going to multiply it by four and then we're going to output  that.  Now what about functions?  Here is a function in fourth, specifically a function which squares a number.  The colon there is just fourth syntax to say hey we have a function definition coming  up then we have the word square, the name of the function.  Next we have dup.  Dup will take whatever happens to be on the stack when you are running this particular  function and duplicate whatever is on the top of the stack.  It'll just copy it and push another one on the stack.  Then we take a multiply command, multiply those two numbers together, put them back  on the top of the stack and the colon says they're at the end of the function.  Now what's really cool about this and what some of you may have noticed is here we have  a definition of a function and it doesn't have any parameters, it doesn't have any return  value.  In fourth you don't have any of that, you don't have variables, there are direct memory  accesses but you don't have variables because you don't need that.  Everything just operates on the stack.  Not only does this make fourth very small, it also makes it very easy to write a fourth  interpreter because you don't have to deal with all the complicated things with objects  and arrays and having variables in your program or return arguments or lexical scope.  None of those things are actually needed because of the way that fourth works.  This makes fourth very easy to implement on very simple, small space constrained and also  processor constrained microcomputers.  So Ron Garrett had these two advantages for him that this was going to be relatively simple  to do.  He also had another secret which was that he was a Lisp programmer.  Now Lisp programmers are often people who lurk in shadows and have long beards and speak  in hush tones in parentheses.  I don't have time to go into how wonderful Lisp is but it is a really amazing language  and Ruby has actually pulled some stuff out of it.  But one of the really great things about Lisp and specifically common Lisp which he used  is it's a fantastic language for just hacking something together because the syntax of the  language and the way it's run is relatively simple.  In the case of Lisp you can construct programs that construct other programs or modify programs  relatively easily.  Meta programming.  Also in the case of common Lisp the parser that parses your code before it evaluates  it is actually something that you have control over at run time.  So in common Lisp you can write some Lisp code that changes the way that the Lisp parser  works so that midway through your code you can start writing everything in Ruby syntax  and it will totally understand that.  I don't know of almost any other language that has that ability and makes it so simple  to do that.  You should probably never do that but in the case of this you don't have that much time  and you need to hack something together.  And so Ron Garrett worked really hard on this.  He did a lot of Lisp work and he managed to come up with a solution and I believe about  a little more than a month and he did get some help from the magnetometer team at JPL.  They helped him validate the program was correct and validate that his bug fix was correct.  But in the end he managed to come up with a patch that could save the Galileo probe  in not that much time working on his own despite all of the issues.  Here is the actual patch that he wrote.  I am not a fourth programmer and I don't know 1802 byte code.  If any of you do know anything about 1802 byte code please come up to me afterwards because  you're probably an interesting person to talk to.  What I can tell you is that up there first we have a function definition, then we have  the compiled version of that function, then we have another function definition which  is another one that you have to patch and then a compiled version of that.  I believe looking at the code what he had to do was fill out the space that was in that  bad bit of memory with some code that would never get executed and then take another function  and clean it up so it's a little more efficient and space compact so that they wouldn't take  over the extra space.  And all this code as well as the tools that he used to do it are available on GitHub for  you to look at if you want to look at 1990's Lisp code.  I actually contacted Ron Garrett not that long ago to ask him about this.  Unfortunately he didn't remember too many things because this was 25 years ago and he  worked on a lot of really cool interesting projects at JPL.  But I did ask him for lessons and he did say that there was one lesson that he could take  away from this project and that might be worth sharing with you.  And that's that Lisp is really awesome.  I mean it's really cool that you're able to do that.  I would like to add the amendment that Forth is really awesome and that you are probably  now the largest audience at any RubyConf to know what Forth is so congratulations.  Now I've told you these four stories.  This is where I normally would give you some takeaways but I don't really want to give  you too much about what I think about these stories.  I've already told you most of what I think.  I want you to come up with sort of what can you learn about your own programming styles  your own development, what you do at work, what you can take out of this.  But there is one thing that I want to mention.  That's that history is really awesome.  We have documentation of a lot of these bugs and what happens because of NASA being a public  organization and they have to share a lot of this stuff.  But there's a lot that we can learn from in history.  If we take the time to learn from other people's bugs to learn from things that were studied  10, 20, even 40 years ago there are really cool things to learn about computer programming.  Things that we still haven't explored yet as programmers.  We're doing ourselves a disservice if we don't explore history more.  So if you'd like to learn more about this stuff, I have a couple of recommendations.  One is to watch the talk, the remote agent experiment which was a Google tech talk that  Ron Garrett gave.  He goes into more detail about some of the other list programming that he did at JPL  as well as a lot of organizational problems that happened.  I'll just leave it at that.  I'd also recommend reading the book, Digital Apollo.  But if you're not that nerdy about space, this is a program that is about the Apollo  guidance computer and the design for it.  There's a lot of really interesting information there about UI design work and also the politics  and philosophy of designing a UI when you're dealing with a situation where you have a bunch  of people who really want to control the system directly and a bunch of engineers who say,  no, no, we want a computer to do it.  It's a really fascinating history and it's a really grabbing read.  The Apollo 11 source code which is what I actually pulled from is up on GitHub if you  want to look at it.  You can actually go through and I believe it actually has proper code coloring on it  too which is quite impressive.  You can go through and read it.  It's really fun to look at all the little Easter eggs that are in there.  If you're really nerdy and you really want to understand it, I would recommend the book,  The Apollo Guidance Computer.  Well, Digital Apollo is really accessible.  The Apollo Guidance Computer is a relatively readable book but it's not very accessible.  It's hundreds of pages explaining how the Apollo Guidance Computer worked, how all  the bits were arranged inside of it, all of the commands.  It was an invaluable resource in putting together this talk and at the very, very end  of it, he tells some stories about various things that went wrong and various bug fixes  they did in the Apollo Guidance Computer but he doesn't tell it from the perspective  of the people who worked on it.  He tells it from the perspective of all the nitty gritty about the bits that actually  were moved about to do these fixes.  Finally, if you don't want to read or don't want to do a bunch of talks, I would recommend  looking up the Moon Machines documentary series.  They have a number of documentaries and they're available online, whether or not it's legal,  I don't know, but they're available on YouTube if you want to go ahead and learn more about  the F1 engines and the Sarin V rocket.  They have another one, The Apollo Guidance Computer, which is really interesting if you're  a computer programmer.  Also I'd recommend watching the one on space suits.  You wouldn't think space suits are interesting but I won't spoil it but you'd be surprised  who actually makes the space suits that we still use in the US to go into space.  A couple of special thanks.  Thank you to Ron Garrett for helping me and sharing the fact that he had put up this code  on GitHub.  Also, thanks to my friend Rachel McGuffin.  She's the one who did all the illustrations that you did in this presentation.  I think that this talk wouldn't have been as good if it wasn't for Rachel McGuffin.  I'd like you to take a moment to please give a round of applause for Ron Garrett for this  amazing bug fix that he did because I don't think anyone's ever given him a really thorough  round of applause for all that work he did and Rachel McGuffin for all the work that  she did putting together these illustrations.  So the Merlin heavy lift, sorry this isn't going to get really nerdy for a moment.  He asked how did the engines compare to, that I talked about to some of the ones on the  spacecraft that Elon Musk's SpaceX has been developing.  I don't know if the F1 engine is more powerful.  The F1 engine actually isn't the most powerful rocket engine ever developed.  It's the most powerful single nozzle liquid propellant rocket engine ever developed.  The reason for that is you see those two big white tubes on the sides of the space shuttle  whenever it launched.  Those rocket engines are stupidly powerful, like orders of magnitude more powerful than  the one that I showed you, but those are solid rocket fuel engines.  They're quite different.  Specifically to your question, the NK33 engine, the only ones more efficient than this are  some theoretical engines that were never built, some derivatives of it, and other engines  designed by the Soviet Union, and then the Merlin engine, which SpaceX developed.  I don't know about some of the details about the specific engines that you developed.  I think they're not necessarily more efficient in terms of thrust to weight ratio, but they  have a lot of other really interesting characteristics.  And because they haven't flown yet, they don't really necessarily count as real rocket engines  until they get on a rocket and fly.  Any other questions?  All righty, well, it's time to go.  If you have any other questions or you want to just come up and see me, I love talking  about this stuff, and thank you again.
Buuuuugs_0_47.png Buuuuugs_48_131.png Buuuuugs_132_287.png Buuuuugs_288_503.png Buuuuugs_504_1307.png Buuuuugs_1308_3611.png Buuuuugs_3612_4991.png Buuuuugs_4992_10487.png Buuuuugs_10488_12191.png Buuuuugs_12192_16595.png Buuuuugs_16596_17147.png Buuuuugs_17148_17951.png Buuuuugs_17952_18155.png Buuuuugs_18156_19643.png Buuuuugs_19644_26579.png Buuuuugs_26580_28667.png Buuuuugs_28668_29243.png Buuuuugs_29244_29951.png Buuuuugs_29952_32303.png Buuuuugs_32304_33863.png Buuuuugs_33864_37295.png Buuuuugs_37296_37979.png Buuuuugs_37980_41699.png Buuuuugs_41700_50075.png Buuuuugs_50076_51251.png Buuuuugs_51252_52139.png Buuuuugs_52140_52691.png Buuuuugs_52692_57707.png Buuuuugs_57708_57803.png Buuuuugs_57804_58259.png Buuuuugs_58260_58355.png Buuuuugs_58356_58451.png Buuuuugs_58452_58751.png Buuuuugs_58752_58799.png Buuuuugs_58800_59015.png Buuuuugs_59016_59087.png Buuuuugs_59088_59327.png Buuuuugs_59328_59567.png Buuuuugs_59568_59615.png Buuuuugs_59616_59735.png Buuuuugs_59736_59951.png Buuuuugs_59952_60119.png Buuuuugs_60120_60143.png