Speed Science
CC Attribution - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Schneeman, Richard
Run your app faster, with less RAM and a quicker boot time today. How? With science! In this talk we'll focus on the process of turning performance problems into reproducible bugs we can understand and squash. We'll look at real world use cases where small changes resulted in huge real world performance gains. You'll walk away with concrete and actionable advice to improve the speed of your app, and with the tools to equip your app for a lifetime of speed. Live life in the fast lane, with science!
 Okay, so I think to disappoint everyone, the title is Thought Here to See Science. We will not actually be talking about methamphetamine production, but we will be talking about science. Instead, we will be talking about goals. We will be talking about goals and our main goal is getting faster and achieving more. One of the things I like to think about whenever I am dealing with any kind of performance problem is that it's the most important. This is really comforting for me because mostly in my application I do bugs every day, every hour, every minute, you know, a lot. And the great thing about bugs that I really love is if you have a bug, if you can reproduce it, and if you can reproduce it, then you can squash it. So a little bit of audience participation here. Does anybody know what this shape is? Okay, so I heard triangle. Any other? Tostal is running quite very sensitive. So I hate to tell you, but this is actually a square. This is known as a speed square. It's used in woodworking. It's used for making fast angles. Here you can see it making fast angles. And square edges. So for now, one for the rest of this talk, this will be a square. You've been informed. So what we're going to do is our own speed square. It's slightly different. And it looks something like this. We've got IICPU and RAM, and these are all resources that we can trade for speed. Similarly, we can trade one resource for another and we're overbooked online. So, okay, how do we make things fast in general? Well, first we're going to find a bottleneck. On the slide here, I highly recommend if you don't have an industrial engineering background, or even if you do, to check out this book called Gold by this person who I can't pronounce their name. So just google the goal in the book. And then that's all I'm going to say about that. Once we've found a bottleneck, or even if you find a bottleneck, we can use the scientific method. It's going to look a little bit something like this. We're going to touch on it later. So I kind of started off relatively quickly. For those of you who don't know me, my name is Richard Sheeman, or Sheen's. I really, really, really love Ruby. And as Keiwi Harman once famously said, if you love something, why don't you marry it? And so I'd like to introduce you to my wife Ruby. She is actually a pipeline programmer. It's like a house divided. And we can work, so we're really excited to announce that we're having a child. So, my baby's coming in June. We've been talking a lot about names. It's going to be a boy, it's going to be a girl. And I was like, all right, if it's a girl, we have to name her girl. I was just like, what kind of boy am I? I don't know, an asshole? And then we were like, well, no matter what, the middle name should be something we could both agree on. And we decided on, we decided on pseudo. We'd be like, pseudo Peter root. Pseudo Peter root. So I worked for Ruby on the Ruby Build Back among some other things. And I'm not going to have time for questions at the end of this, but we will have a booth that's going to be open up tomorrow. Please come by the booth. I will be there a lot. Also, Kujisasa has a talk tomorrow. Please go to that. It's going to be really good. Also, Terence Lee has a talk. So please check this out. Also, I'll be doing a book signing. I wrote a book. I don't be giving you count free copies. If you come by the booth, you can get some more information. So I also have Rails commit bit, and I've run a really small conference down in Austin called Ruby Rear. All by myself and not with Caleb Thompson sitting in the front row. Very tricky Ruby Rear chair right there. Okay. So I'd like to start out with a story. Once upon a time, there was an application. And I was working on this application. It had some requirements. It did some things. It downloaded a really large, large, easy file. It then depressed the file, and we did some work on that file. And that was pretty much it. That's the gist. That's really all you need to know. Was this thing fast? Does it sound fast? It does not sound fast. Taking a look at our speed square, we can kind of look at those individual things and say, okay, what resources are we using? It's like, okay, we're using IOTO. We're using IOTO and CPU. We're also using RAM and CPU for the processing. So first of all, I have a hypothesis, and I was like, okay, this thing isn't fast. Let's use some more CPU. Ruby has this really nice thing, this really neat feature that we launched, I don't know how you went, but relatively recently, called the PXDino. It's got 60 bytes of RAM, and one PXDino is actually an entire AWS instance, which is pretty cool. And the benefit of this is you get a lot, there's no sharing on it. We're not going to subdivide up that AWS instance into other smaller containers, so you get less noisy neighbors. Once we have this, I've now got a bunch more resources at my disposal, but now I need to take advantage of them. To do that, we can use more threads, and more threads is going to be more CPU load. Has anybody ever used Sidekick? Okay, cool. There's going to be a bunch of rhetorical questions. A lot of hand raising. Get that flood flowing. Okay, cool. More upgrades, so you're familiar with Sidekick background burgers. The first thing I did with this, I was like, oh, I have this huge box. I'm going to set Sidekick concurrency to something huge, like 30, and at the same time, we're going to be downloading, on zipping and processing, with 30 threads, kind of all at the same time. And when we did this, well, so it was definitely faster than we were sequentially doing it, but it was still pretty slow. I was like, man, why is that? My next theory was that we still had on-use CPU. I was looking at some metrics, and like CPU, we still had plenty of it. I was like, okay, great. What's changes from 30 up to 60? And does anybody know what happened? Not really. Okay, well, yeah, it got slower. When this happened, I was like, come on, you got to be kidding me. So it turns out that we weren't CPU bound, so we can strike that off. We go back to our little diagram and say, oh, what else do we have a lot of? What else are we doing a lot of? It turns out that we have a lot of I-O in this process. So there are different types of I-O. First, we have network I-O, so we're actually downloading the file. We're using the network adapter. On the second one, we're only using the disk. So my first theory was like, hey, networks are inherently slow, so that's got to be the problem. To test this out, I tried a bunch of different solutions. I tried a bunch of different libraries, curl, per, shelling out, I-O-Is. And then I came to try the thing that all Ruby users advocate when they're talking about making things fast. I heard caching is not quite really what everybody advocates is using Go. That is the way to make Ruby faster according to 90% of people on the internet. By the way, 70% of statistics in this talk are going to be made up of the spot, including this one. One of my co-workers has a really great utility called HTCat, and this is written in Go, and it will parallel download and stream large gift requests. It's really cool. There are other libraries like it. One more thing I need to talk about this one is people actually start piping it. It will start streaming to the pipe, similar like if you're going to cat a file, you can pipe that to something else. And it will start doing that immediately as soon as it can. Unfortunately for us, and maybe for a lot of people on Twitter, Go is not the answer, and network I-O is not the problem. So that kind of leaves us with really one thing. And that's disk I-O. In this case, it was pretty simple to test that out. So I had sci-tech concurrency of 30, and I changed that to 4. And then everything was like streaming fast, like really fast. Previously, we were not able to keep up with the queue, and then all of a sudden, it's like we've got spare cycles. And what actually happened was our hard drive, the disk was at max of read-write capacity. We were downloading that file and putting it onto the disk, and we were unzipping it and basically copying from one area to the disk to an even larger area to the disk. And we had all these threads just kind of sitting there idle. As if you had a dinner party of 30, and you only have four forks, and everybody's just grabbing that fork and trying to eat it fast at the end, but nobody can get anything done because there's a limiting resource. So in the end, we actually ended up using 2x diodes. And the moral of this story is that you can blindly try things all over your body, but it's not very scientific. It will work eventually, but maybe there's a little bit of a better way. So going back, I had a slide of the scientific method, and we could take a look and see here's the things that we kind of did. You heard me use the word theory and hypothesis, then I tried something, and then we took, we observed from that, and then kind of repeated the whole thing over and over and over again until we got to the results that we went in. Notice that we are sort of missing out on this one, on this research one, which is pretty critical. So one thing that is very, very important in all of science is that it's repeatable. Does anybody know who these two people are? Pleistopentopons. Pleistopentopons, wow, that's great. I was not expecting anybody to get that. Okay, so this is Pleistopentopons, and this is not cold fusion. So cold fusion is the idea that, so fusion is the thing that powers the sun, and the idea of cold fusion is that you could have that at temperatures less than the sun, which would be great, and it would provide a lot of energy. And these people claimed and actually published and said, hey, we've done it, we've done it, and unfortunately it wasn't repeatable, and lots and lots of people tried it, and it is good in the sense that it's kind of shown that this whole huge review thing can't work, and it does work. But they are, unfortunately, the kosher children of making these kind of hyperbolic scientific claims without actually using science. So they did not use good science. We only have better science than the best science. We are going to measure in benchmark. So, Groke did add some metrics. This is an older view of what our dashboard looks like. We explicitly allow speed, throughput, and then CPU and RAM. And the interesting thing for me is you can actually correlate and be like, oh, hey, you know, CPU used to shot up here, and then like throughput, you know, to Venosa or not. And this will tell you why you're slow or what exactly is going on, but it will give you a high-level metric that you can use to then ask the question and get a low-level metric. So that's kind of our basic research. Again, we want to reproduce the slow. So RAM is the one thing that we have a little touch on. Ruby is typically going to be RAM-bounded, and this is because Ruby is a managed language. So some of you might already know this, just bear with me for a little bit, but this is crucial to understanding the rest of my talk. So Ruby uses a garbage collector. And if you have questions about the garbage collector, can we just type it? It's currently here. It's great. We have a new hire team and a new guru, and that's got to work on MRI full-time, which is pretty cool. So the garbage collector is going to allow us to do things like just say, here's a string letter. I want a string without having to say, all right, we're going to allocate, you know, five bytes for this string. A lot of people are really familiar, intimately familiar with the different ways that Ruby can consume more memory. So one of the most obvious is what people think of when they think of memory use in Ruby is retaining the objects. So here we've got a loop, and we're just looping around. We've got this constant, constant, sorry, never garbage collector, they're global. You can't, you can't collect something that's global. And we're just adding a bunch of strings to it. And when we do this, and we take a look at the process memory, it's 7.32 gigabytes. It's kind of large. I mean, I guess it is 100 million, is that a million? Yeah, that's 100 million strings. So why is it so large? So Ruby allocates chunks of memory, and then in those chunks of memory, it has slots where it would put objects. So as we are going through and we're looping, it is going to put 1, 2, 3, 4, 5, 6, 7 into different slots. It needs one slot for each of those. Whenever we're out of those slots, we need to allocate more slots. So then now we have, we have twice in that, even you can think of this, I've chosen the bus analogy, and kind of a shipping container, really anything you can put things in. And so now we can actually go all the way up to 14, but we have to go up to 100 million. So we just keep on allocating more and more and more until we get enough. So this, to me, anyway, makes a lot of sense where we say, okay, Ruby's using a large amount of memory, and here's where it's coming from. We need this. It is a retained object. So the second way, which is a lot less intuitive, at least to me, is allocated objects. We're doing the exact same thing, except for instead of storing it as an array, we're just going to output it, output the string. We use it and throw it away. This is only going to take 21 megabytes, which makes sense. We're not actually doing anything. We fill up our bus. We are going to run our collection. Our architecture is going to say, like, hey, it doesn't look like you actually need any of these objects. 234567, guess what? You're not going to use it again. So we can get rid of it, and we can put another object there. And this is a very oversimplified version of garbage collection, but it will help us later. So it looks like garbage collection is going to save us from everything forever, and it's like the perfect answer to all things. If that was true, I wouldn't be up here. So here's another example that is kind of an in-between. So what we're going to be doing is we're going to be looping through a million elements again. We're going to be adding them to an array. The interesting thing is we're going to not return that array. We're not going to do anything with it. So it's not constant. Nothing has a reference to it. And then we can call garbage collection. We can force a garbage collection. So what happens when we run this? Well, your application is still going to use 7.32 gigabytes of RAM. It's just kind of surprising, right? It's like, well, the garbage collection could just get rid of all that stuff. It just throws it away. Well, to understand why, you can, more to, I guess, to verify this, improve this, you can take a look at your GC stat. I mean, we can look at total-free objects and we can verify and say, okay, the garbage collection is doing its job. It did free 100 million objects here. So why is it still so large? Why is our memory so large? So we can't clear a slot while a reference to that item still exists. So what's happening is we build up our buses and after we've done this, we've gotten to 14 and we need to keep going. We're going to run garbage collection and after the entire program has finished executing, yes, we can clear those slots. But Ruby never actually releases memory. Ruby only memory usage ever goes up. We basically assume that once we've allocated memory, we're going to need that for the future. Also, free memory to the operating system is an expensive operation. So even though we don't need those things anymore, we still are going to hold on to that memory. So it's important not necessarily that it's retained forever, but just that it's retained for a while. So that array actually, you referenced all those things. While the array was in memory, we couldn't clear them. So I do highly, highly recommend that you test everything. Even these slides, as I was writing these, I realized I was double checking everything and realized there's a giant mistake in them. So please, just if you're interested in getting started with performance stuff, double check me, verify and then when I've said that's good science, peer review. So the experiment is super, super critical in benchmark performance. Like never go in and be like, I think I know why something is slow. Please always double check. So, okay, back to Steve. How does this actually affect your Rails app? So you can kind of think of your Rails app as a collection of retained memory. These are going to be things that live forever, or forevery, like a Ruby world, such as like an abase connection or your high level Rails app, your controllers, any constants. Any code that you've loaded. So that's going to have a base set of memory. And on top of that, if they be larger, maybe smaller, we're going to kind of spike up and down to like a level on a sound board, this allocated memory. And our total memory usage is going to be the combination of both of those. So why this really affects you, it helps that we take a little task and be like, let's say we've got a bunch of memory and we're in a Rails app and we're in a Request Arts. So we're processing, we're getting database, we're processing, we're loading up templates, and we're processing and doing some other stuff, Turbos, Rockets, 3, we're processing. And so we're creating these objects and well, this request is still going, we still need them, so we can't clear them. So now we're out of memory, all of our slots are full, what do we do? Okay, well Ruby can run through the garbage collection, but it looks like we need to reference all of these objects. So we can allocate memory. So we add an extra bus onto our, onto our memory, and then finally we're done. We're done with the request, we can deliver the response, and we no longer need a lot of those objects. Granted, this is definitely an oversimplification, but it helps. Eventually, we can go back and we can say, alright, we're going to run garbage collection and we don't need those objects. We can just mark them, or we can just, we can just get rid of them. And we do, we get rid of them, but you notice that our, our bus count is still high. So Ruby is going to increase memory use without retention, even if you're not constantly retaining objects, even if that retention number stays the same. If you have one request that is loads of, you know, millions and millions and millions of objects, and only one person ever gets that request once in a day, guess what, your app is going to just be pegged of that memory usage, and you can, and charge that memory usage by the operating system for, until that process dies. So object creation takes, it takes a brand and it takes a time. I do also recommend checking out Generational GC. It's, don't have time to cover right here. So, okay, that was all relatively critical for a customer story that I have. One day, a ticket came into Ruby, and I mentioned I work on the Ruby, Ruby Build Pack. It's the source thing whenever you run, you get pushed to Ruby Master, and all that stuff flies by the screen. That's actually me sitting at a computer, like, curiously typing. So if anybody tries it right now, it won't work. Sorry. That's not entirely true. Terrence, Terrence, help us out. So, one of the things that we do is make recommendations, and we say use this web server, or do this other thing. In order to do that, we have to provide invigoration. And some of the things that we care about are some of the things that our customers care about, which makes sense. So one day, I think it came in, it regular support, and the customer said, you know, my app is slow, and the support will look at it, and you're like, I don't know, maybe it looks like a Ruby thing, like we'll escalate it up there. And so I get to look at escalated tickets. And they said the app is slow, I checked metrics. It looks something kind of like this. And you can see that the RAM they were using is just like way too high. So they're using a swap. That's this red bar. And that's actually indicating that we've used up all of the available RAM, and we're starting to use the disk, which is really, really slow. So this is what I was thinking is actually making their applications slow. The next question, of course, is going to be why. And obviously I asked the customer, and if you've ever worked in support, you've got answers like this. It used to happen. And we didn't change anything. It's totally you. And then I'm like, we haven't deployed anything in like a month. So whenever these types of things happen in the verbi, a lot of times what people don't realize is that their GEM file or the GEM file that lock changed one of their, or their Devs ran bubble update something, and a minor version or major version had some bugs in it, and they didn't realize that that was an issue, or even think to look into it. So my number one hypothesis was that RAM was increasing due to misbehaving code in a GEM somewhere. We don't know. In order to test this out, we want to, I want it to be able to boot the app, hit it with requests, and profile the memory. In order to do this, I wrote something called dvarel benchmarks. It allows you to actually do those things without having to run your server somewhere. You can do those in a single process. And it uses RAPMoc. And then we can do really neat things like wrap it in other tools like Stackpop and memory profile. So dvarel benchmarks is going to boot the app, it's going to process the request, and then, conveniently enough, we're going to use memory profiler by by Sam Saffron, who is at this conference. Please go, like, in my high five, and be like, your memory work is awesome. Also, Koichi Sassada has another GEM called allocation tracer, which helps as well. So in dvarel benchmarks, you can run bringing performance. And I have an application called code3aige.com. They help people get started with open source. It sends you one RAPMoc, one GitHub issue per day. And this is just something I run so that I can have something in production on the platform that I care about, and I give a real customer experience. So for those of you who weren't paying attention, that was code3aige.com. Code3aige.com. Sign up today. Okay. And so when I read this, I got something that looked like this. On every single request, here's all of the object counts of what we're using. And RAP, okay, yeah, you're using a bunch of RAP, you're using Azure Plag, that makes sense. Or it's like, whoa, hashi, where did hashi come from? Did you know me? You know I have certain feelings about hashi? Certain blog posts about hashi. And it was really remarkable to me that hashi is using, everybody gives active support a hard time. It's like more objects than active support. So hashi is incredibly expensive. And in the other case, he was using lots of unneeded objects. And I said, okay, I didn't put hashi in my project, I'm not using it. I look at my jump on the lock and apparently it turns out that omg uses hashi, which actually makes sense in the same offer. Hashi is the same offer of omg. It's like, oh, they use the same tooling. Originally, I went to a root hashi from omg. And the author of omg is like that might be a little heavy-handed. So I basically found some hotspots where, and we were not offing. This is every single request to any action in your entire application. Yeah, so I fixed it by just memorizing a couple items and instead of having to re-create the object, we just used the same one. So basically, we are using up more memory by retaining that, but ultimately creating less objects, which is less memory than using enough. So I did this and the application limit of our send over to the customer to try it out, the application is still over the RAM limit. So okay, hypothesis number two, we're still restricted to the gem file. Maybe it's something else. Maybe it's a bad gem using too much RAM that requires on. Okay, in order to do that, how can we test it out? I have a customer sending me the gem file and the gem file. And I was like, well, hey, how can we figure out which of these gems is, it's just like if the gem loads and retains its own code, how can we know about that? Maybe it's not something on every request. At the time, there was no tooling for benchmarking on require memory use. And so, let's write one. All right, the general concept is pretty simple. We're going to measure RAM before and after require is called. And we're going to be using a library called git process memory. And now one thing to note if you're like a super memory geek is that uses RSS. RSS stands for resident set size. And this is not memory. This is not, this is, does not take into account shared memory use. It is a closest approximation of memory. But in our case, it's probably good enough. Yeah, it doesn't take into account shared memory. The next thing we're going to do is we're going to lump patch for a little number of wire because we can. Yeah, lumpy patch because you can. Okay, and then we're going to run it and we're going to get out for like this. It's going to spit out every single file and the associated cost. So here it looks like real passionate active record costs 1.49 megabytes, which is a little large, but I don't think that's where our main problem is. Now that we know where all of our memory is, we can sort this and it looks like mail is using about 40 megabytes of memory, which is slightly larger than the old hatching aids 1.5. When we count it all up, it's one gem and it's 65% of all of the RAM at the time. And we haven't even done any work. We haven't like hit out of the quest or anything. Remember in that slide I had the routine on the bottom of this. This is the starting memory that you have to deal with and anything you have to do after that is just going to go up. So in order to debug further, we're going to have to dig deeper. Whenever you require one gem, it requires other files and we need to see where exactly inside of mail is causing that problem. So we're going to do, we're going to use a tree in order to do that. Each node can have many children. Each child has a cost. It's going to end up looking like this where mail will load one layer and that will load the next layer and that will load the next layer. We have a tree class that we're going to store these things in and then when we're done, we can sort the children and print them or personally. It's something that looks kind of like this. The final process looks like this. We're going to instantiate a new tree, a new tree node and this might be mail. So we've already required the application and now we're going to be requiring the mail gem. We are using a stack in order to keep track of where we are. So the last thing in the stack would be the application and that is going to be the parent and we're going to push the child node, which is mail, onto the parent. So we're saying that mail belongs to the application. Finally, we're going to add the mail gem and say, okay, now we're going to require everything inside of the mail gem. So we need to push that onto the stack. We take a measurement before we call require and this is actually what goes down and then this thing will be called again and again and again and again and again until all of mail has been fully required. When it's done, we pop the last thing off the stack, which at this point in time will be mail because everything else has been popped off the stack and we take a memory measurement again. Finally, we record the cost and store that. So one thing if you're interested in doing this yourself, it is really important to note the insure block. One thing I realized, well, I realized, but didn't fully sink in, was that a lot of people actually require files that don't exist. They expect that to raise that, a load error and if we're not insuring that we're popping something off the stack and reporting the memory, then in the event that we try to require something that's bad, then this code just won't run. It'll be bad. Okay, the final result of all of this looks something like this where we have, we can see sort of the tree structure, we see application requires mail, require mail parsers and we see, oh, it looks like mail parsers takes up 20 megabytes nearly, 19.24 megabytes. I've opened up an issue. I raise awareness. One thing I'd like to point out is you don't necessarily always have to fix the bottom that or performance bug. Whenever you find one, even just pointing out, hey, there is one, raising visibility sometimes can be just as important or is just as important. And I didn't fix this, but somebody else did in Mikhail mail number 817. Basically, they just said, so if you're not familiar, inside of mail, there's a parser that actually parses mail. And then you reply back to GitHub and you say, hey, this issue looks great and that gets parsed by probably Ruby code and they pull out what you said and drop the rest of the text from the summit and all that other stuff. And we'll actually generate new comments. So some applications do actually need a parser in there in mail switched over from using the tree top parser to a regular parser, which is much more efficient, much faster. It uses the trade off of more memory at that time. So if most applications don't use this, which is why our customer wasn't using this most applications are not parsing mail. So they don't they don't even need it. So the fix was actually just lazy load that or if somebody really wants to wants to load it and take advantage of copyright organizations and they can require. For your application works. So okay, what happened after you do this? Well, the customers ran each drop dramatically. It was no longer swapping and instead of just kind of crawling along, they were now back there like screaming back like giving me high lives over the internet and they're like great job. I was feeling really good about myself and you don't have to care about that as a way. I work at home alone. So that was really like me giving this a high five. And my desk. So okay, some of some of the takeaways is like we want to remember to reproduce the splonus. These are problems that are problems in this case, and it was fixed with code or to be fixed with a different choir versions of times. We want to definitely make sure to get visibility benchmarks and like in use this method and use this this process of repeatedly asking questions, digging deeper and and gathering more and more metrics. So kind of real quick. Has anybody here ever made a bad code deployed on a Friday? Okay, so you know this will happen and then you'll you'll like oh I'm going to deploy a lot fixed and you're like, oh I totally fixed it. And it's like, whoa. And then like your weekend just like gone and your coworkers weekend just gone and you have a sport and they're just like, ah, and he's like why didn't she deploy that. So this happens to everyone. Like whether you raise your hand or not, but it's kind of an eventuality that it will happen. So I have a much proposal. Don't deploy on Friday. Now instead you can go shopping or ride roller coaster. Okay, so that's not going to fly instead of just not doing any work on proposing something on calling Monday, Friday. If your application is slow, maybe instead of complaining about it, just like, oh, why is this thing so slow? You could look into it on Monday, Friday. You can run some benchmarks. You can maybe implement some of this. But if you're constantly complaining, you can't find files. You can't find methods. And the code base is a mess. It's like Monday, Friday. Or if you are like constantly having to stop your it's like, you're mostly all this feature has to be delivered tomorrow and they come over to your desk and you have like 1000 files open and like 10 branches checked out. And they're just like, what are you doing? And you're like, I couldn't stand it anymore. I'm just adding a refactor right now instead of doing that on Friday. A lot of times I hear companies wish more companies sponsored open source. And guess what? Your company just sponsored you to work on open source on Monday, Friday. If you're not sold, it's okay. This is actually not just something I'm randomly recommending. It's it will give you more accurate deadlines. It's a well-proven technique. It's called time boxing. It will hopefully lead to less burnout. And if your boss is never like, man, I really want to hire more senior developers. Like I don't know the boss who doesn't say that these days. And you'd be like, oh, you mean like developers that make revacaries and speed improvements and contribute to open source. You know, the library is really, really well. It's like, maybe you can have a stupid Monday, Friday and learn all of those things instead. So it's not just as simple as doing this. Unfortunately, it never is. I hate to tell you, but if you're going to participate on Monday, Friday, you have to report your progress to your team. You have to let other people know you're doing this. Report it to your team. Report it to your boss. You're going to be amazed. So I actually kind of started doing this without really telling anybody. And then after the fact, I would be like, hey, I made this patch to rack or to rails or like, look at this thing. And then now, like whenever somebody's a company's like, hey, I've got a question in rails. Like they come to me. My boss is like, hey, there's this critical thing that's failing for our customers. Why don't you like, you know, take a look at that. And I'm like, wow, this is amazing. It's like no longer, it's not even Monday, Friday. It's like Monday, Tuesday. And I get to work out of the course. I am like the luckiest guy in the world. So in addition to reporting your boss, please report to Twitter, hashtag Monday, Friday. There were a lot of other people just also randomly tweeting Monday, Friday, so it's okay. If you mentioned me, then I'll like, re-queen it. And hopefully if you see somebody doing this, like give them love and be like, even if they're like, hey, I spent eight hours looking at benchmarks and like gotten nowhere. It's like that's progress. And or they're like, or maybe they're like, I signed up for coachreage.com, coachreage.com, coachreage.com. And it is awesome. So please, please get love, get love. I know we're a little excited. We've had a lot of fun today. Thank you very much for coming. I want to leave off with two relatively serious questions. And this is silent meditation. Don't scream out any answers or raise your hand or anything. So, thank you very much. Thank you.
 Okay, so I think to disappoint everyone, the title is Thought Here to See Science.  We will not actually be talking about methamphetamine production, but we will be talking about science.  Instead, we will be talking about goals.  We will be talking about goals and our main goal is getting faster and achieving more.  One of the things I like to think about whenever I am dealing with any kind of performance problem is that it's the most important.  This is really comforting for me because mostly in my application I do bugs every day, every hour, every minute, you know, a lot.  And the great thing about bugs that I really love is if you have a bug, if you can reproduce it, and if you can reproduce it, then you can squash it.  So a little bit of audience participation here. Does anybody know what this shape is?  Okay, so I heard triangle. Any other?  Tostal is running quite very sensitive. So I hate to tell you, but this is actually a square.  This is known as a speed square. It's used in woodworking. It's used for making fast angles. Here you can see it making fast angles.  And square edges. So for now, one for the rest of this talk, this will be a square. You've been informed.  So what we're going to do is our own speed square. It's slightly different.  And it looks something like this. We've got IICPU and RAM, and these are all resources that we can trade for speed.  Similarly, we can trade one resource for another and we're overbooked online.  So, okay, how do we make things fast in general?  Well, first we're going to find a bottleneck. On the slide here, I highly recommend if you don't have an industrial engineering background,  or even if you do, to check out this book called Gold by this person who I can't pronounce their name.  So just google the goal in the book. And then that's all I'm going to say about that.  Once we've found a bottleneck, or even if you find a bottleneck, we can use the scientific method.  It's going to look a little bit something like this. We're going to touch on it later.  So I kind of started off relatively quickly. For those of you who don't know me, my name is Richard Sheeman, or Sheen's.  I really, really, really love Ruby. And as Keiwi Harman once famously said, if you love something, why don't you marry it?  And so I'd like to introduce you to my wife Ruby. She is actually a pipeline programmer.  It's like a house divided. And we can work, so we're really excited to announce that we're having a child.  So, my baby's coming in June. We've been talking a lot about names. It's going to be a boy, it's going to be a girl.  And I was like, all right, if it's a girl, we have to name her girl.  I was just like, what kind of boy am I? I don't know, an asshole?  And then we were like, well, no matter what, the middle name should be something we could both agree on.  And we decided on, we decided on pseudo. We'd be like, pseudo Peter root. Pseudo Peter root.  So I worked for Ruby on the Ruby Build Back among some other things.  And I'm not going to have time for questions at the end of this, but we will have a booth that's going to be open up tomorrow.  Please come by the booth. I will be there a lot.  Also, Kujisasa has a talk tomorrow. Please go to that. It's going to be really good.  Also, Terence Lee has a talk. So please check this out.  Also, I'll be doing a book signing. I wrote a book. I don't be giving you count free copies.  If you come by the booth, you can get some more information.  So I also have Rails commit bit, and I've run a really small conference down in Austin called Ruby Rear.  All by myself and not with Caleb Thompson sitting in the front row.  Very tricky Ruby Rear chair right there.  Okay. So I'd like to start out with a story.  Once upon a time, there was an application. And I was working on this application.  It had some requirements. It did some things. It downloaded a really large, large, easy file.  It then depressed the file, and we did some work on that file.  And that was pretty much it. That's the gist. That's really all you need to know.  Was this thing fast? Does it sound fast? It does not sound fast.  Taking a look at our speed square, we can kind of look at those individual things and say,  okay, what resources are we using? It's like, okay, we're using IOTO. We're using IOTO and CPU.  We're also using RAM and CPU for the processing.  So first of all, I have a hypothesis, and I was like, okay, this thing isn't fast. Let's use some more CPU.  Ruby has this really nice thing, this really neat feature that we launched, I don't know how you went,  but relatively recently, called the PXDino. It's got 60 bytes of RAM,  and one PXDino is actually an entire AWS instance, which is pretty cool.  And the benefit of this is you get a lot, there's no sharing on it.  We're not going to subdivide up that AWS instance into other smaller containers, so you get less noisy neighbors.  Once we have this, I've now got a bunch more resources at my disposal, but now I need to take advantage of them.  To do that, we can use more threads, and more threads is going to be more CPU load.  Has anybody ever used Sidekick? Okay, cool.  There's going to be a bunch of rhetorical questions. A lot of hand raising.  Get that flood flowing. Okay, cool.  More upgrades, so you're familiar with Sidekick background burgers.  The first thing I did with this, I was like, oh, I have this huge box.  I'm going to set Sidekick concurrency to something huge, like 30, and at the same time, we're going to be downloading,  on zipping and processing, with 30 threads, kind of all at the same time.  And when we did this, well, so it was definitely faster than we were sequentially doing it,  but it was still pretty slow. I was like, man, why is that?  My next theory was that we still had on-use CPU. I was looking at some metrics,  and like CPU, we still had plenty of it. I was like, okay, great.  What's changes from 30 up to 60?  And does anybody know what happened?  Not really.  Okay, well,  yeah, it got slower. When this happened, I was like, come on, you got to be kidding me.  So it turns out that we weren't CPU bound, so we can strike that off.  We go back to our little diagram and say, oh, what else do we have a lot of?  What else are we doing a lot of?  It turns out that we have a lot of I-O in this process.  So there are different types of I-O.  First, we have network I-O, so we're actually downloading the file.  We're using the network adapter.  On the second one, we're only using the disk.  So my first theory was like, hey, networks are inherently slow,  so that's got to be the problem.  To test this out, I tried a bunch of different solutions.  I tried a bunch of different libraries, curl, per, shelling out, I-O-Is.  And then I came to try the thing that all Ruby users advocate when they're talking about making things fast.  I heard caching is not quite really what everybody advocates is using Go.  That is the way to make Ruby faster according to 90% of people on the internet.  By the way, 70% of statistics in this talk are going to be made up of the spot, including this one.  One of my co-workers has a really great utility called HTCat, and this is written in Go, and it will parallel download and stream large gift requests.  It's really cool. There are other libraries like it.  One more thing I need to talk about this one is people actually start piping it.  It will start streaming to the pipe, similar like if you're going to cat a file, you can pipe that to something else.  And it will start doing that immediately as soon as it can.  Unfortunately for us, and maybe for a lot of people on Twitter, Go is not the answer, and network I-O is not the problem.  So that kind of leaves us with really one thing.  And that's disk I-O. In this case, it was pretty simple to test that out.  So I had sci-tech concurrency of 30, and I changed that to 4.  And then everything was like streaming fast, like really fast.  Previously, we were not able to keep up with the queue, and then all of a sudden, it's like we've got spare cycles.  And what actually happened was our hard drive, the disk was at max of read-write capacity.  We were downloading that file and putting it onto the disk, and we were unzipping it and basically copying from one area to the disk to an even larger area to the disk.  And we had all these threads just kind of sitting there idle.  As if you had a dinner party of 30, and you only have four forks, and everybody's just grabbing that fork and trying to eat it fast at the end, but nobody can get anything done because there's a limiting resource.  So in the end, we actually ended up using 2x diodes.  And the moral of this story is that you can blindly try things all over your body, but it's not very scientific.  It will work eventually, but maybe there's a little bit of a better way.  So going back, I had a slide of the scientific method, and we could take a look and see here's the things that we kind of did.  You heard me use the word theory and hypothesis, then I tried something, and then we took, we observed from that, and then kind of repeated the whole thing over and over and over again until we got to the results that we went in.  Notice that we are sort of missing out on this one, on this research one, which is pretty critical.  So one thing that is very, very important in all of science is that it's repeatable.  Does anybody know who these two people are?  Pleistopentopons.  Pleistopentopons, wow, that's great. I was not expecting anybody to get that.  Okay, so this is Pleistopentopons, and this is not cold fusion.  So cold fusion is the idea that, so fusion is the thing that powers the sun, and the idea of cold fusion is that you could have that at temperatures less than the sun, which would be great, and it would provide a lot of energy.  And these people claimed and actually published and said, hey, we've done it, we've done it, and unfortunately it wasn't repeatable, and lots and lots of people tried it, and it is good in the sense that it's kind of shown that this whole huge review thing can't work, and it does work.  But they are, unfortunately, the kosher children of making these kind of hyperbolic scientific claims without actually using science.  So they did not use good science. We only have better science than the best science.  We are going to measure in benchmark. So, Groke did add some metrics. This is an older view of what our dashboard looks like.  We explicitly allow speed, throughput, and then CPU and RAM. And the interesting thing for me is you can actually correlate and be like, oh, hey, you know, CPU used to shot up here, and then like throughput, you know, to Venosa or not.  And this will tell you why you're slow or what exactly is going on, but it will give you a high-level metric that you can use to then ask the question and get a low-level metric.  So that's kind of our basic research. Again, we want to reproduce the slow.  So RAM is the one thing that we have a little touch on. Ruby is typically going to be RAM-bounded, and this is because Ruby is a managed language.  So some of you might already know this, just bear with me for a little bit, but this is crucial to understanding the rest of my talk.  So Ruby uses a garbage collector. And if you have questions about the garbage collector, can we just type it? It's currently here. It's great.  We have a new hire team and a new guru, and that's got to work on MRI full-time, which is pretty cool. So the garbage collector is going to allow us to do things like just say, here's a string letter.  I want a string without having to say, all right, we're going to allocate, you know, five bytes for this string.  A lot of people are really familiar, intimately familiar with the different ways that Ruby can consume more memory.  So one of the most obvious is what people think of when they think of memory use in Ruby is retaining the objects.  So here we've got a loop, and we're just looping around. We've got this constant, constant, sorry, never garbage collector, they're global.  You can't, you can't collect something that's global. And we're just adding a bunch of strings to it.  And when we do this, and we take a look at the process memory, it's 7.32 gigabytes. It's kind of large.  I mean, I guess it is 100 million, is that a million? Yeah, that's 100 million strings. So why is it so large?  So Ruby allocates chunks of memory, and then in those chunks of memory, it has slots where it would put objects.  So as we are going through and we're looping, it is going to put 1, 2, 3, 4, 5, 6, 7 into different slots.  It needs one slot for each of those. Whenever we're out of those slots, we need to allocate more slots.  So then now we have, we have twice in that, even you can think of this, I've chosen the bus analogy,  and kind of a shipping container, really anything you can put things in.  And so now we can actually go all the way up to 14, but we have to go up to 100 million.  So we just keep on allocating more and more and more until we get enough.  So this, to me, anyway, makes a lot of sense where we say, okay, Ruby's using a large amount of memory,  and here's where it's coming from. We need this. It is a retained object.  So the second way, which is a lot less intuitive, at least to me, is allocated objects.  We're doing the exact same thing, except for instead of storing it as an array, we're just going to output it, output the string.  We use it and throw it away. This is only going to take 21 megabytes, which makes sense.  We're not actually doing anything. We fill up our bus.  We are going to run our collection. Our architecture is going to say, like, hey, it doesn't look like you actually need any of these objects.  234567, guess what? You're not going to use it again. So we can get rid of it, and we can put another object there.  And this is a very oversimplified version of garbage collection, but it will help us later.  So it looks like garbage collection is going to save us from everything forever, and it's like the perfect answer to all things.  If that was true, I wouldn't be up here.  So here's another example that is kind of an in-between.  So what we're going to be doing is we're going to be looping through a million elements again.  We're going to be adding them to an array. The interesting thing is we're going to not return that array. We're not going to do anything with it.  So it's not constant. Nothing has a reference to it.  And then we can call garbage collection. We can force a garbage collection.  So what happens when we run this? Well, your application is still going to use 7.32 gigabytes of RAM.  It's just kind of surprising, right? It's like, well, the garbage collection could just get rid of all that stuff. It just throws it away.  Well, to understand why, you can, more to, I guess, to verify this, improve this, you can take a look at your GC stat.  I mean, we can look at total-free objects and we can verify and say, okay, the garbage collection is doing its job.  It did free 100 million objects here. So why is it still so large? Why is our memory so large?  So we can't clear a slot while a reference to that item still exists.  So what's happening is we build up our buses and after we've done this, we've gotten to 14 and we need to keep going.  We're going to run garbage collection and after the entire program has finished executing, yes, we can clear those slots.  But Ruby never actually releases memory. Ruby only memory usage ever goes up.  We basically assume that once we've allocated memory, we're going to need that for the future.  Also, free memory to the operating system is an expensive operation.  So even though we don't need those things anymore, we still are going to hold on to that memory.  So it's important not necessarily that it's retained forever, but just that it's retained for a while.  So that array actually, you referenced all those things.  While the array was in memory, we couldn't clear them.  So I do highly, highly recommend that you test everything.  Even these slides, as I was writing these, I realized I was double checking everything and realized there's a giant mistake in them.  So please, just if you're interested in getting started with performance stuff, double check me, verify and then when I've said that's good science, peer review.  So the experiment is super, super critical in benchmark performance.  Like never go in and be like, I think I know why something is slow. Please always double check.  So, okay, back to Steve. How does this actually affect your Rails app?  So you can kind of think of your Rails app as a collection of retained memory.  These are going to be things that live forever, or forevery, like a Ruby world, such as like an abase connection or your high level Rails app, your controllers, any constants.  Any code that you've loaded. So that's going to have a base set of memory.  And on top of that, if they be larger, maybe smaller, we're going to kind of spike up and down to like a level on a sound board, this allocated memory.  And our total memory usage is going to be the combination of both of those.  So why this really affects you, it helps that we take a little task and be like, let's say we've got a bunch of memory and we're in a Rails app and we're in a Request Arts.  So we're processing, we're getting database, we're processing, we're loading up templates, and we're processing and doing some other stuff, Turbos, Rockets, 3, we're processing.  And so we're creating these objects and well, this request is still going, we still need them, so we can't clear them.  So now we're out of memory, all of our slots are full, what do we do?  Okay, well Ruby can run through the garbage collection, but it looks like we need to reference all of these objects.  So we can allocate memory. So we add an extra bus onto our, onto our memory, and then finally we're done.  We're done with the request, we can deliver the response, and we no longer need a lot of those objects.  Granted, this is definitely an oversimplification, but it helps.  Eventually, we can go back and we can say, alright, we're going to run garbage collection and we don't need those objects.  We can just mark them, or we can just, we can just get rid of them.  And we do, we get rid of them, but you notice that our, our bus count is still high.  So Ruby is going to increase memory use without retention, even if you're not constantly retaining objects, even if that retention number stays the same.  If you have one request that is loads of, you know, millions and millions and millions of objects, and only one person ever gets that request once in a day,  guess what, your app is going to just be pegged of that memory usage, and you can, and charge that memory usage by the operating system for, until that process dies.  So object creation takes, it takes a brand and it takes a time.  I do also recommend checking out Generational GC. It's, don't have time to cover right here.  So, okay, that was all relatively critical for a customer story that I have.  One day, a ticket came into Ruby, and I mentioned I work on the Ruby, Ruby Build Pack.  It's the source thing whenever you run, you get pushed to Ruby Master, and all that stuff flies by the screen. That's actually me sitting at a computer, like, curiously typing.  So if anybody tries it right now, it won't work. Sorry.  That's not entirely true. Terrence, Terrence, help us out.  So, one of the things that we do is make recommendations, and we say use this web server, or do this other thing. In order to do that, we have to provide invigoration.  And some of the things that we care about are some of the things that our customers care about, which makes sense.  So one day, I think it came in, it regular support, and the customer said, you know, my app is slow, and the support will look at it, and you're like, I don't know, maybe it looks like a Ruby thing,  like we'll escalate it up there. And so I get to look at escalated tickets.  And they said the app is slow, I checked metrics. It looks something kind of like this.  And you can see that the RAM they were using is just like way too high. So they're using a swap. That's this red bar.  And that's actually indicating that we've used up all of the available RAM, and we're starting to use the disk, which is really, really slow.  So this is what I was thinking is actually making their applications slow.  The next question, of course, is going to be why. And obviously I asked the customer, and if you've ever worked in support, you've got answers like this.  It used to happen. And we didn't change anything. It's totally you.  And then I'm like, we haven't deployed anything in like a month. So whenever these types of things happen in the verbi,  a lot of times what people don't realize is that their GEM file or the GEM file that lock changed one of their,  or their Devs ran bubble update something, and a minor version or major version had some bugs in it,  and they didn't realize that that was an issue, or even think to look into it.  So my number one hypothesis was that RAM was increasing due to misbehaving code in a GEM somewhere.  We don't know. In order to test this out, we want to, I want it to be able to boot the app, hit it with requests, and profile the memory.  In order to do this, I wrote something called dvarel benchmarks. It allows you to actually do those things without having to run your server somewhere.  You can do those in a single process. And it uses RAPMoc. And then we can do really neat things like wrap it in other tools like Stackpop and memory profile.  So dvarel benchmarks is going to boot the app, it's going to process the request, and then, conveniently enough, we're going to use memory profiler by  by Sam Saffron, who is at this conference. Please go, like, in my high five, and be like, your memory work is awesome.  Also, Koichi Sassada has another GEM called allocation tracer, which helps as well.  So in dvarel benchmarks, you can run bringing performance. And I have an application called code3aige.com.  They help people get started with open source. It sends you one RAPMoc, one GitHub issue per day.  And this is just something I run so that I can have something in production on the platform that I care about, and I give a real customer experience.  So for those of you who weren't paying attention, that was code3aige.com. Code3aige.com.  Sign up today. Okay.  And so when I read this, I got something that looked like this. On every single request, here's all of the object counts of what we're using.  And RAP, okay, yeah, you're using a bunch of RAP, you're using Azure Plag, that makes sense.  Or it's like, whoa, hashi, where did hashi come from?  Did you know me? You know I have certain feelings about hashi? Certain blog posts about hashi.  And it was really remarkable to me that hashi is using, everybody gives active support a hard time.  It's like more objects than active support. So hashi is incredibly expensive.  And in the other case, he was using lots of unneeded objects. And I said, okay, I didn't put hashi in my project, I'm not using it.  I look at my jump on the lock and apparently it turns out that omg uses hashi, which actually makes sense in the same offer.  Hashi is the same offer of omg. It's like, oh, they use the same tooling.  Originally, I went to a root hashi from omg. And the author of omg is like that might be a little heavy-handed.  So I basically found some hotspots where, and we were not offing.  This is every single request to any action in your entire application.  Yeah, so I fixed it by just memorizing a couple items and instead of having to re-create the object, we just used the same one.  So basically, we are using up more memory by retaining that, but ultimately creating less objects,  which is less memory than using enough.  So I did this and the application limit of our send over to the customer to try it out, the application is still over the RAM limit.  So okay, hypothesis number two, we're still restricted to the gem file. Maybe it's something else.  Maybe it's a bad gem using too much RAM that requires on.  Okay, in order to do that, how can we test it out? I have a customer sending me the gem file and the gem file.  And I was like, well, hey, how can we figure out which of these gems is, it's just like if the gem loads and retains its own code,  how can we know about that? Maybe it's not something on every request.  At the time, there was no tooling for benchmarking on require memory use. And so, let's write one.  All right, the general concept is pretty simple. We're going to measure RAM before and after require is called.  And we're going to be using a library called git process memory.  And now one thing to note if you're like a super memory geek is that uses RSS.  RSS stands for resident set size. And this is not memory. This is not, this is, does not take into account shared memory use.  It is a closest approximation of memory. But in our case, it's probably good enough.  Yeah, it doesn't take into account shared memory.  The next thing we're going to do is we're going to lump patch for a little number of wire because we can.  Yeah, lumpy patch because you can. Okay, and then we're going to run it and we're going to get out for like this.  It's going to spit out every single file and the associated cost.  So here it looks like real passionate active record costs 1.49 megabytes, which is a little large, but I don't think that's where our main problem is.  Now that we know where all of our memory is, we can sort this and it looks like mail is using about 40 megabytes of memory,  which is slightly larger than the old hatching aids 1.5.  When we count it all up, it's one gem and it's 65% of all of the RAM at the time.  And we haven't even done any work. We haven't like hit out of the quest or anything.  Remember in that slide I had the routine on the bottom of this.  This is the starting memory that you have to deal with and anything you have to do after that is just going to go up.  So in order to debug further, we're going to have to dig deeper.  Whenever you require one gem, it requires other files and we need to see where exactly inside of mail is causing that problem.  So we're going to do, we're going to use a tree in order to do that.  Each node can have many children. Each child has a cost.  It's going to end up looking like this where mail will load one layer and that will load the next layer and that will load the next layer.  We have a tree class that we're going to store these things in and then when we're done, we can sort the children and print them or personally.  It's something that looks kind of like this.  The final process looks like this. We're going to instantiate a new tree, a new tree node and this might be mail.  So we've already required the application and now we're going to be requiring the mail gem.  We are using a stack in order to keep track of where we are.  So the last thing in the stack would be the application and that is going to be the parent and we're going to push the child node, which is mail, onto the parent.  So we're saying that mail belongs to the application.  Finally, we're going to add the mail gem and say, okay, now we're going to require everything inside of the mail gem.  So we need to push that onto the stack.  We take a measurement before we call require and this is actually what goes down and then this thing will be called again and again and again and again and again until all of mail has been fully required.  When it's done, we pop the last thing off the stack, which at this point in time will be mail because everything else has been popped off the stack and we take a memory measurement again.  Finally, we record the cost and store that.  So one thing if you're interested in doing this yourself, it is really important to note the insure block.  One thing I realized, well, I realized, but didn't fully sink in, was that a lot of people actually require files that don't exist.  They expect that to raise that, a load error and if we're not insuring that we're popping something off the stack and reporting the memory, then in the event that we try to require something that's bad, then this code just won't run.  It'll be bad.  Okay, the final result of all of this looks something like this where we have, we can see sort of the tree structure, we see application requires mail, require mail parsers and we see, oh, it looks like mail parsers takes up 20 megabytes nearly, 19.24 megabytes.  I've opened up an issue.  I raise awareness.  One thing I'd like to point out is you don't necessarily always have to fix the bottom that or performance bug.  Whenever you find one, even just pointing out, hey, there is one, raising visibility sometimes can be just as important or is just as important.  And I didn't fix this, but somebody else did in Mikhail mail number 817.  Basically, they just said, so if you're not familiar, inside of mail, there's a parser that actually parses mail.  And then you reply back to GitHub and you say, hey, this issue looks great and that gets parsed by probably Ruby code and they pull out what you said and drop the rest of the text from the summit and all that other stuff.  And we'll actually generate new comments.  So some applications do actually need a parser in there in mail switched over from using the tree top parser to a regular parser, which is much more efficient, much faster.  It uses the trade off of more memory at that time.  So if most applications don't use this, which is why our customer wasn't using this most applications are not parsing mail.  So they don't they don't even need it.  So the fix was actually just lazy load that or if somebody really wants to wants to load it and take advantage of copyright organizations and they can require.  For your application works.  So okay, what happened after you do this?  Well, the customers ran each drop dramatically.  It was no longer swapping and instead of just kind of crawling along, they were now back there like screaming back like giving me high lives over the internet and they're like great job.  I was feeling really good about myself and you don't have to care about that as a way.  I work at home alone.  So that was really like me giving this a high five.  And my desk.  So okay, some of some of the takeaways is like we want to remember to reproduce the splonus.  These are problems that are problems in this case, and it was fixed with code or to be fixed with a different choir versions of times.  We want to definitely make sure to get visibility benchmarks and like in use this method and use this this process of repeatedly asking questions, digging deeper and and gathering more and more metrics.  So kind of real quick.  Has anybody here ever made a bad code deployed on a Friday?  Okay, so you know this will happen and then you'll you'll like oh I'm going to deploy a lot fixed and you're like, oh I totally fixed it.  And it's like, whoa. And then like your weekend just like gone and your coworkers weekend just gone and you have a sport and they're just like, ah, and he's like why didn't she deploy that.  So this happens to everyone.  Like whether you raise your hand or not, but it's kind of an eventuality that it will happen.  So I have a much proposal.  Don't deploy on Friday.  Now instead you can go shopping or ride roller coaster.  Okay, so that's not going to fly instead of just not doing any work on proposing something on calling Monday, Friday.  If your application is slow, maybe instead of complaining about it, just like, oh, why is this thing so slow?  You could look into it on Monday, Friday.  You can run some benchmarks.  You can maybe implement some of this.  But if you're constantly complaining, you can't find files.  You can't find methods.  And the code base is a mess.  It's like Monday, Friday.  Or if you are like constantly having to stop your it's like, you're mostly all this feature has to be delivered tomorrow and they come over to your desk and you have like 1000 files open and like 10 branches checked out.  And they're just like, what are you doing?  And you're like, I couldn't stand it anymore.  I'm just adding a refactor right now instead of doing that on Friday.  A lot of times I hear companies wish more companies sponsored open source.  And guess what?  Your company just sponsored you to work on open source on Monday, Friday.  If you're not sold, it's okay.  This is actually not just something I'm randomly recommending.  It's it will give you more accurate deadlines.  It's a well-proven technique.  It's called time boxing.  It will hopefully lead to less burnout.  And if your boss is never like, man, I really want to hire more senior developers.  Like I don't know the boss who doesn't say that these days.  And you'd be like, oh, you mean like developers that make revacaries and speed improvements and contribute to open source.  You know, the library is really, really well.  It's like, maybe you can have a stupid Monday, Friday and learn all of those things instead.  So it's not just as simple as doing this.  Unfortunately, it never is.  I hate to tell you, but if you're going to participate on Monday, Friday, you have to report your progress to your team.  You have to let other people know you're doing this.  Report it to your team.  Report it to your boss.  You're going to be amazed.  So I actually kind of started doing this without really telling anybody.  And then after the fact, I would be like, hey, I made this patch to rack or to rails or like, look at this thing.  And then now, like whenever somebody's a company's like, hey, I've got a question in rails.  Like they come to me.  My boss is like, hey, there's this critical thing that's failing for our customers.  Why don't you like, you know, take a look at that.  And I'm like, wow, this is amazing.  It's like no longer, it's not even Monday, Friday.  It's like Monday, Tuesday.  And I get to work out of the course.  I am like the luckiest guy in the world.  So in addition to reporting your boss, please report to Twitter, hashtag Monday, Friday.  There were a lot of other people just also randomly tweeting Monday, Friday, so it's okay.  If you mentioned me, then I'll like, re-queen it.  And hopefully if you see somebody doing this, like give them love and be like, even if they're like, hey, I spent eight hours looking at benchmarks and like gotten nowhere.  It's like that's progress.  And or they're like, or maybe they're like, I signed up for coachreage.com, coachreage.com, coachreage.com.  And it is awesome.  So please, please get love, get love.  I know we're a little excited.  We've had a lot of fun today.  Thank you very much for coming.  I want to leave off with two relatively serious questions.  And this is silent meditation.  Don't scream out any answers or raise your hand or anything.  So, thank you very much.  Thank you.
Speed_Science_0_155.png Speed_Science_156_1535.png Speed_Science_1536_1595.png Speed_Science_1596_2111.png Speed_Science_2112_2291.png Speed_Science_2292_4031.png Speed_Science_4032_4187.png Speed_Science_4188_5651.png Speed_Science_5652_6491.png Speed_Science_6492_6971.png Speed_Science_6972_7679.png Speed_Science_7680_8747.png Speed_Science_8748_9143.png Speed_Science_9144_10235.png Speed_Science_10236_10643.png Speed_Science_10644_10883.png Speed_Science_10884_11111.png Speed_Science_11112_11231.png Speed_Science_11232_12035.png Speed_Science_12036_12971.png Speed_Science_12972_13763.png Speed_Science_13764_14531.png Speed_Science_14532_19775.png Speed_Science_19776_21155.png Speed_Science_21156_22475.png Speed_Science_22476_22535.png Speed_Science_22536_23819.png Speed_Science_23820_26483.png Speed_Science_26484_29015.png Speed_Science_29016_30011.png Speed_Science_30012_30695.png Speed_Science_30696_33179.png Speed_Science_33180_34391.png Speed_Science_34392_34775.png Speed_Science_34776_35483.png Speed_Science_35484_35651.png Speed_Science_35652_36467.png Speed_Science_36468_36575.png Speed_Science_36576_36839.png Speed_Science_36840_38255.png Speed_Science_38256_39839.png Speed_Science_39840_41459.png Speed_Science_41460_41579.png Speed_Science_41580_42395.png Speed_Science_42396_43463.png Speed_Science_43464_43655.png Speed_Science_43656_44495.png Speed_Science_44496_44543.png Speed_Science_44544_45371.png Speed_Science_45372_45875.png Speed_Science_45876_47711.png Speed_Science_47712_49139.png Speed_Science_49140_49859.png Speed_Science_49860_51119.png Speed_Science_51120_51671.png Speed_Science_51672_52535.png Speed_Science_52536_53399.png Speed_Science_53400_53627.png Speed_Science_53628_54023.png Speed_Science_54024_55343.png Speed_Science_55344_55487.png Speed_Science_55488_55643.png Speed_Science_55644_55775.png Speed_Science_55776_56435.png Speed_Science_56436_56543.png Speed_Science_56544_56651.png Speed_Science_56652_57275.png Speed_Science_57276_60851.png Speed_Science_60852_61235.png Speed_Science_61236_61407.png