Facial Hair & the Road to Faster JavaScript
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Fillmore, Joel
Ingebretsen, Robby
For far too long the web has gone without a fast, reliable framework to render fake mustaches. The handlebar, mutton chop, and soul patch will guide us in our journey to optimize canvas and JavaScript performance. We’ll discuss our pragmatic approach to performance optimization and walk through real world challenges we solved while developing HTML5 games like Cut the Rope and Contre Jour.
 Hey, guys. Thanks for sticking around for a very, this will be the, well, at least the very last talk of the day. It's not the best. So, hopefully the best. So, my name is Robbie Ingebrezen. Do you guys recognize Ingebrezen as a Norwegian name? I've never asked this in a, when I've spoken before, but is there anybody here who can tell me how to pronounce my name? How do you do it? Okay, I'm not even going to try. So, my great grandfather was from, and this I will get wrong, but it's SKIEN in Telomark. Cheyenne, I think? Cheyenne? Are you, is anybody from Cheyenne? Do you know Ingebrezen's there? Dang it. We, I just went down there actually on this trip and looked for traces of my family. There's a cemetery there, like the main cemetery, and there's two actually, but the big one, like in the downtown area, and we found a whole bunch of Ingebrezens, which may or may not be relatives, but I hope they are. Anyway, so, it's fun for me to be in Norway where I can pronounce my, say my name and people don't have to look at me in a weird way. So, I'm Robbie Ingebrezen, and then this is Joel Fillmore, my good friend and colleague, and together we are a small company called Pixelab. We're based out of Seattle, although when you're just two people, you can kind of be based out of wherever your living room is that day. So, this week we're based out of Oslo. So, that's another connection that we have to Norway. So, we are focused primarily on HTML5 and JavaScript and sort of the, a web stack, although both of us have history with Microsoft technologies. I was part of the WPF and SilverEye teams. Any Microsoft guys here? Couple? Yay. Does anybody use Kazaml, by chance? It's a little, okay. So, Kazaml is my, I wrote that. And then Joel was on the SharePoint team at Microsoft. So, two really exciting products that brought us together to work on HTML5 applications. And actually we've had pretty good success with HTML5 stuff. We, in fact, we just won a Webby last week for a project that we did with Red Bull. And we're going to talk about some of the other projects that we've done. But our kind of focus in our, I guess our area of interest is sort of forward looking kinds of HTML5 experiences that sort of push the envelope and what you can do inside of a browser. And that's why performance is kind of a natural fit. And so, for that reason, we thought it would be fun to talk about performance and share some of the things that we've learned about JavaScript performance. And that'll be the focus of the talk today. Tomorrow we have another talk, though, where we're talking more about some of the creative aspects of using HTML5 to create forward looking experiences. And that is called the future of extreme web browsing. And I think it's like tomorrow at, anyway, just look it up. Come, though. It's tomorrow at 10 or 11 somewhere around there. So we'd love to see you guys there, too. So this is us. If you want to track us down, thinkpixelab.com. And then that is my Twitter handle, which you are the only country in the world that will remember that. So I don't need to spend too much time there. And we are Canuthians. And I will let Joel tell you what we mean when we say that. All right. So who is Canuth? This guy. So some of you may recognize him. He's a famous computer scientist. He's a professor emeritus at Stanford University. Most people have heard of Donald Canuth from the series of books that he's written, the most famous of which is the Art of Computer Programming. So this is the series of books that every serious programmer has on his bookshelf, but it's never read. It's pretty daunting. I've paged through it a few times, but have not read the whole thing. Donald Canuth is famous with this particular set of books, and I think his others as well. If anyone found an error in the book, he promised to send them a reward. And the reward was a hexadollar, which is 256 cents. And so he would write out the check for a hexadollar. And to date, people take a lot of pride in sort of cashing these checks. And he's written over $20,000 worth of checks, but supposedly very few of those have been cashed because they're kind of like a badge of honor. So he's famous for a number of quotes. We gathered a few of our favorites. This one is Beware of Bugs in the Above Code. I've only proved it correct, not tried it. That's pretty impressive when you have the mental compiler or the mental theorem prover to prove the code correct. The next one is Science is What We Understand Well Enough to Explain to a Computer. Art is Everything Else We Do. This is Robbie's Favorite. I Can't Go to a Restaurant and Order Food because I keep looking at the fonts on the menu. And this is great. You know that Donald Canuth was also the inventor of latex. This is the performance one that we're going to be talking about today. The quote is, we should forget about small efficiencies, say about 97% of the time. Premature optimization is the root of all evil. So when I was beginning my career at Microsoft, I was a new developer. One of the first tasks I was given was to implement a change to a series of data structures that we had to manipulate in a certain way. And so I wrote some code and I was really sort of new at the time. I was worried that it had to be fast. We were working on a server product where lots of concurrent requests could come in. And I was really careful to optimize this so that it would filter down the set of results in an efficient way. And I checked the code in and the next day I noticed that one of my colleagues had basically undone everything that I had written. And so I sort of mentioned, hey, did you, were there problems in what I had written? He said I had to make a few modifications and it was really difficult to understand. And so that particular piece of code we don't have to worry about from a performance perspective because there are so few items in that list. So I rewrote it so that it's a little simpler to understand. And he said, you know that quote, premature optimization is the root of all evil. I didn't know the quote. But I went and looked it up. And it was sort of a lesson to me that we shouldn't focus on optimization until we're absolutely sure it's necessary. So our kind of approach and what we would like to talk about today is the balance between optimizing code and maintaining code. So on one hand you have maintenance where you want code to be easy to understand for people that have to come in and maintain it. You want it to be efficient to write. And you also want it to be optimized for execution. And those are competing interests. So today we're going to talk a little bit about how to balance those interests. The rest of the quote we started off, the beginning is we should forget about small inefficiencies, about 97% of the time. Premature optimization is the root of all evil. The remaining part of the quote is also important. Yet we should not pass up our opportunities in that critical 3%. So that really is the key. It's to use the tools and methods we have to focus on the things that are most important. And for the rest of it, use the principles that guide us in writing easy to maintain, easy to write code. Okay. So we're going to talk about three different things. The first is tools. So we're going to talk about how tools can guide that process to find the 3% that is worth optimizing. Next we'll talk about an approach-based method where looking for different ways to optimize code rather than focusing in on the specific optimizations of a one chunk of code, stepping back and looking at the algorithmic or the method approach to optimizing code. And then third we're going to talk about the process of experimentation and refining. It's an iterative process to get your code in good state. Okay. Real-world examples. The first one we're going to talk about is a game called Cut the Rope. We did, so Cut the Rope, I'm sure many of you are familiar with, it's a very popular iOS, Android game. We did the HTML5 port for Cut the Rope a few years ago. And it was one of the first big, really challenging HTML5 projects that we did. Before we get too far into the details, for those of you who aren't familiar with Cut the Rope, we'll give you a quick video intro. Cut the Rope is a casual game which was initially released for mobile devices. It's essentially a game where you have to deliver candy to the little green monster called Omnom by cutting the ropes. Currently it has been distributed on two platforms and it totally has been downloaded more than 60 million times. It's quite successful. We had this basic concept of delivering something from point A to point B in the very beginning of the game. That's when we got the idea of candy being fed to the little green monster. And that idea was absurd and adorable at the same time. So we thought that it's a good way to go. We have more than 6 million play sessions every day. When we want to expand and HTML5 is just another great platform to expand to, it will bring us much more users, much more players. If any of you guys played Cut the Rope? Okay, so a lot of people. Anybody played on the web? My chance? Awesome. Well, yeah, so that was the project that we did. And it was a big project. Yes, I think when we first started out they gave us the Objective-C code base. It was about 150 files, 15,000 lines of code. They had written their own custom physics engine. So that was a little daunting to see all that code giving a relatively short project timeline to get that ported. The result was about 1.2 megs of JavaScript unminified. It got much better minified. But you kind of get the sense for what a daunting task this was. During the initial development of the project, we did some early prototypes. So we took the ropes, which are sort of the premier element in Cut the Rope. You want those to look amazing. And we wanted to see if we could do one rope. And so the challenges with the ropes are that they used polygons to render those, and they had some pretty complex calculations to do busier curves to make sure that the ropes felt lifelike so that they would swing, the physics felt right, and they looked beautiful. So on the left, you're going to see one of our early prototypes. So this was sort of our first attempt. We ported the code, used the same approach that they did, and this is the result. So you can see that the rope is a little slow. What you're seeing is definitely not 60 frames a second. You can see as the rope sort of curls down, there's some artifacts sort of at the kink where it was difficult to render those polygons in a small amount of area. And on the right is the final version of the game. And so this is sort of one section of the game. It's still a single rope. But there's much more going on in this. You can see that the spider is traversing the path of the rope, which is changing according to the physics engine. The sort of blower up at the top is sending an impulse of wind, which affects the candy, the rope, the path the spider's on. So it's significantly more complicated. There's a lot of elements. So our concern was how do we go from the prototype on the left where we're barely like rendering a rope. It doesn't look great to the full game where we have multiple ropes, multiple game objects, lots of animation, lots of forces going on. And so that was a real concern and challenge. And we weren't sure that we were going to be able to do it. So let's talk about how we did it. The first thing that we did is fire up our profiler and memory debugger. And this was sort of the profile that we got. So you can see that there's actually not that much memory being used. This is over a relatively short period of time. So I think this total period is two seconds. And what you're seeing with that sawtooth pattern is the number of memory allocations go up, and then the garbage collector comes in and goes down. And so we've got at least two or three garbage collections per second there. So sort of the quick intro to how garbage collection works. We've got, well, we're from Seattle, and Seattle drinks a lot of coffee. So the easy way to understand garbage collection is if you had a coffee mug and you had a disposable coffee cup, and imagine you needed to drink, I don't know, 20 cups of coffee every 16 milliseconds. Yeah, 16 milliseconds to draw a frame. And so every sort of object that you allocate has to be collected at some point, right? So maybe it's not that bad if you drink 20. If you drink maybe 100, that's a little more work for the garbage collector. It's going to require more processing time, JavaScript is single threaded. So typically they'll pause and collect the garbage and then let it resume. And it just gets worse and worse. So you can imagine how bad it got. We did the profiler. And there's a couple interesting things in this profiler. Typically when I'm profiling an app, I will sort by the exclusive time. So I'll look at functions where a lot of work is being done. And those are up at the top pretty obvious. So there's bezier calculations, so that's calculating the curves on the rope. There's the constraint engine, which is the physics and the objects and how those interact. Some more bezier calculations. And you can see sort of the difference that the typical approach with JavaScript is methods are underscores or lower case, lower camel case, and objects are uppercase. So there's sort of two objects in there. You can see the vector and the array. So they're not necessarily at the top of the list. They're not taking the most amount of time. They are definitely in the top 10, but they're not the highest. But the count right there is pretty important. So you can see over that two or three second period that we allocated 2.2 million vector objects. That's a lot of vectors. And so you can imagine the amount of pressure that puts on the garbage collector, which has to go in there and collect all those objects that are no longer being used. Similarly with the arrays, that's not as bad as the vector, but 262,000 arrays is pretty significant. So after the profiling, we were able to get this down. So sort of the end game, so you can see the difference from that initial prototype where we had one rope that was barely rendering to the full game. We went from 2.2 million vector allocations down to about 30,000, which is still pretty high. But given the amount of calculations and they're going on the game, it's much, much better. The array allocations we got down significantly as well, 262,000 down to just 400. So how did we do that? We're going to go through a little bit of code. This is the intensive part of the talk. And we'll show you a simple example of how we were able to get rid of those allocations. So on the left, we have the native implementation of a vector. So it's a struct. Those are relatively lightweight. You have some utility methods that are static to add and multiply vectors. Those are pretty straightforward. You add the X and Y coordinate or you multiply by a scalar. And then we have the direct port. So this is sort of our naive direct port implementation of a vector where we would do the same thing. We had a static or a utility method to add two vectors and the same thing for multiply. And so the key here is that new. You're allocating a new vector every time you need to add. So the vectors in this example are used by a game object. So a game object could be omnum. It could be the candy. It could be a piece of the rope. It could be any element in the game. And so it's pretty common for a game object to have a position within the screen. And forces will act upon that game object to move it around. So in this case, we apply an impulse to an object. And basically what that does is it takes the delta or the time slice that has occurred since the last frame was rendered and figures out the distance that the object should move based on the impulse that was applied to that object. So you can see that we're basically doing two things. We're multiplying the impulse force by the time slice to get the offset. And then we're adding that to the position. So that would be basically moving a character according to some speed. So here's our first attempt at reducing the number of allocations. We realized quickly that we're changing the position of a game object or a game element. We don't need to allocate a new vector for that, right? Because he just needs to know his position. We don't need to create a new element and then sign that new element. That's just asking the garbage collector to do extra work to collect that extra element. So the first thing we did was instead of a static method to add a vector which has the new inside there, we created an instance method which adds the X and Y coordinates of another vector to the existing vector. And so this was perfect for the position because the object only needs to keep track of its position. And we move the object according to that impulse. There's no vector added. So we're able to get rid of quite a few vector allocations using techniques like that. So here's the multiply. We can't do the same thing for the multiply though, right? If we were to multiply the impulse vector in place, then it changes the speed of the object. You can't do that. So the strategy there is to use local variables and just do the calculation in place. And this is, again, going back to the quote that we talked about at the beginning, not something that we would do in every place where a vector is used. And similarly, if you have a convenience method that adds a vector, it's okay to use that in the vast majority of cases. You want to use the profiler to guide your optimization and find the chunks of code that are path critical and that are really constraining the frame rate. And so once you find those, you can use the techniques. That's the 3% to really make them run quickly. For extremely critical paths, we found that inlining the function in the same way that the native compiler would do with the inline hint, it would actually inline the code. You can do the same thing with JavaScript by taking simple functions and avoiding the apply impulse function call entirely and just doing the calculations in place. And the results of that, you can kind of see, this slide is a little bit outdated, but you can see the direct port is in blue. The reusing the same vector after we made that change, we got a little bit better performance. Switching to local variables, performance went up significantly. And inline code where we completely avoided any function call was obviously the best. This was done a couple years ago. We tried it just out of curiosity last night and it's really incredible to see the difference that today's browsers have. It was just astronomical the difference. We wouldn't have found the problem in today's browser because the browser is so fast that it hides the problem. It was kind of crazy to see that because we only cut the rope two and a half years ago or something. It wasn't that long ago. I kind of had this feeling that two years ago browsers got really, really fast and then we've just sort of been living the dream. But it turns out that there's been a huge amount of improvement in what a browser, especially the JavaScript engines, across all browsers are doing right now. It would be interesting if you had a chance to go in and compare old browsers versus new browsers. Yeah. JavaScript engines today are really incredible. So we talked about the memory profile at the beginning and you see the sawtooth pattern. After the changes, you can see the difference that reducing those allocations makes. So it's much smoother. And what that translates into is a higher frame rate. So by the time we were able to ship, we were getting 60 frames a second on most devices and we were really, really happy with those results. So Joel did a great job of kind of explaining, I guess, one of the first approaches to performance should be, which is you see a performance problem and you can mostly, as you're writing your code, like Joel said, you can mostly ignore it, right? Don't optimize until you know there's a problem. Then as the problems begin to emerge, and actually, there's a little caveat with that, obviously. We're assuming that you're writing reasonable code as you go, right? Using best practices. But I guess as we're talking about sort of that tension between writing optimized code versus writing readable code, we're saying that it's okay to write readable, reusable, understandable code rather than focusing on the highly optimized code until you see that there's a problem. Now when you see that there's a problem, Joel did a great job describing one of the first approaches you can take, which is to actually look at your code and use the tools to find the hotspots and then once you find the hotspots, then you can address those directly. But there are times when the code just sort of can't be optimized. Like you're trying to accomplish something where the approach actually doesn't allow you to get fast enough. And we ran into this with a project for a game called Contre Jure. Has anybody played this game? It's a little bit less popular than Cut the Rope. It's a super cool game. So those of you who have played it can attest to that. It's really just a super beautiful game. And to understand sort of the pressure we felt in terms of performance, you have to understand who created it. So we wanted to just take a couple of minutes and show you a video. This guy named Max, and I actually don't know his last name, he's a Ukrainian guy and I guarantee this guy is like on the front line right now. But he is dedicated to everything he does. And we had sort of, I guess, the privilege and also the challenge of trying to help import his game to the web. So this is just three minutes about it. Belief in a third world country. You know, the main force that pushes Ukraine forward right now in software development. If it's about gaming scene, it's the stage of early development. We cannot officially buy an Xbox or any Xbox games because we don't have Xbox live in Ukraine. If I create a game for Xbox live, I will have no possibility to buy it. Crazy. My name is Max and I'm head of Mockers games studio. When I started to create games, post match games, it started different market compared to the mobile or Xbox PC, etc. You cannot make a lot of money on a flash market and you need some money to buy pizza and to buy a few for my motorcycles. When I started, I did one game, one month and release. It is how it is done in the flash world. Now everything changes. We have downloadable markets, we have upstores, we have mobile market and it's much easier to create a product for mobile market. When I started to create ContraJewel, I realized that it's something bigger. So I decided not to release it on a flash platform, but to try to make it mobile. Right now, after release of ContraJewel, we are in financial point of view. We are just free. It's very comfortable when you can wake up, just have a few steps and you are at work. For me, everything is the same. It's my work, here are my friends and it's my hobby, it's what I like to do, so it's everything together. A few years ago, I think it was impossible in Ukraine. We had no developed game and scene, now everything changes. Do you think you will always be making games? No, definitely no. If you would ask me the same question, 10 years ago you would ask Max, do you think you would be breakdancing all of your life? Yeah, 10 years ago maybe the answer would be yes, but right now I am a game developer. This gym is 4 times for a week and I have another gym just for breakdancing. The second one is 3 times for a week, so one training for a day. It's actually a gymnastics gym. I do some tricking, some parkour. The floor is not the best here for breakdancing. Actually, it's for gymnastics. I'm not in the best fit, but I still can do something. I do this because I have fun. I enjoy it. Even if I get older and I cannot, for example, win some battles no more, but I still enjoy it. When you work, you work with your brain. You challenge mainly your mind. Here you can challenge your body and when I go out from the training, my mind is totally cleared. It's like a reboot. In the former Soviet Union, we had no breakdancing, no modern street dances. I got very interested and I wanted to try. We stopped it there because it just gets more awesome. That guy is, honestly, we have a huge amount of respect for him and also we know that he could beat us up. When we inherited this project from another agency, this is what he was facing and he was just not happy at all about the way that they were rendering. They call these plasticines. You can see there's like the little, like over, well, I don't know what he's talking about. You can see that there's the native one on the left-hand side and then on, or I guess that would be, on your right-hand side. Then on the left-hand side is what they were rendering inside of HTML5. They were basically just rendering the same object, but making it smaller and using that to produce a gradient. I've enhanced the effect a little bit here. It wasn't quite that bad, but it was pretty bad. He was mad. We took on the project and our job was to come up with a way to render a better highlight on that shape. The thing we ran into is that in the native version of the game, they were using a shader to render that. If you're familiar with the shader, you know that a shader is a tiny little bit of code that you can essentially inject into the GPU and then it runs directly on the GPU. The job of a shader is to evaluate, you know, if it's in 3D to evaluate a triangle or a polygon or if it's in 2D to evaluate an individual pixel or depends on the type of shader, but then modify that pixel according to some small amount of code. GPUs are really, really good at executing this kind of code really, really, really fast. JavaScript engines are not really good at executing that kind of code really, really fast. You can imagine you've got a plasticine that's made up of hundreds of thousands of pixels and to try to execute that code on every single pixel, every single frame is just killing the performance. There's no way to do it. One of the challenges of these plasticines is they're not static. It's not like we could render them once. If you saw during the gameplay, part of the gameplay is to move the plasticines in order to move Petit, who's the main character in the game. It's really a challenge. It's not one of the things where we could render initially and get the shading perfect and then it would be fine. It has to move at the same frame rate as the game because that plasticine is moving, it's contorting its shape. So that's why in that initial implementation they came up with something that actually was really fast, but it did not produce the right result. This was an example where we looked at the code and there was really no way to port the right code over in a way that would essentially meet the performance goals of the game. So the pixel by pixel approach was out. We started looking at some options and the first exercise is we did this little report where we looked at five or six options. The first one we looked at was SVG filters. The thinking here was that SVG has some filters. One of them is a filter that creates an effect like this and we thought maybe we could produce something with that. The issue with this was first of all now we were mixing two media modes because most of the game is rendered into a canvas if you're familiar with that. Then we would have to have this vector object sitting on top of the canvas and then do the job frame by frame to keep those two things in sync. Also you probably know that SVG is not known for its great performance or speed. So we had some concerns about this and worst of all it just didn't look right. So we quickly crossed that one off our list. Then we started trying to break the problem down into chunks. We started thinking are there small bits of the plasticine that we could render individually and then cache some or create if we didn't need to render them that frame or else somehow break the problem down into smaller pieces. So we had this idea what if we divided the plasticine up into this pie and then we can look at each one of those shapes individually only modify the ones that we need to and it kind of breaks the problem down for us to get smaller. We looked at this but we kind of realized that we had problems on the edges where we could break this into multiple pies but kind of on the corners we still had sort of these hard edges. Again it just didn't look right. Plus this quickly began to feel sort of like a tough geometry problem like getting everything to sort of line up properly at the sort of the seams. Our initial hope with this was that we could by dividing it into pie slices eliminate the rendering below the plasticine because typically it's the sun or the moon that's shining down and creating that shadow. So we thought we could eliminate sort of half or more of the plasticine but it turned out that based on the angle and the manipulation of that plasticine it was pretty challenging. But we did gain some insight from that and the insight was that we really only needed to focus on the upper half of the plasticine. So from that we kind of came up with this idea well what if we kind of took the other approach which is we obscured the bottom half. We thought we can do that by just placing sort of another object that was like half the same shape on top of the bottom portion of the plasticine and then kind of fade out the edges. So this actually looked pretty good. So this is where we got with that and you can see it's we're getting a lot closer with that one. We ran into some issues with rendering this one and then also it just the motion still didn't quite feel right on this one. And that but it did again kind of lead us to what we ended up doing ultimately and you know that looks really different than it did in the game. That's funny. But basically what we ended up doing is stroking the edges of each of those shapes with a gradient. And the thing that made this work is in the same way that the shader kind of took advantage of what the GPU can do. We were sort of taking advantage of the native functionality of what you can do in a canvas which is apply a gradient. Right. The canvas probably instructs the GPU but the canvas knows how to render gradient so it knows how to do that math to sort of go from one color to another and by kind of stacking these on top of each other we could get something really close to the effect that ultimately he wanted. And it's funny because on the monitor it doesn't quite look right but. Oh does it really? That's funny. Yeah that's right. Anyway the game looked really really good. And Max I'll be honest Max was kind of only begrudgingly said that this was okay. This was after like you know serious effort trying to find something that would work. That guy is hardcore and we have a ton of respect for him because of it. But in the end we took this approach and the great thing about this is because we were doing something that canvas knew how to do quickly we actually I don't know with this and some other optimizations that we made we actually ended up with a version of the game that was faster and also had was closer to the effect that ultimately they wanted in the game. And so and this is so this was an example of where we kind of needed to step outside of the code that had already been written and look at the problem and try to find like a novel solution to that. And that is the perfect segue into the last thing that we want to talk about of course which is Tom Selleck. So we imagine a world and I think Tom Selleck is sort of the best way to think about this where we we could all have the the the benefit of Tom Selleck's mustache. In fact we found somebody else who celebrates this goal with us and we wanted to share this quick video with you as an introduction to what's coming. This is where you can stand up and dance on your seat. So I hope you guys feel it because we feel it. In fact we feel it so much that we created a JavaScript library to help us see the end goal here which is called StashKit. So StashKit web based facial hair delivered as a service. This is the world premiere of the service available soon. Why? Obviously because we want to be first to market. Nobody else that we know of is doing this. Industry knowledge both Joel and I have at some point grown facial hair and most importantly for this talk it turned out that this is an awesome performance test bed. So some of the things that we want to share with you about performance can be really well illustrated through facial hair. Okay let's see it. Here we go. So we'll just orient you quickly to the product. We've got a photo of a handsome gentleman there and he's got an amazing mustache. Of course that is not his mustache. That is rendered with StashKit technology. There's a number of options. So if you wanted to apply the mount man growth formula you could grow the mustache. Or if you're just happy with the instant week we can go back to that. We recognize that there are a number of popular color choices. So if you want a red mustache or if you've got a fair complexion you've got that black. Brown's pretty nice. You could change the curl of the Stash. So maybe wavy, little extra curl there. And of course you've got lots of different choices. So maybe we'll go with this guy. That's kind of fun. We'll shrink it down. That looks pretty good right? Looks amazing. That lucky man. So briefly I'll give you a quick overview of as to how this is working. So we have sort of the templates you can see along the bottom which define the Stash as we call it in code. It's just more fun to call everything a Stash. And so in the actual images we apply a little bit of opacity towards the bottom of the mustache. We load that image into a canvas and parse out the individual pixels. Based on the opacity of the particular pixel we give it a probability that determines the density of the mustache. You obviously wouldn't want a mustache where it's completely dense. You want to be able to control the density. And so that opacity fade gives us a little bit of confidence so that the mustache sort of fades and gives you a natural look. Now of course our favorite feature is the aroma. And maybe I should... Yeah you want to see this one. Let's increase this guy so you can see it in the right place. So it's hard to visually represent an aroma, right? So we put our head against the wall and this is what we came up with. So this was the big challenge. How do we render this mustache at 60 frames a second? You can see right here this is not 60 frames a second. I'll do it again so you can see because you... I'll isolate it. Let's put it over here. You can see... Oh, let's put it up here. Here we'll put it right there. There we go. So you can see that there's a little bit of stutter in between those frames. It's not completely smooth. And basically what's happening is when we parse out that pixel data, depending on the mustache, there are quite a few hairs in there. So I think one of the bigger ones that we looked in it was I think around 15,000 hairs. And so what we're telling the canvas to do for each of those pixels is to take the beginning point, take the length of the mustache, take the inflection point where the curl is, calculate the curl strength using a curve, and then stroke that line. Repeat 15,000 times. And you have to do that every frame, every 16 milliseconds. It's pretty challenging to do even with the hardware acceleration that the canvas provides because of there are so many calls consecutively. So let's switch back. Let's see if I can do it this time. There we go. What just happened? So I kind of gave you an overview of what happened and how the basics of the framework work. When we did the initial implementation of this, we got around 10 stashes per second. We'd like that to become a new sort of benchmark in JavaScript performance. So if you could take that back and start profiling your code in stashes per second, that would be great. Clearly not good enough for a mustache rendering service. So the first thing we tried to do was we thought maybe we can render the mustaches by color group. So if you saw the mustaches, you noticed that they're not all one color because that's not going to look natural. It would just look like one big mass. So there's a little bit of variation. So we had, I think, between five and six colors for each complexion. And we would vary those based on just sort of randomly distribute them through the stash. So we thought maybe we can speed up. Instead of doing 15,000 strokes, maybe we can do all the colors and stroke them all at once. Canvas has a great API where you can draw a line, move, draw another line, and then stroke it all at once. And in a lot of the games that we've done and projects in the past, we found really good performance benefits by using that technique where rather than doing individual operations on the canvas, you group as many together as possible and then do them all at once. So sort of the four different colors right here, you take the first implementation that we did where they're interspersed. You'd have different colors. We decided to sort the array so that we would have all the colors sort of in one group. We could stroke them all. We thought this is going to be fantastic. And the performance was great. We got great performance out of this. But the problem is you can see the stash on the left has sort of got that natural look to it. We didn't realize that by grouping them by color, we're basically saying all the colors are going to be sort of on top of each other in layers because it doesn't look right at all. It actually doesn't look that bad here, but if you had darker colors on top, it just doesn't look right. So we thought, well, that was a good try. Clearly not going to cut it. So then we thought, let's cache the stash. So the approach here, and it's one that we've used on a number of games, is when you have a sequence of frames that need to get rendered but will be reused, you can actually draw that to an off-screen canvas and extract the image from the canvas and sort of create your own little mini sprite where you have the frames for each image that are rendered, and then you replay that. So in this case, we also had a growing mustache animation. So you can see up at the upper left, it's sort of barely starting to come in. And then each frame, it gets a little longer, a little longer, and you have the full stash. And with this technique, we were able to get the 60 frames per second, which Robbie calls a stash stash. It is stash stash. There's no other way to say it. That's also a benchmark when you hit 60 SPS at stash stash. So let's switch back to the app. So this is, I'll show you the, we've got our mustache here. So this is the render by color. We'll show you that. So you can see the, a mustache does not look great. So we've kind of got all the colors. Let's try a different color. Some of them, it really does not look great because they're grouped together sort of all in one layer. Switch back to this guy. So let's turn on the pre-render. The challenge with the pre-render, and you sort of have to find the right time to use it, is that even though it's going to give you that 60 frames a second, it does take a little bit of time to pre-render all those frames, and then you catch those results. So we'll click the aroma. You'll see that it basically does nothing for a few seconds while it's caching the result. And then you can see the smooth stash-tastic animation. And silky. Just like you want it. So you can see that that is much, much smoother than the original implementation. And it's because we're basically just replaying that series of frames that we've already rendered. We thought, go ahead. We thought we'd have a little bit of fun with the webcam API. So we added that feature in. Let's see if we can do it. Any volunteers? Should we make Robbie do it? I'll do it. There you go. All right. So that's a lot of stash for me. I'm not complaining. I think the lighting might be... We do have an enhanced too to help with the lighting. Let me turn that on. Oh, there we go. Oh, yeah. That's way better. There we go. Robbie looks amazing. I think we can all agree. So this is coming for all of you. So keep an eye out for StashKit. You can stay tuned on thinkpixelab.com. We really will deploy this. We've actually been working on this for on and off for like two years. We've been having fun. We felt like maybe we could do eyebrows. There's one sort of Yosemite Sam thing that looks like it could be a toupee. Although David Hasselhoff definitely does not need that. But it doesn't hurt. Yeah. All right. All right. So yeah, with that we have six days BS and we have perfect rendering as you can imagine. So, you know, I think the things we talked about today in terms of performance are probably things that you sort of understand. But if you took one thing away today, it would be to write great code that you can understand that other people can understand and then optimize 3% of it. And then when you're optimizing that 3%, this is our approach. We look at the hotspots using the best tools. IE has, we showed you the IE performance tools because that was a project for Internet Explorer. Chrome and Firefox also have great DevTools. All of them I think have a great profiler now. A lot of times you will find that the performance just won't get better by refining your code. So that's when you get to think creatively. You think about the approach that you're taking or there are other ways that you can accomplish the same thing. And then last is an invitation to experiment and refine because this is where you learn and it's also where you typically tend to solve the hardest problems. And I think that is it. So yeah, we hope you guys have enjoyed the talk. If you have feedback, we'd love to hear that. If you have questions, we're open for those as well. Oh, there's a question. All right. Let's take it. Go ahead. In fact, Contrudeur's shading thing, I understand it wasn't an option because it brought us support, but did you play with WebGL at all? It wasn't an option, so we didn't play with it. Today I think we would. This was about two years ago and there was just not a lot of great WebGL support at the time. Now there is. Also, there's a project with Microsoft and IE, Internet Explorer in particular did not support WebGL at the time, so it would have been awkward for them had we shipped a game with WebGL. But that really is the right thing to do for sure. And that's how we would do it today. Definitely. That's a great question. Yeah. Performance is usually quite critical in games. How did you find moving from doing normal software across into playing with it? So I forgot we're supposed to repeat the question. So the question was performance is obviously a critical part of writing games. How did we find the process of going from sort of typical maybe line of business software to creating games and maybe what are the different kind of performance constraints that you deal with there? Do you want to take that one? Yeah, at least for me, I came from a server background where I was working on server code and there actually is a big performance focus there because you have to handle so many concurrent users. So for me it was sort of a natural, I've always loved performance. I think the challenge is sort of what we talked about is not getting overly enthusiastic about performance and really letting the profiler guide that process of optimization. I think it's so easy as developers we get excited about maybe an article that we read that says like this little JavaScript tip led to huge performance gains and really knowing when to use that. In most cases it's probably not necessary and it will over complicate the code. And so I think as a developer over my career it's more of learning when the right time to optimize is and really using the tools to let that guide the choice and where to optimize. That's great. Yeah. When you are rendering out what you're seeing onto the campus, is your render loop basically just loads and loads of manual calls to go every little bit just in terms of primitive like rectangle and stuff? Are there any libraries for working with campus that you recommend to use? Do you want me to take that? Sure, you can start. Yeah. So the question was when you're working with Canvas, so for those of you who aren't familiar with it, Canvas is basically just a surface onto which you can draw whatever you want. You have like open season with pixels onto Canvas in HTML5. And the question was is there a library that we like to use or are we using the native Canvas API? We've done kind of through several libraries. So there's one by Grant Skinner that, what is that one called? EZL.js, which we use for a couple of early projects which we liked. Recently we've been using a kind of a gaming framework called Phaser.io, if you guys are familiar with that. And that one actually has WebGL support for both 2D and 3D if it's available. It's really cool. We're actually really happy with that one. With Cut the Rope where we were doing, and also with ContraJour we were doing a port. And so because we really needed to be stay in sync with the code that that developer had already written, it wasn't as easy or didn't make as much sense to use another framework. So both of those are, we're just rendering directly with the Canvas API. Yeah, and in terms of primitives versus, I think it's sort of a mix. In games there are a lot of sprite based animations, so a lot of it is drawing images. And then there's some primitives, whether it's polygons or lines or things like that. I think you had a question. When talking the physics engine from Objective C to JavaScript, did you have to change anything in the algorithm? Is there any mass computation that's inherently impossible or very hard to do in JavaScript? It's a great question. Yeah. So the question was, were we able to take the Objective C physics engine and use that directly in JavaScript or did we have to modify any of the algorithms? We tried not to modify any of the algorithms. We tried to focus on sort of low level optimization just because we were afraid if we changed the algorithm that it would change the results. And we wanted the game to feel identical to iOS. But we had to do a lot of optimization. So those early code examples where we talked about removing vector allocations, removing arrays, being really careful about calling functions inside tight loops where the physics engine was operating, those are sort of the areas that we had to focus on. So we didn't change the algorithms, but we had to do a lot of performance work to get it to run at high speed. Go ahead. Are there any good tips on, you know, you spoke about benchmarking, but you really can benchmark this. Some canvas operations are a lot slower than others. And all the Chrome or whatever profile will tell you is time spent on canvas, but maybe you could make it faster by having it a pack and not semi-transparent or things like that. Are there any good ways to profile in that area? Yeah, that's a great question. So the question is, are there any good ways to sort of narrow down the areas when you're drawing to be more specific and find the hotspots? And I think for us it was a little bit of trial and error and a little bit of reading. There's lots of good resources on the web that talk about tips for performance. We went through a lot of those and some of those panned out for us. Some of them didn't, depending on the platform. One in particular I can remember was that there was a recommendation to use multiple canvases and sort of stack them on top of each other. We had a hard time getting the performance out of that. It may have changed now. Another sort of interesting thing is that because the browser engines are improving so quickly that it really is a moving target, like one technique that works today may not work tomorrow. And so it is better if you can find sort of the approach-based methods or algorithmic approaches where they're always able to do better. And I think for us it was just more of a process of trial and error combined with sort of some guided points to start. Yeah, that's exactly. Any other questions? Well, we've had a lot of fun, so thank you guys. Appreciate it. Thank you.
 Hey, guys. Thanks for sticking around for a very, this will be the, well, at least the  very last talk of the day. It's not the best. So, hopefully the best. So, my name is Robbie  Ingebrezen. Do you guys recognize Ingebrezen as a Norwegian name? I've never asked this  in a, when I've spoken before, but is there anybody here who can tell me how to pronounce  my name? How do you do it? Okay, I'm not even going to try. So, my great grandfather was  from, and this I will get wrong, but it's SKIEN in Telomark. Cheyenne, I think? Cheyenne?  Are you, is anybody from Cheyenne? Do you know Ingebrezen's there? Dang it. We, I just  went down there actually on this trip and looked for traces of my family. There's a cemetery  there, like the main cemetery, and there's two actually, but the big one, like in the  downtown area, and we found a whole bunch of Ingebrezens, which may or may not be relatives,  but I hope they are. Anyway, so, it's fun for me to be in Norway where I can pronounce  my, say my name and people don't have to look at me in a weird way. So, I'm Robbie Ingebrezen,  and then this is Joel Fillmore, my good friend and colleague, and together we are a small  company called Pixelab. We're based out of Seattle, although when you're just two people,  you can kind of be based out of wherever your living room is that day. So, this week we're  based out of Oslo. So, that's another connection that we have to Norway. So, we are focused  primarily on HTML5 and JavaScript and sort of the, a web stack, although both of us have  history with Microsoft technologies. I was part of the WPF and SilverEye teams. Any  Microsoft guys here? Couple? Yay. Does anybody use Kazaml, by chance? It's a little, okay.  So, Kazaml is my, I wrote that. And then Joel was on the SharePoint team at Microsoft.  So, two really exciting products that brought us together to work on HTML5 applications.  And actually we've had pretty good success with HTML5 stuff. We, in fact, we just won  a Webby last week for a project that we did with Red Bull. And we're going to talk about  some of the other projects that we've done. But our kind of focus in our, I guess our  area of interest is sort of forward looking kinds of HTML5 experiences that sort of push  the envelope and what you can do inside of a browser. And that's why performance is kind  of a natural fit. And so, for that reason, we thought it would be fun to talk about performance  and share some of the things that we've learned about JavaScript performance. And that'll  be the focus of the talk today. Tomorrow we have another talk, though, where we're talking  more about some of the creative aspects of using HTML5 to create forward looking experiences.  And that is called the future of extreme web browsing. And I think it's like tomorrow at,  anyway, just look it up. Come, though. It's tomorrow at 10 or 11 somewhere around there.  So we'd love to see you guys there, too. So this is us. If you want to track us down,  thinkpixelab.com. And then that is my Twitter handle, which you are the only country in  the world that will remember that. So I don't need to spend too much time there. And we  are Canuthians. And I will let Joel tell you what we mean when we say that.  All right. So who is Canuth? This guy. So some of you may recognize him. He's a famous computer  scientist. He's a professor emeritus at Stanford University. Most people have heard of Donald  Canuth from the series of books that he's written, the most famous of which is the Art  of Computer Programming. So this is the series of books that every serious programmer has  on his bookshelf, but it's never read. It's pretty daunting. I've paged through it a few  times, but have not read the whole thing. Donald Canuth is famous with this particular  set of books, and I think his others as well. If anyone found an error in the book, he promised  to send them a reward. And the reward was a hexadollar, which is 256 cents. And so he  would write out the check for a hexadollar. And to date, people take a lot of pride in  sort of cashing these checks. And he's written over $20,000 worth of checks, but supposedly  very few of those have been cashed because they're kind of like a badge of honor.  So he's famous for a number of quotes. We gathered a few of our favorites. This one  is Beware of Bugs in the Above Code. I've only proved it correct, not tried it. That's  pretty impressive when you have the mental compiler or the mental theorem prover to prove  the code correct. The next one is Science is What We Understand Well Enough to Explain  to a Computer. Art is Everything Else We Do. This is Robbie's Favorite. I Can't Go to  a Restaurant and Order Food because I keep looking at the fonts on the menu. And this  is great. You know that Donald Canuth was also the inventor of latex. This is the performance  one that we're going to be talking about today. The quote is, we should forget about small  efficiencies, say about 97% of the time. Premature optimization is the root of all evil. So when  I was beginning my career at Microsoft, I was a new developer. One of the first tasks  I was given was to implement a change to a series of data structures that we had to manipulate  in a certain way. And so I wrote some code and I was really sort of new at the time.  I was worried that it had to be fast. We were working on a server product where lots of  concurrent requests could come in. And I was really careful to optimize this so that it  would filter down the set of results in an efficient way. And I checked the code in and  the next day I noticed that one of my colleagues had basically undone everything that I had  written. And so I sort of mentioned, hey, did you, were there problems in what I had  written? He said I had to make a few modifications and it was really difficult to understand.  And so that particular piece of code we don't have to worry about from a performance perspective  because there are so few items in that list. So I rewrote it so that it's a little simpler  to understand. And he said, you know that quote, premature optimization is the root  of all evil. I didn't know the quote. But I went and looked it up. And it was sort of  a lesson to me that we shouldn't focus on optimization until we're absolutely sure it's  necessary. So our kind of approach and what we would like to talk about today is the balance  between optimizing code and maintaining code. So on one hand you have maintenance where  you want code to be easy to understand for people that have to come in and maintain  it. You want it to be efficient to write. And you also want it to be optimized for execution.  And those are competing interests. So today we're going to talk a little bit about how  to balance those interests. The rest of the quote we started off, the beginning is we  should forget about small inefficiencies, about 97% of the time. Premature optimization  is the root of all evil. The remaining part of the quote is also important. Yet we should  not pass up our opportunities in that critical 3%. So that really is the key. It's to use  the tools and methods we have to focus on the things that are most important. And for  the rest of it, use the principles that guide us in writing easy to maintain, easy to write  code. Okay. So we're going to talk about three different things. The first is tools. So we're  going to talk about how tools can guide that process to find the 3% that is worth optimizing.  Next we'll talk about an approach-based method where looking for different ways to optimize  code rather than focusing in on the specific optimizations of a one chunk of code, stepping  back and looking at the algorithmic or the method approach to optimizing code. And then  third we're going to talk about the process of experimentation and refining. It's an iterative  process to get your code in good state. Okay. Real-world examples. The first one we're going  to talk about is a game called Cut the Rope. We did, so Cut the Rope, I'm sure many of  you are familiar with, it's a very popular iOS, Android game. We did the HTML5 port for  Cut the Rope a few years ago. And it was one of the first big, really challenging HTML5  projects that we did. Before we get too far into the details, for those of you who aren't  familiar with Cut the Rope, we'll give you a quick video intro.  Cut the Rope is a casual game which was initially released for mobile devices. It's essentially  a game where you have to deliver candy to the little green monster called Omnom by cutting  the ropes. Currently it has been distributed on two platforms and it totally has been downloaded  more than 60 million times. It's quite successful. We had this basic concept of delivering something  from point A to point B in the very beginning of the game. That's when we got the idea of  candy being fed to the little green monster. And that idea was absurd and adorable at the  same time. So we thought that it's a good way to go.  We have more than 6 million play sessions every day. When we want to expand and HTML5  is just another great platform to expand to, it will bring us much more users, much more  players. If any of you guys played Cut the Rope? Okay,  so a lot of people. Anybody played on the web? My chance? Awesome.  Well, yeah, so that was the project that we did. And it was a big project.  Yes, I think when we first started out they gave us the Objective-C code base. It was  about 150 files, 15,000 lines of code. They had written their own custom physics engine.  So that was a little daunting to see all that code giving a relatively short project timeline  to get that ported. The result was about 1.2 megs of JavaScript  unminified. It got much better minified. But you kind of get the sense for what a daunting  task this was. During the initial development of the project,  we did some early prototypes. So we took the ropes, which are sort of the premier element  in Cut the Rope. You want those to look amazing. And we wanted to see if we could do one rope.  And so the challenges with the ropes are that they used polygons to render those, and they  had some pretty complex calculations to do busier curves to make sure that the ropes  felt lifelike so that they would swing, the physics felt right, and they looked beautiful.  So on the left, you're going to see one of our early prototypes. So this was sort of  our first attempt. We ported the code, used the same approach that they did, and this  is the result. So you can see that the rope is a little slow.  What you're seeing is definitely not 60 frames a second. You can see as the rope sort of  curls down, there's some artifacts sort of at the kink where it was difficult to render  those polygons in a small amount of area. And on the right is the final version of the  game. And so this is sort of one section of the game. It's still a single rope. But there's  much more going on in this. You can see that the spider is traversing the path of the rope,  which is changing according to the physics engine. The sort of blower up at the top is  sending an impulse of wind, which affects the candy, the rope, the path the spider's  on. So it's significantly more complicated. There's a lot of elements.  So our concern was how do we go from the prototype on the left where we're barely like rendering  a rope. It doesn't look great to the full game where we have multiple ropes, multiple  game objects, lots of animation, lots of forces going on. And so that was a real concern  and challenge. And we weren't sure that we were going to be able to do it.  So let's talk about how we did it. The first thing that we did is fire up our profiler  and memory debugger. And this was sort of the profile that we got. So you can see that  there's actually not that much memory being used. This is over a relatively short period  of time. So I think this total period is two seconds. And what you're seeing with that  sawtooth pattern is the number of memory allocations go up, and then the garbage collector comes  in and goes down. And so we've got at least two or three garbage collections per second  there. So sort of the quick intro to how garbage collection works. We've got, well, we're from  Seattle, and Seattle drinks a lot of coffee. So the easy way to understand garbage collection  is if you had a coffee mug and you had a disposable coffee cup, and imagine you needed to drink,  I don't know, 20 cups of coffee every 16 milliseconds. Yeah, 16 milliseconds to draw a frame. And  so every sort of object that you allocate has to be collected at some point, right?  So maybe it's not that bad if you drink 20. If you drink maybe 100, that's a little more  work for the garbage collector. It's going to require more processing time, JavaScript  is single threaded. So typically they'll pause and collect the garbage and then let it resume.  And it just gets worse and worse. So you can imagine how bad it got. We did the profiler.  And there's a couple interesting things in this profiler. Typically when I'm profiling  an app, I will sort by the exclusive time. So I'll look at functions where a lot of work  is being done. And those are up at the top pretty obvious. So there's bezier calculations,  so that's calculating the curves on the rope. There's the constraint engine, which is the  physics and the objects and how those interact. Some more bezier calculations. And you can  see sort of the difference that the typical approach with JavaScript is methods are underscores  or lower case, lower camel case, and objects are uppercase. So there's sort of two objects  in there. You can see the vector and the array. So they're not necessarily at the top of the  list. They're not taking the most amount of time. They are definitely in the top 10, but  they're not the highest. But the count right there is pretty important. So you can see  over that two or three second period that we allocated 2.2 million vector objects. That's  a lot of vectors. And so you can imagine the amount of pressure that puts on the garbage  collector, which has to go in there and collect all those objects that are no longer being  used. Similarly with the arrays, that's not as bad as the vector, but 262,000 arrays is  pretty significant. So after the profiling, we were able to get this down. So sort of  the end game, so you can see the difference from that initial prototype where we had one  rope that was barely rendering to the full game. We went from 2.2 million vector allocations  down to about 30,000, which is still pretty high. But given the amount of calculations  and they're going on the game, it's much, much better. The array allocations we got down  significantly as well, 262,000 down to just 400. So how did we do that? We're going to  go through a little bit of code. This is the intensive part of the talk. And we'll show  you a simple example of how we were able to get rid of those allocations. So on the left,  we have the native implementation of a vector. So it's a struct. Those are relatively lightweight.  You have some utility methods that are static to add and multiply vectors. Those are pretty  straightforward. You add the X and Y coordinate or you multiply by a scalar. And then we have  the direct port. So this is sort of our naive direct port implementation of a vector where  we would do the same thing. We had a static or a utility method to add two vectors and  the same thing for multiply. And so the key here is that new. You're allocating a new  vector every time you need to add. So the vectors in this example are used by a game  object. So a game object could be omnum. It could be the candy. It could be a piece of  the rope. It could be any element in the game. And so it's pretty common for a game object  to have a position within the screen. And forces will act upon that game object to move  it around. So in this case, we apply an impulse to an object. And basically what that does  is it takes the delta or the time slice that has occurred since the last frame was rendered  and figures out the distance that the object should move based on the impulse that was  applied to that object. So you can see that we're basically doing two things. We're multiplying  the impulse force by the time slice to get the offset. And then we're adding that to  the position. So that would be basically moving a character according to some speed.  So here's our first attempt at reducing the number of allocations. We realized quickly  that we're changing the position of a game object or a game element. We don't need to  allocate a new vector for that, right? Because he just needs to know his position. We don't  need to create a new element and then sign that new element. That's just asking the garbage  collector to do extra work to collect that extra element. So the first thing we did was  instead of a static method to add a vector which has the new inside there, we created  an instance method which adds the X and Y coordinates of another vector to the existing  vector. And so this was perfect for the position because the object only needs to keep track  of its position. And we move the object according to that impulse. There's no vector added.  So we're able to get rid of quite a few vector allocations using techniques like that. So  here's the multiply. We can't do the same thing for the multiply though, right? If we  were to multiply the impulse vector in place, then it changes the speed of the object. You  can't do that. So the strategy there is to use local variables and just do the calculation  in place. And this is, again, going back to the quote that we talked about at the beginning,  not something that we would do in every place where a vector is used. And similarly, if  you have a convenience method that adds a vector, it's okay to use that in the vast  majority of cases. You want to use the profiler to guide your optimization and find the chunks  of code that are path critical and that are really constraining the frame rate. And so  once you find those, you can use the techniques. That's the 3% to really make them run quickly.  For extremely critical paths, we found that inlining the function in the same way that  the native compiler would do with the inline hint, it would actually inline the code. You  can do the same thing with JavaScript by taking simple functions and avoiding the apply impulse  function call entirely and just doing the calculations in place. And the results of  that, you can kind of see, this slide is a little bit outdated, but you can see the direct  port is in blue. The reusing the same vector after we made that change, we got a little  bit better performance. Switching to local variables, performance went up significantly.  And inline code where we completely avoided any function call was obviously the best.  This was done a couple years ago. We tried it just out of curiosity last night and it's  really incredible to see the difference that today's browsers have. It was just astronomical  the difference.  We wouldn't have found the problem in today's browser because the browser is so fast that  it hides the problem. It was kind of crazy to see that because we only cut the rope  two and a half years ago or something. It wasn't that long ago. I kind of had this feeling  that two years ago browsers got really, really fast and then we've just sort of been living  the dream. But it turns out that there's been a huge amount of improvement in what a browser,  especially the JavaScript engines, across all browsers are doing right now. It would  be interesting if you had a chance to go in and compare old browsers versus new browsers.  Yeah. JavaScript engines today are really incredible.  So we talked about the memory profile at the beginning and you see the sawtooth pattern.  After the changes, you can see the difference that reducing those allocations makes. So  it's much smoother. And what that translates into is a higher frame rate. So by the time  we were able to ship, we were getting 60 frames a second on most devices and we were really,  really happy with those results.  So Joel did a great job of kind of explaining, I guess, one of the first approaches to performance  should be, which is you see a performance problem and you can mostly, as you're writing  your code, like Joel said, you can mostly ignore it, right? Don't optimize until you  know there's a problem. Then as the problems begin to emerge, and actually, there's a little  caveat with that, obviously. We're assuming that you're writing reasonable code as you  go, right? Using best practices. But I guess as we're talking about sort of that tension  between writing optimized code versus writing readable code, we're saying that it's okay  to write readable, reusable, understandable code rather than focusing on the highly optimized  code until you see that there's a problem.  Now when you see that there's a problem, Joel did a great job describing one of the first  approaches you can take, which is to actually look at your code and use the tools to find  the hotspots and then once you find the hotspots, then you can address those directly. But there  are times when the code just sort of can't be optimized. Like you're trying to accomplish  something where the approach actually doesn't allow you to get fast enough. And we ran  into this with a project for a game called Contre Jure. Has anybody played this game?  It's a little bit less popular than Cut the Rope. It's a super cool game. So those of  you who have played it can attest to that. It's really just a super beautiful game.  And to understand sort of the pressure we felt in terms of performance, you have to  understand who created it. So we wanted to just take a couple of minutes and show you  a video. This guy named Max, and I actually don't know his last name, he's a Ukrainian  guy and I guarantee this guy is like on the front line right now. But he is dedicated  to everything he does. And we had sort of, I guess, the privilege and also the challenge  of trying to help import his game to the web. So this is just three minutes about it.  Belief in a third world country. You know, the main force that pushes Ukraine forward  right now in software development. If it's about gaming scene, it's the stage of early  development. We cannot officially buy an Xbox or any Xbox games because we don't have Xbox  live in Ukraine. If I create a game for Xbox live, I will have no possibility to buy it.  Crazy.  My name is Max and I'm head of Mockers games studio. When I started to create games, post  match games, it started different market compared to the mobile or Xbox PC, etc. You cannot  make a lot of money on a flash market and you need some money to buy pizza and to buy  a few for my motorcycles. When I started, I did one game, one month and release. It  is how it is done in the flash world.  Now everything changes. We have downloadable markets, we have upstores, we have mobile  market and it's much easier to create a product for mobile market. When I started to create  ContraJewel, I realized that it's something bigger. So I decided not to release it on  a flash platform, but to try to make it mobile. Right now, after release of ContraJewel, we  are in financial point of view. We are just free.  It's very comfortable when you can wake up, just have a few steps and you are at work.  For me, everything is the same. It's my work, here are my friends and it's my hobby, it's  what I like to do, so it's everything together.  A few years ago, I think it was impossible in Ukraine. We had no developed game and  scene, now everything changes.  Do you think you will always be making games?  No, definitely no. If you would ask me the same question, 10 years ago you would ask  Max, do you think you would be breakdancing all of your life?  Yeah, 10 years ago maybe the answer would be yes, but right now I am a game developer.  This gym is 4 times for a week and I have another gym just for breakdancing. The second  one is 3 times for a week, so one training for a day.  It's actually a gymnastics gym. I do some tricking, some parkour. The floor is not the  best here for breakdancing. Actually, it's for gymnastics. I'm not in the best fit,  but I still can do something.  I do this because I have fun. I enjoy it. Even if I get older and I cannot, for example,  win some battles no more, but I still enjoy it.  When you work, you work with your brain. You challenge mainly your mind. Here you can  challenge your body and when I go out from the training, my mind is totally cleared.  It's like a reboot. In the former Soviet Union, we had no breakdancing, no modern street  dances. I got very interested and I wanted to try.  We stopped it there because it just gets more awesome. That guy is, honestly, we have a  huge amount of respect for him and also we know that he could beat us up. When we inherited  this project from another agency, this is what he was facing and he was just not happy  at all about the way that they were rendering. They call these plasticines. You can see there's  like the little, like over, well, I don't know what he's talking about. You can see  that there's the native one on the left-hand side and then on, or I guess that would be,  on your right-hand side. Then on the left-hand side is what they were rendering inside of  HTML5. They were basically just rendering the same object, but making it smaller and  using that to produce a gradient. I've enhanced the effect a little bit here. It wasn't quite  that bad, but it was pretty bad. He was mad. We took on the project and our job was to  come up with a way to render a better highlight on that shape.  The thing we ran into is that in the native version of the game, they were using a shader  to render that. If you're familiar with the shader, you know that a shader is a tiny little  bit of code that you can essentially inject into the GPU and then it runs directly on  the GPU. The job of a shader is to evaluate, you know, if it's in 3D to evaluate a triangle  or a polygon or if it's in 2D to evaluate an individual pixel or depends on the type  of shader, but then modify that pixel according to some small amount of code. GPUs are really,  really good at executing this kind of code really, really, really fast. JavaScript engines  are not really good at executing that kind of code really, really fast. You can imagine  you've got a plasticine that's made up of hundreds of thousands of pixels and to try  to execute that code on every single pixel, every single frame is just killing the performance.  There's no way to do it.  One of the challenges of these plasticines is they're not static. It's not like we could  render them once. If you saw during the gameplay, part of the gameplay is to move the plasticines  in order to move Petit, who's the main character in the game. It's really a challenge. It's  not one of the things where we could render initially and get the shading perfect and  then it would be fine. It has to move at the same frame rate as the game because that plasticine  is moving, it's contorting its shape.  So that's why in that initial implementation they came up with something that actually  was really fast, but it did not produce the right result. This was an example where we  looked at the code and there was really no way to port the right code over in a way that  would essentially meet the performance goals of the game. So the pixel by pixel approach  was out.  We started looking at some options and the first exercise is we did this little report  where we looked at five or six options. The first one we looked at was SVG filters. The  thinking here was that SVG has some filters. One of them is a filter that creates an effect  like this and we thought maybe we could produce something with that. The issue with this was  first of all now we were mixing two media modes because most of the game is rendered  into a canvas if you're familiar with that. Then we would have to have this vector object  sitting on top of the canvas and then do the job frame by frame to keep those two things  in sync. Also you probably know that SVG is not known for its great performance or speed.  So we had some concerns about this and worst of all it just didn't look right. So we quickly  crossed that one off our list. Then we started trying to break the problem down into chunks.  We started thinking are there small bits of the plasticine that we could render individually  and then cache some or create if we didn't need to render them that frame or else somehow  break the problem down into smaller pieces. So we had this idea what if we divided the  plasticine up into this pie and then we can look at each one of those shapes individually  only modify the ones that we need to and it kind of breaks the problem down for us to  get smaller. We looked at this but we kind of realized that we had problems on the edges  where we could break this into multiple pies but kind of on the corners we still had sort  of these hard edges. Again it just didn't look right. Plus this quickly began to feel  sort of like a tough geometry problem like getting everything to sort of line up properly  at the sort of the seams. Our initial hope with this was that we could by dividing it  into pie slices eliminate the rendering below the plasticine because typically it's the  sun or the moon that's shining down and creating that shadow. So we thought we could eliminate  sort of half or more of the plasticine but it turned out that based on the angle and  the manipulation of that plasticine it was pretty challenging.  But we did gain some insight from that and the insight was that we really only needed  to focus on the upper half of the plasticine. So from that we kind of came up with this  idea well what if we kind of took the other approach which is we obscured the bottom half.  We thought we can do that by just placing sort of another object that was like half  the same shape on top of the bottom portion of the plasticine and then kind of fade out  the edges. So this actually looked pretty good. So this is where we got with that and  you can see it's we're getting a lot closer with that one. We ran into some issues with  rendering this one and then also it just the motion still didn't quite feel right on this  one. And that but it did again kind of lead us to what we ended up doing ultimately and  you know that looks really different than it did in the game. That's funny. But basically  what we ended up doing is stroking the edges of each of those shapes with a gradient. And  the thing that made this work is in the same way that the shader kind of took advantage  of what the GPU can do. We were sort of taking advantage of the native functionality of what  you can do in a canvas which is apply a gradient. Right. The canvas probably instructs the GPU  but the canvas knows how to render gradient so it knows how to do that math to sort of  go from one color to another and by kind of stacking these on top of each other we could  get something really close to the effect that ultimately he wanted. And it's funny because  on the monitor it doesn't quite look right but. Oh does it really? That's funny. Yeah  that's right. Anyway the game looked really really good. And Max I'll be honest Max was  kind of only begrudgingly said that this was okay. This was after like you know serious  effort trying to find something that would work. That guy is hardcore and we have a ton  of respect for him because of it. But in the end we took this approach and the great thing  about this is because we were doing something that canvas knew how to do quickly we actually  I don't know with this and some other optimizations that we made we actually ended up with a version  of the game that was faster and also had was closer to the effect that ultimately they  wanted in the game. And so and this is so this was an example of where we kind of needed  to step outside of the code that had already been written and look at the problem and try  to find like a novel solution to that. And that is the perfect segue into the last thing  that we want to talk about of course which is Tom Selleck. So we imagine a world and  I think Tom Selleck is sort of the best way to think about this where we we could all  have the the the benefit of Tom Selleck's mustache. In fact we found somebody else who  celebrates this goal with us and we wanted to share this quick video with you as an introduction  to what's coming.  This is where you can stand up and dance on your seat.  So  I hope you guys feel it because we feel it. In fact we feel it so much that we created  a JavaScript library to help us see the end goal here which is called StashKit. So StashKit  web based facial hair delivered as a service. This is the world premiere of the service  available soon. Why? Obviously because we want to be first to market. Nobody else that  we know of is doing this. Industry knowledge both Joel and I have at some point grown facial  hair and most importantly for this talk it turned out that this is an awesome performance  test bed. So some of the things that we want to share with you about performance can be  really well illustrated through facial hair.  Okay let's see it.  Here we go. So we'll just orient you quickly to the product. We've got a photo of a handsome  gentleman there and he's got an amazing mustache. Of course that is not his mustache. That is  rendered with StashKit technology. There's a number of options. So if you wanted to apply  the mount man growth formula you could grow the mustache. Or if you're just happy with  the instant week we can go back to that. We recognize that there are a number of popular  color choices. So if you want a red mustache or if you've got a fair complexion you've  got that black. Brown's pretty nice. You could change the curl of the Stash. So maybe wavy,  little extra curl there. And of course you've got lots of different choices. So maybe we'll  go with this guy. That's kind of fun. We'll shrink it down. That looks pretty good right?  Looks amazing. That lucky man. So briefly I'll give you a quick overview of as to how this  is working. So we have sort of the templates you can see along the bottom which define  the Stash as we call it in code. It's just more fun to call everything a Stash. And so  in the actual images we apply a little bit of opacity towards the bottom of the mustache.  We load that image into a canvas and parse out the individual pixels. Based on the opacity  of the particular pixel we give it a probability that determines the density of the mustache.  You obviously wouldn't want a mustache where it's completely dense. You want to be able  to control the density. And so that opacity fade gives us a little bit of confidence so  that the mustache sort of fades and gives you a natural look. Now of course our favorite  feature is the aroma. And maybe I should... Yeah you want to see this one. Let's increase  this guy so you can see it in the right place. So it's hard to visually represent an aroma,  right? So we put our head against the wall and this is what we came up with. So this  was the big challenge. How do we render this mustache at 60 frames a second? You can see  right here this is not 60 frames a second. I'll do it again so you can see because you...  I'll isolate it. Let's put it over here. You can see... Oh, let's put it up here. Here  we'll put it right there. There we go. So you can see that there's a little bit of stutter  in between those frames. It's not completely smooth. And basically what's happening is  when we parse out that pixel data, depending on the mustache, there are quite a few hairs  in there. So I think one of the bigger ones that we looked in it was I think around 15,000  hairs. And so what we're telling the canvas to do for each of those pixels is to take  the beginning point, take the length of the mustache, take the inflection point where  the curl is, calculate the curl strength using a curve, and then stroke that line. Repeat  15,000 times. And you have to do that every frame, every 16 milliseconds. It's pretty  challenging to do even with the hardware acceleration that the canvas provides because  of there are so many calls consecutively. So let's switch back. Let's see if I can do  it this time. There we go. What just happened? So I kind of gave you an overview of what  happened and how the basics of the framework work. When we did the initial implementation  of this, we got around 10 stashes per second. We'd like that to become a new sort of benchmark  in JavaScript performance. So if you could take that back and start profiling your code  in stashes per second, that would be great. Clearly not good enough for a mustache rendering  service. So the first thing we tried to do was we thought maybe we can render the mustaches  by color group. So if you saw the mustaches, you noticed that they're not all one color  because that's not going to look natural. It would just look like one big mass. So there's  a little bit of variation. So we had, I think, between five and six colors for each complexion.  And we would vary those based on just sort of randomly distribute them through the stash.  So we thought maybe we can speed up. Instead of doing 15,000 strokes, maybe we can do all  the colors and stroke them all at once. Canvas has a great API where you can draw a line,  move, draw another line, and then stroke it all at once. And in a lot of the games that  we've done and projects in the past, we found really good performance benefits by using  that technique where rather than doing individual operations on the canvas, you group as many  together as possible and then do them all at once.  So sort of the four different colors right here, you take the first implementation that  we did where they're interspersed. You'd have different colors. We decided to sort the array  so that we would have all the colors sort of in one group. We could stroke them all.  We thought this is going to be fantastic. And the performance was great. We got great  performance out of this. But the problem is you can see the stash on the left has sort  of got that natural look to it. We didn't realize that by grouping them by color, we're  basically saying all the colors are going to be sort of on top of each other in layers  because it doesn't look right at all. It actually doesn't look that bad here, but if you had  darker colors on top, it just doesn't look right. So we thought, well, that was a good  try. Clearly not going to cut it. So then we thought, let's cache the stash.  So the approach here, and it's one that we've used on a number of games, is when you have  a sequence of frames that need to get rendered but will be reused, you can actually draw that  to an off-screen canvas and extract the image from the canvas and sort of create your own  little mini sprite where you have the frames for each image that are rendered, and then  you replay that. So in this case, we also had a growing mustache animation. So you can  see up at the upper left, it's sort of barely starting to come in. And then each frame,  it gets a little longer, a little longer, and you have the full stash. And with this  technique, we were able to get the 60 frames per second, which Robbie calls a stash stash.  It is stash stash. There's no other way to say it. That's also a benchmark when you hit  60 SPS at stash stash. So let's switch back to the app. So this is, I'll show you the,  we've got our mustache here. So this is the render by color. We'll show you that. So you  can see the, a mustache does not look great. So we've kind of got all the colors. Let's  try a different color. Some of them, it really does not look great because they're grouped  together sort of all in one layer. Switch back to this guy. So let's turn on the pre-render.  The challenge with the pre-render, and you sort of have to find the right time to use  it, is that even though it's going to give you that 60 frames a second, it does take  a little bit of time to pre-render all those frames, and then you catch those results.  So we'll click the aroma. You'll see that it basically does nothing for a few seconds  while it's caching the result. And then you can see the smooth stash-tastic animation.  And silky. Just like you want it. So you can see that that is much, much smoother than  the original implementation. And it's because we're basically just replaying that series  of frames that we've already rendered. We thought, go ahead. We thought we'd have a  little bit of fun with the webcam API. So we added that feature in. Let's see if we can  do it. Any volunteers? Should we make Robbie do it?  I'll do it.  There you go. All right. So that's a lot of stash for me. I'm not complaining. I think  the lighting might be... We do have an enhanced too to help with the lighting. Let me turn  that on. Oh, there we go.  Oh, yeah. That's way better.  There we go.  Robbie looks amazing. I think we can all agree.  So this is coming for all of you. So keep an eye out for StashKit. You can stay tuned  on thinkpixelab.com. We really will deploy this. We've actually been working on this  for on and off for like two years.  We've been having fun. We felt like maybe we could do eyebrows. There's one sort of  Yosemite Sam thing that looks like it could be a toupee. Although David Hasselhoff definitely  does not need that.  But it doesn't hurt.  Yeah.  All right. All right. So yeah, with that we have six days BS and we have perfect rendering  as you can imagine. So, you know, I think the things we talked about today in terms  of performance are probably things that you sort of understand. But if you took one thing  away today, it would be to write great code that you can understand that other people  can understand and then optimize 3% of it. And then when you're optimizing that 3%, this  is our approach. We look at the hotspots using the best tools. IE has, we showed you the  IE performance tools because that was a project for Internet Explorer. Chrome and Firefox  also have great DevTools. All of them I think have a great profiler now. A lot of times  you will find that the performance just won't get better by refining your code. So that's  when you get to think creatively. You think about the approach that you're taking or there  are other ways that you can accomplish the same thing. And then last is an invitation  to experiment and refine because this is where you learn and it's also where you typically  tend to solve the hardest problems. And I think that is it.  So yeah, we hope you guys have enjoyed the talk. If you have feedback, we'd love to hear  that. If you have questions, we're open for those as well.  Oh, there's a question. All right.  Let's take it. Go ahead.  In fact, Contrudeur's shading thing, I understand it wasn't an option because it brought us  support, but did you play with WebGL at all?  It wasn't an option, so we didn't play with it. Today I think we would. This was about  two years ago and there was just not a lot of great WebGL support at the time. Now there  is. Also, there's a project with Microsoft and IE, Internet Explorer in particular did  not support WebGL at the time, so it would have been awkward for them had we shipped  a game with WebGL. But that really is the right thing to do for sure. And that's how  we would do it today. Definitely. That's a great question.  Yeah.  Performance is usually quite critical in games. How did you find moving from doing normal  software across into playing with it?  So I forgot we're supposed to repeat the question. So the question was performance is obviously  a critical part of writing games. How did we find the process of going from sort of  typical maybe line of business software to creating games and maybe what are the different  kind of performance constraints that you deal with there? Do you want to take that one?  Yeah, at least for me, I came from a server background where I was working on server code  and there actually is a big performance focus there because you have to handle so many concurrent  users. So for me it was sort of a natural, I've always loved performance. I think the  challenge is sort of what we talked about is not getting overly enthusiastic about performance  and really letting the profiler guide that process of optimization. I think it's so easy  as developers we get excited about maybe an article that we read that says like this little  JavaScript tip led to huge performance gains and really knowing when to use that. In most  cases it's probably not necessary and it will over complicate the code. And so I think as  a developer over my career it's more of learning when the right time to optimize is and really  using the tools to let that guide the choice and where to optimize.  That's great. Yeah.  When you are rendering out what you're seeing onto the campus, is your render loop basically  just loads and loads of manual calls to go every little bit just in terms of primitive  like rectangle and stuff? Are there any libraries for working with campus that you recommend  to use? Do you want me to take that? Sure, you can start.  Yeah. So the question was when you're working with Canvas, so for those of you who aren't  familiar with it, Canvas is basically just a surface onto which you can draw whatever  you want. You have like open season with pixels onto Canvas in HTML5. And the question was  is there a library that we like to use or are we using the native Canvas API? We've  done kind of through several libraries. So there's one by Grant Skinner that, what is  that one called? EZL.js, which we use for a couple of early projects which we liked.  Recently we've been using a kind of a gaming framework called Phaser.io, if you guys are  familiar with that. And that one actually has WebGL support for both 2D and 3D if it's  available. It's really cool. We're actually really happy with that one. With Cut the Rope  where we were doing, and also with ContraJour we were doing a port. And so because we really  needed to be stay in sync with the code that that developer had already written, it wasn't  as easy or didn't make as much sense to use another framework. So both of those are, we're  just rendering directly with the Canvas API. Yeah, and in terms of primitives versus, I  think it's sort of a mix. In games there are a lot of sprite based animations, so a lot  of it is drawing images. And then there's some primitives, whether it's polygons or  lines or things like that. I think you had a question.  When talking the physics engine from Objective C to JavaScript, did you have to change anything  in the algorithm? Is there any mass computation that's inherently impossible or very hard to  do in JavaScript? It's a great question. Yeah.  So the question was, were we able to take the Objective C physics engine and use that  directly in JavaScript or did we have to modify any of the algorithms? We tried not to modify  any of the algorithms. We tried to focus on sort of low level optimization just because  we were afraid if we changed the algorithm that it would change the results. And we wanted  the game to feel identical to iOS. But we had to do a lot of optimization. So those early  code examples where we talked about removing vector allocations, removing arrays, being  really careful about calling functions inside tight loops where the physics engine was  operating, those are sort of the areas that we had to focus on. So we didn't change the  algorithms, but we had to do a lot of performance work to get it to run at high speed. Go ahead.  Are there any good tips on, you know, you spoke about benchmarking, but you really can benchmark  this. Some canvas operations are a lot slower than others. And all the Chrome or whatever  profile will tell you is time spent on canvas, but maybe you could make it faster by having  it a pack and not semi-transparent or things like that. Are there any good ways to profile  in that area? Yeah, that's a great question. So the question is, are there any good ways  to sort of narrow down the areas when you're drawing to be more specific and find the hotspots?  And I think for us it was a little bit of trial and error and a little bit of reading.  There's lots of good resources on the web that talk about tips for performance. We went  through a lot of those and some of those panned out for us. Some of them didn't, depending  on the platform. One in particular I can remember was that there was a recommendation to use  multiple canvases and sort of stack them on top of each other. We had a hard time getting  the performance out of that. It may have changed now. Another sort of interesting thing is  that because the browser engines are improving so quickly that it really is a moving target,  like one technique that works today may not work tomorrow. And so it is better if you  can find sort of the approach-based methods or algorithmic approaches where they're always  able to do better. And I think for us it was just more of a process of trial and error  combined with sort of some guided points to start.  Yeah, that's exactly. Any other questions? Well, we've had a lot of fun, so thank you guys.  Appreciate it.  Thank you.
Joel_Fillmore_-_Facial_Hair_Road_to_Faster_JavaScript_0_7871.png Joel_Fillmore_-_Facial_Hair_Road_to_Faster_JavaScript_7872_7931.png Joel_Fillmore_-_Facial_Hair_Road_to_Faster_JavaScript_7932_10223.png Joel_Fillmore_-_Facial_Hair_Road_to_Faster_JavaScript_10224_10979.png Joel_Fillmore_-_Facial_Hair_Road_to_Faster_JavaScript_10980_13679.png Joel_Fillmore_-_Facial_Hair_Road_to_Faster_JavaScript_13680_13991.png Joel_Fillmore_-_Facial_Hair_Road_to_Faster_JavaScript_13992_14207.png Joel_Fillmore_-_Facial_Hair_Road_to_Faster_JavaScript_14208_14351.png Joel_Fillmore_-_Facial_Hair_Road_to_Faster_JavaScript_14352_14399.png Joel_Fillmore_-_Facial_Hair_Road_to_Faster_JavaScript_14400_15011.png Joel_Fillmore_-_Facial_Hair_Road_to_Faster_JavaScript_15012_16343.png Joel_Fillmore_-_Facial_Hair_Road_to_Faster_JavaScript_16344_27251.png Joel_Fillmore_-_Facial_Hair_Road_to_Faster_JavaScript_27252_35879.png Joel_Fillmore_-_Facial_Hair_Road_to_Faster_JavaScript_35880_36467.png Joel_Fillmore_-_Facial_Hair_Road_to_Faster_JavaScript_36468_36515.png Joel_Fillmore_-_Facial_Hair_Road_to_Faster_JavaScript_36516_36587.png Joel_Fillmore_-_Facial_Hair_Road_to_Faster_JavaScript_36588_36911.png Joel_Fillmore_-_Facial_Hair_Road_to_Faster_JavaScript_36912_37103.png Joel_Fillmore_-_Facial_Hair_Road_to_Faster_JavaScript_37104_37175.png Joel_Fillmore_-_Facial_Hair_Road_to_Faster_JavaScript_37176_37295.png Joel_Fillmore_-_Facial_Hair_Road_to_Faster_JavaScript_37296_37583.png Joel_Fillmore_-_Facial_Hair_Road_to_Faster_JavaScript_37584_37655.png Joel_Fillmore_-_Facial_Hair_Road_to_Faster_JavaScript_37656_37715.png Joel_Fillmore_-_Facial_Hair_Road_to_Faster_JavaScript_37716_38267.png Joel_Fillmore_-_Facial_Hair_Road_to_Faster_JavaScript_38268_38363.png Joel_Fillmore_-_Facial_Hair_Road_to_Faster_JavaScript_38364_38615.png Joel_Fillmore_-_Facial_Hair_Road_to_Faster_JavaScript_38616_38675.png Joel_Fillmore_-_Facial_Hair_Road_to_Faster_JavaScript_38676_38735.png Joel_Fillmore_-_Facial_Hair_Road_to_Faster_JavaScript_38736_39215.png Joel_Fillmore_-_Facial_Hair_Road_to_Faster_JavaScript_39216_39539.png Joel_Fillmore_-_Facial_Hair_Road_to_Faster_JavaScript_39540_39971.png Joel_Fillmore_-_Facial_Hair_Road_to_Faster_JavaScript_39972_41291.png Joel_Fillmore_-_Facial_Hair_Road_to_Faster_JavaScript_41292_41615.png Joel_Fillmore_-_Facial_Hair_Road_to_Faster_JavaScript_41616_41807.png Joel_Fillmore_-_Facial_Hair_Road_to_Faster_JavaScript_41808_42431.png Joel_Fillmore_-_Facial_Hair_Road_to_Faster_JavaScript_42432_50015.png Joel_Fillmore_-_Facial_Hair_Road_to_Faster_JavaScript_50016_52847.png Joel_Fillmore_-_Facial_Hair_Road_to_Faster_JavaScript_52848_53807.png Joel_Fillmore_-_Facial_Hair_Road_to_Faster_JavaScript_53808_54083.png Joel_Fillmore_-_Facial_Hair_Road_to_Faster_JavaScript_54084_54191.png Joel_Fillmore_-_Facial_Hair_Road_to_Faster_JavaScript_54192_54515.png Joel_Fillmore_-_Facial_Hair_Road_to_Faster_JavaScript_54516_54719.png Joel_Fillmore_-_Facial_Hair_Road_to_Faster_JavaScript_54720_54839.png Joel_Fillmore_-_Facial_Hair_Road_to_Faster_JavaScript_54840_55019.png Joel_Fillmore_-_Facial_Hair_Road_to_Faster_JavaScript_55020_56555.png Joel_Fillmore_-_Facial_Hair_Road_to_Faster_JavaScript_56556_56627.png Joel_Fillmore_-_Facial_Hair_Road_to_Faster_JavaScript_56628_62327.png Joel_Fillmore_-_Facial_Hair_Road_to_Faster_JavaScript_62328_64667.png Joel_Fillmore_-_Facial_Hair_Road_to_Faster_JavaScript_64668_65807.png Joel_Fillmore_-_Facial_Hair_Road_to_Faster_JavaScript_65808_67607.png Joel_Fillmore_-_Facial_Hair_Road_to_Faster_JavaScript_67608_69995.png Joel_Fillmore_-_Facial_Hair_Road_to_Faster_JavaScript_69996_71015.png Joel_Fillmore_-_Facial_Hair_Road_to_Faster_JavaScript_71016_83585.png