JavaScript Forensics
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Gardner, Todd H.
Something terrible happened here. Traces of errors litter the floor; memory leaking from cracks in the ceiling. Someone lost their object context in the corner. Everything reeks of jank. In this session, a JavaScript error tracking expert breaks down a series of common and complex crimes against web applications. You’ll leave the session armed with techniques and tools to detect, diagnose, and fix your JavaScript web applications. Bring your bugs and let’s fix up our web.
 My clock says it's time to start, so should we start? So I'm super excited. This is my third NDC, but it's my first on one of these really cool suspended stages. And you might have noticed, but I am American. And so I really hope these chains are going to, I'm a little nervous about that. This talk is called JavaScript Forensics, and it's about diagnosing and debugging our web applications. It's about seeing how it all goes wrong when our customers are actually using it, not when it's in our local development environments. My name is Todd H. Gardner. This is what I look like on the Internet. I spend a lot of time thinking about debugging JavaScript, so much so that about two years ago now, I started a little company called TrackJS. And with TrackJS, I spend, I help my customers understand how their web applications are failing in their customers' hands. I see all of the common things that they run into, and just about every customer has a very common set of problems that they talk about. And so that's what we're going to talk about here today. That's what I'd like to share, is six of the most common web application errors that affect production JavaScript applications. And they can affect your applications maybe without you even knowing about it. Maybe they're only occurring in production, and only for a certain small percentage of your users. So that's what we're going to talk about today, how to see them, how to understand them, and how to fix them. So let's get started. These are our six errors. I've attached awesome mug shots to each of them so that we can better identify with them. Because a JavaScript error is kind of this weird abstract thing, we can't dislike a message. So I put caricatures on these people so that we can start thinking about them. And here's the first one. Script error. If you've ever tried to understand how your production JavaScript fails, you've probably run into this right away. Because this is by far the most common message that we ever capture from our production JavaScript. But it doesn't really tell you much. Script error doesn't talk a lot. It just says something broke. But you don't get to know why, and you don't get to know where, and you don't really even get to know who. Something is broke. It masks and drowns out all other useful information that we get from the page. So this is the first error that we have to capture. So now I'd like to show you our crime scene. This is a little app I built called Soliloquy. Has anybody used Soliloquy? I'm sure not. We're going to be huge next year. Soliloquy is kind of like Twitter. It's got a timeline like Twitter. You can share your thoughts like Twitter. It's got inline ads like Twitter for awesome things like Hasselhoff and Bacon and Bears. But it leaves out the most irritating part of Twitter, other people. Because with Soliloquy, it's all about you. Like I said, this is going to be huge. Next year, giant booth out here. So this is our crime scene. But Soliloquy has some errors. And so this is our crime scene that I'm going to use today to start debugging into what's going on here. So let's pop open our console. So here, when I refresh the page, the Chrome debugging tools immediately show me there's an error. But the error that Chrome is able to report here is different than the error that is actually available to us in the JavaScript layer. When I go and attach an error handler to Window on Error or I add error listeners, what I'm able to capture programmatically isn't this good information here. It's simply Script Air. And Script Air doesn't tell us anything at all. I know that, hey, Window blew up with the message Script Air. And I don't have a stack trace. And I don't really know what file it came from. And I don't really know what line it came from. And I don't know what column it came from. And I don't know anything at all. Something broke. So this is the first thing that we really need to dig into. And Script Air happens a lot across all browsers. When you first start tracking what errors are coming from your client-side application, this is going to totally flood in. And it's going to happen on all browsers. Here I have a list of modern ones that's happening too. Happens on all versions of I of Internet Explorer, going back as far as it's supported JavaScript. Oops, too far. So how do we fix this? Because there is an underlying error that Chrome is actually willing to tell us about. We'll talk about that one later. But we need to figure out why is the browser obfuscating this Script Air. And so if we look at the file this is coming from, this sidebar ad provider, what's important to note here is not that what's in the file itself, but where it's coming from. Soliloquy right now is being hosted off my local host. Sidebar ad provider is being hosted off of this other domain. I'm pulling it in from my advertising network, which on this case I'm hosting at ads.local at port 3000. And so I'm pulling this script in from a different domain, a different origin. The browsers are set up with an origin policy that is trying to isolate information from leaking between different origins. You might have run into this with dealing with iframes or making Ajax's on to it or trying to make an Ajax request to a different origin, like trying to post to a different domain and you have to add cores headers. The same problem applies to JavaScript that you load from other domains and errors that attempt to leak out from a JavaScript file loaded separately are obfuscated this way. But fortunately, new browsers, we can teach it to trust it. The first thing is that the script files themselves need to be served with the cores headers. The cores are the cross origin request headers. You serve those. It's a header that you put on the request. You can easily configure Apache and IIS and Nginx and CloudFront and Azure and all the different things to serve with those. But that's half the problem. The other half is actually teaching our application to look for them. So let's dig into a little bit of code. This is the main index file for Soliloquy. There's just a little bit of bootstrap here to make it not look disgusting. At the bottom here is where I'm loading up all of my different scripts. Here I'm pulling in some basic libraries from a CDN. I'm pulling in jQuery and moment and underscore and backbone. And then I'm pulling in the minified version of my own JavaScript file. Then I'm loading advertising providers from this other network. Now, being that all of these are other domains, even my own scripts here are from a different domain. They're all hosted on a CDN. I need to tell the browser to trust all of these for me. So we do that by populating a new attribute on this, not anonymous, called cross origin anonymous. You can also put a different value other than anonymous if you want to do some authentication stuff. But for most cases, we just do cross origin anonymous. And what that's going to educate the browser to do is that when it's downloading this file, it's going to check for the course headers. And if they're there, it's not going to use its isolation mechanisms. It's going to let real errors through. Caution though, if you put this attribute on these scripts and you don't serve with course headers, the script fails to load entirely. It just will not load. It treats it as a bad request. So you need to make sure that you've added the headers appropriately before decorating this and not just do this blankedly to all scripts. So I put this on all of our scripts and reloaded. And now if I rebuild the page, you'll notice that Chrome still has this error. We haven't actually solved the underlying issue. But programmatically, we're actually able to capture some real details. I don't just know script error anymore. I know getRandomAd was not defined. And I know that this came from this particular sidebar ad provider file on line 20 with this message and a full-on stack trace. And I can understand all of these details programmatically so that I could pull them back to my network, log them, aggregate them, understand what's really going wrong for my customers. That is script error. Script error, the primary problem this is causing is browser obfuscation. The browser is trying to protect information from leaking between origins. But it's really noisy. And chances are you own both origins. This happens whenever you're using a third-party script or loading your stuff from CDNs, which is really popular right now and I'm sure a lot of you are doing that. But it conceals all the other errors. So when you're doing it, be sure to put the course headers on and use the cross origin attribute. So now let's move on to some real errors. Get rid of just the masking. And this one, getRandomAds is not defined. This is the error we saw on console earlier. Now this error happened really suddenly. Now it's a little quiz been launched for months. And you know, we haven't really made a real change to the application recently. But all of a sudden yesterday, bam, we just started getting hammered with errors. They were just flowing into our system faster than we could handle. And we don't really know why. So let's take a look. So the root issue here, or so the error that we're capturing here in Chrome, is this uncaught reference error. GetRandomAd is not defined. Now this JavaScript is again being loaded from our advertising provider. We don't own this script. We're just referencing it. I pay a monthly service to get awesome ads for bacon and Hasselhoff on my page. And in exchange, they just tell me to use this script. But now it's broken. If we dig into it, the stack trace tells me that, hey, something happened here in this sidebar ad provider and then a bunch of jQuery stuff that I don't really need to understand. Let's dig into the code and see if we can understand what's actually happening. Here in here, we're calling this function called getRandomAd. And getRandomAd isn't a thing. So if we reload with a breakpoint, we try to inspect this. GetRandomAd. It's definitely not a thing. So let's take a look at their source code and see if we can figure out what's going wrong. Let's hold a copy of the sidebar ad provider down. And here's the line in question, this getRandomAd. Now down here at the bottom, there's a function called getRandomAds, which is obviously just a typo. So if we just correct this typo for them and reload it, hey, we get a bunch more ads. Now I have a whole sidebar full of advertisements. The problem here, I mean, it was just a simple developer typo like I've made a thousand times. But the problem is that I didn't know that it was going to happen. I wasn't planning a change cycle. I wasn't prepared to get a 3AM phone call saying my site was down. I relied on a third party. And that third party can break your application whenever they want. And so when we're building modern web apps, we are leveraging probably dozens of third party services today. Social providers, payment processors, social networks, I've probably already said that, analytics tools, all kinds of maybe content injection, all kinds of different things can come into our page. But every time we trade that away, we're paying for value that somebody else is building, we're introducing some risk into our application of what happens when they change their code, which can break our application without us knowing, because the entire application is assembled in the user's browser. So this can introduce all kinds of uncontrolled changes. Any time you are using an externally hosted script, something you don't control, this is something you lose. And so the performance and benefits you get with third party scripts come with this risk that you need to recognize. So now let's move into some errors that are actually ours. So what we started getting in our error logging was that a small percentage of customers would get this error, that they cannot read property destroy of undefined. And the profile of this one was a little bit different. We when we would get this error, and we would only get it like maybe 10, 20 percent of users, but every time we got it, it looked like they were trying to delete a statement out of their timeline. And we had a really direct correlation. So we understand some context of how the user got to this problem. And so let's try and recreate that. So here's my timeline. Whole lot of interesting lorem if some statements in mind. But if we go ahead and try and delete one of our statements out of here, we see an error pop up in the console right away. And it does appear to be the error that we were logging on our servers. This cannot read property destroy of undefined. Here in statement view JS line 30. Now here, I want to just take a second, because this is my production version of soliloquy. And in my production version, I don't think I remember referencing statement view JS directly. My code here is only referencing one file that composes my custom code. And that's the soliloquy.min.js. Yet Chrome here is able to recognize that this error is coming from statement view JS, which is one of my base level source files. If we open this up a little bigger. The reason for this, because you'll see that I'm only loading soliloquy.min.js, and it is a minified one line file that would produce nasty error messages like C is not defined. But Chrome sees that in this soliloquy.min.js, I have this source map attribution. Source map is a newer concept. It's been around for probably the last three years. It is still being formalized and it is still changing. But source maps allow you to bundle together your JavaScript for your production level application to reduce it to make it smaller, put it on one line, obfuscate your variables. So when you debug stuff to have a pointer file back to where the original files came from. And if everything is publicly accessible, if you make your map publicly accessible and your original source file is publicly accessible, then Chrome does some awesome stuff and interprets it live on the fly on your behalf. And so Chrome caught the message that cannot read property, destroy, undefined. And the actual stack trace that was generated, see if I can put this in here. The actual stack trace that was generated was not that nice thing. It was this one thing, cannot read property, destroy, undefined, at soliloquy.min.js, line one, column 2338, which is not very helpful. I don't know if you've ever tried to understand what's going wrong by interpreting the column in a JavaScript file. But sometimes that is not fun. Chrome, when we loaded this file, instead interpreted this source map live for us and tracked down what was at that column, where did, what was the file that originally came from and even lets us navigate to the original source file, which is pretty awesome. So now let's actually take a look at solving this bug. Sometimes we have source maps because we have the original files, it becomes way easier to debug our apps in production. So here's the underlying problem. When we try and delete a statement out of our timeline, we cannot read property, destroy, of undefined. Now destroy is being called right here on this.model. And right away, this, this is the most common JavaScript error I still run into. And you can see a smell right away. Because we've passed a function into a callback, and in that function we're using the word this. And so this is not what you think it is. This is something else. And you don't know what it is. So when we call this, this is probably something like window or document or an event or something we don't even know. And so that, this doesn't have a thing called model and it probably doesn't have a thing called destroy. And so the whole thing breaks down. Now a really common way, statement view, a really common way to solve this a few years ago would have been to do something like var that equals this or var self or maybe even var model equals this.model. So we store the value out in the outer closure so that when we reference it in here, we don't reference it by this. And this would totally work. This is, this is totally cool. However, if you are using newer browsers, I really like bind. Bind allows us to set, to hard code to force the value of this when a particular function gets called. And so I am saying, I'm declaring this function here and then I am binding its context to the, to this, to the outer context when it gets called. And when I execute it now, this is the correct value. If you do need to support older browsers, I'm sorry, you should probably take a look at a utility library like lodash or underscore or hopefully very soon they will be the same thing which provides a whole bunch of utility functions that allow you to make this really safe to use everywhere. So rather than specifying.bind this in the underscore or lodash way, you would say underscore bind this. Which is the safe will work in all browsers. You don't have to worry about it way and will fall back to the native implementation if it exists. And it still works. One of the other things you might notice as we are debugging through as long as we have a clean console here is I print a lot of console messages out. And I do that for a couple of reasons. One is this is a demo and it's a lot easier for y'all to understand what's going on if I print out like a bunch of console messages. But I also do this in my real applications because I'll build soliloquy and then I might not touch it for weeks or months at a time. And when I come back to it, it's really helpful to see what are the underlying actions that my application is taking when things are happening. It is an invaluable debugging tool and it is really easy to turn off if you don't want them in production and it's really easy to polyfill if you need to worry about older browsers. And we can talk about that later if you're really interested. All right. So now we have a functioning application. The user can delete all of the statements they want out of their timeline. I don't know why they'd want to. Clearly this content is gold. Context airs. This is still the most common JavaScript RIC. Any time that you are using a callback or a promise or you are passing a function into some asynchronous event, be it an Ajax event, a timeout, an interval, a click handler, whatever and you reference the word this, this has a high potential of happening. You can usually discover this through testing. There's nothing magic about production in this happening. However, because JavaScript is so asynchronous and dynamic, there are so many edge cases through any respectively sized application that you probably will not be able to test for every potential edge case. And if you haven't thought through every potential order that your callbacks could come back in, you could run into this problem. And in fact, many of my customers do. So text substring is not a function. Something's gone wrong again. We got a call from a particular customer saying that they were having a really bad time. They were getting this error and it showed up in our logging. We saw it happen a lot. Over 2,000 times last week this happened. A.text.substring is not a function. But it only happened to one dude. Like one person just had a really bad time of it. So this isn't a widespread error. This isn't like something that we just failed to test for in all circumstances. But there's definitely some edge case, either with our code, their data, or their browser that is causing this to break down. So let's see if we can troubleshoot this and understand what's happening. So let's load up that customer's data. So when we load soliloquy for them, they still have ads. So that's good. Getting a little revenue. But they don't have anything at all to look at. They don't have any status updates. Here I see the error that we've been recording, this a.text.substring is not a function. If we try and type in a status for them, well, that shows up. But nothing else does. So maybe let's dig in and see if we can understand what's happening here. Let's take a look at this error here in the console. So with Chrome, we can expand an error that happens in the console just by dropping it down. And we can see all kinds of stuff in it. So we have an anonymous function, a function that wasn't named here in statement model blowing up. Looks like we have a bunch of underscore functions happening that I don't really know what's going on there. Everything from a jQuery callback is down here at the bottom. And a jQuery Ajax transport. So that's kind of interesting. We see that the root cause of this whole thing is probably an Ajax callback of some kind. Now if I wanted to go even farther, like what started this Ajax? What are we requesting here? In the latest couple of versions of Chrome, they introduced this async button right here, which is, in my opinion, the coolest thing that they have ever done. So if we go ahead and turn that on and reload this, now we can take a look at this stack trace again. And we see the same stuff is on the top. It's a text substring is not a function and a bunch of other stuff. But here, where the stack trace used to end, we see that this was separated by an XMLHP request send event, an Ajax call. That Ajax call was started by this file right here, or was last touched by this file here. It was initiated through a jQuery Ajax transport by Ajax send from backbone, from underscore, all the way to a jQuery ready. So there is something happening in jQuery ready that's firing this thing off when we first start, an Ajax goes off, some period of time later it comes back, blows up, and we end up with this function. So if we look at the network, and filter this down to XHR, it's probably one of these two first requests that are happening right at page load. So let's take a look at the code itself and see if we can figure out what's happening. So because we clicked on this, Chrome interpreted our source maps and gave us the actual file. You'll notice that some of these lines are grayed out here, and that's because there's no corresponding line in the original source map to this file. And so we could not put a break point on certain lines, because Chrome doesn't know how to handle that. But it renders it so we can see what's going on, and it points as to where this error is happening. So this is a parse function, and a parse function that takes a response. So based on that stack trace, this seems like we are trying to parse our XMLHP request, or parsing the server response. And this has some expectations about what this is supposed to look like. Response is something that we can enumerate across. We are looping over it. And for each thing in this list, it's expecting those things to be objects of some kind, and those objects to have a field called text. And text, we are calling a substring on. And so all of those expectations about what our data looks like is baked into this. So if we go and look at what's actually in this request, we can expand this out, and the response is indeed something we can enumerate over. We do have an array, and we do have an array of objects, and those objects do have a text property, but that text property is not a string. And so what the server is actually giving us back, and what our JavaScript is expecting, fell out of sync. And so this can happen for all kinds of reasons. In big enterprises, it's really common for the team responsible for building the API, and the team responsible for building the front-end code, or might be different teams, and might have different release schedules. There you have the possibility that somebody changes a type or changes a shape of an object. If this is a third-party API, perhaps I'm calling into Twitter's API to do something, Twitter could change its API out from under me. Twitter probably wouldn't, but there's all kinds of other vendors that could. Let's take a look at how we could actually deal with this particular issue so that we could let this customer have a little bit better time. So here we have our parse function, and all of the expectations in it. Let's see if we can reduce those expectations or make it safer. So the first thing I'm doing is underscore map, which should handle non-enumerable responses just fine. We can't do anything, and we'd get an empty array. And so that should be okay. The next thing is that we expect item to be an object. And so we can fake that out. We could say, you know, item is an item, or it's an object. That's just a little handy piece of syntax that you can force an object to be at least something, and it won't be undefined. And then we can do the same thing here where, you know what, first we should really check to see if text.item is a string. So I'm going to use another underscore function, but there's all kinds of ways to do it. I'm just going to use the helper isString, item.text. If it's not a string, we should do something different. If it's not a string, like it's a number in this case, we should assign it some value. We should say item.text is item.text or an empty string. And this is just a truthy value. So when text is 42, which is the case we're dealing with, text will still be a truthy value. But the truthy values in JavaScript tend to have a two-string method. And the empty string case allows us to solve for maybe an edge case where we have item.text is undefined or item.text is not a number or other weird things like that that might not have two-string implemented on them. And so with all of those things set, we should have normalized this data down to item.text should at this point be a string and have the function substring on it. So if we save this and reload, take off that breakpoint, we can now render all of their old, maltyped data. Maybe they don't want the number 42 printed in over and over again. But the client side is trying to be a little bit more tolerant of changes in their API. If you wanted to be a little bit more strict, we could also just make a simple change and say, hey, if this isn't a string, return empty string, and we should probably log something, console.error. Got bad data. That might be the little bit more stricter way to handle this problem. Where the new message that I put in is still rendered, however, all the old messages are not, the page continues to function, and we have an error that's a little bit more descriptive about what's happening, actually getting recorded. Apparently that causes another error downstream, though. So I'm going to go back to the way it was. All right, so now they can see all of their data, they can see their awesome inline ads, and everything is happy. So this is a dirty data error. And it can be caused by a couple of different reasons. It can either be caused by unvalidated input, which I talk about here, where somehow data got into the system in a bad form. Somebody was able to post a number value rather than a string value, and we had all kinds of downstream problems from it. In this case, it was a simple error, but this can cause all kinds of issues like cross-site scripting vulnerabilities, major production faults, all kinds of things like that. This can also be caused just by simple lack of communication between teams. Like I mentioned earlier, the server-side people not talking to the client-side people when they change an API and a contract gets malformed. This can be really difficult to test for. I have struggled with validation problems on every single client-side app I've ever worked on, and it's always a really tricky problem to solve. Window.inline ads is not a function. So let's dig into this error. Now, this error has a little bit different profile for us. More normal traffic, because Soliloquy is amazing, is 20,000, 30,000 hits a day. We don't have a lot of these errors. This isn't a super big deal. But 134 people, or 129 people encountered this error 134 times. It's not insignificant. This is a real problem that is affecting some small segment of our user population. Let's try and understand how inline ads could be an error. I'm going to go back to our other account. Cool. So, when I load our default account here, I'm not seeing that error. I see inline ads, so I'm not really sure how this could be a problem. So if we search our code for references to inline ads, the only place it shows up is in our main bootstrapper code, right here. So we are setting up our application on the original state, and we're calling a function called window.inline ads. So why wouldn't that ever be a function? So window.inline ads is the API given to us by one of our advertising providers. The advertising provider that I load right here. So what would happen if that failed to load? If I just, you know, fake this out and comment it out, well, there's our error. A.inline ads is not a function. I mean, that appears to be what's happening, but look at the UI when this happens. That's a terrible user experience, right? Not only did I not load any inline ads, I didn't load any sidebar ads, I didn't load any of their content, I just gave them a big empty screen, and they think all their data is gone. It would be terrible if a user ran into this problem. So, so inline ads appears to be failing some of the time, and that is really, really, really likely. So, based on our metrics, about one request out of every about five to ten thousand somewhere in that range, depending on the customer, has some asset loading problem. One JavaScript file from your page will fail to load. Not all of them. Just one. What would happen if one file failed to load? So, I had a project where we loaded Stripe. Stripe is a credit card payment processor, and Stripe works on the page by basically capturing the form input and stealing the credit card, not stealing, taking the credit card data, and forwarding it to the Stripe servers and getting a token back and then returning that. And so, we had a form set up that, you know, took the credit card data and then would just, you know, send on the token when it was all done. But some percentage of the time, for a lot of our customers in Australia for some reason, some of the time Stripe.js would just fail to load. It could have been due to a bad network connection. Maybe Stripe CDN was having a problem. Maybe they had some sort of corporate firewall that was blocking it. There could be all kinds of reasons. But when Stripe failed to load, the behavior of that page was to take the credit card details and post them in clear text to my server. Because I wasn't expecting that to ever go wrong. I thought Stripe had that handled. But in those rare occasions where Stripe.js fails to load, I was making a major security vulnerability. Think about your applications. What would happen if one of your scripts fails to load? It's probably not a big deal if you bundle everything together, or it's not a big deal if there's a catastrophic failure and nothing loads. If you're building a single page app, the user probably gets a big white screen. Or hopefully you're doing progressive enhancement and everything works great out of the box. But what if a major piece of your content fails to load? What if your transaction processor fails to load? So that's what Soliloquy failed to do here. Is window.inlineads was in my core pipeline. I called this inlineads function and then I called the sidebar add function and then higher up this file I called my main start the application function. So when inlineads failed, everything broke. We just didn't proceed any farther. So what I should really do here is be a little bit safer. Inlineads isn't a critical piece of my infrastructure. If it fails, it's not the end of the world for me. So what we should really just do is just check to see if window.inlineads is a thing. Or rather not, let's check if it's not a thing. And if it's not a thing, let's write a warning message. This way, window.inlineads will only start itself up if it has successfully loaded. And on times when it doesn't load, I will capture some logging statements back to my own systems to give me some indication of what percentage of the time is this happening. So I can go back and complain to them or figure out what exactly is going wrong. So now with this in place and we load up the page, you know, the user, they don't have inlineads, but they at least see their content. And that's what's important here. So loading errors. This happens way more frequently than I ever thought was likely. But all kinds of times in poor network connections, corporate proxies, long distance things, edge networks, American mobile networks on Norwegian cellular networks, you get sporadic times packets drop. And sometimes those drops will happen when you're loading a critical piece of your web application. All right. This is the last thing I'm going to talk about today. But it's a doozy. This is when the whole browser crashes out from underneath you. And unfortunately, I don't have a good automated way to capture this because the entire browser is crashing. When you're monitoring a runtime and that runtime crashes, you really have no way to report that. But luckily, if you crash a browser enough time, people complain, usually on Twitter, complaining about Soliloquy on Twitter. And so we heard a few people complaining that, you know, they would use Soliloquy all day. And by the end of the day, it was just so slow, they could barely type into it. And then Chrome would crash on them. And they'd have to restart and they would, you know, lose all of their tabs. And so that sounds like a memory leak, getting slower and slower and slower all day. So let's see if we can find a memory leak in Soliloquy. And there are some really fun tools to find memory leaks in Chrome today. So I'm going to pop over into the timeline tab. And the timeline tab lets you record what's going on in your application or what is the state of the memory of your web application while you're taking certain actions. And so what I'm going to do is I'm going to record a bunch of actions and then I'm going to write a few statements into Soliloquy and delete those statements back out, supposedly putting us back into the exact same state that we were when we started. And then just see how the different parts of the DOM, how the different parts of memory have changed while I'm doing this. So I'm going to restart this one, make sure it's clean. And then we're going to record. And I'm going to say, hey, NDC, some other statement. Let's go drinking. And then we'll delete those and stop. And so now what we can do is we can start digging into this timeline here and understand what's going on. And there's a lot of data here. It's kind of hard to interpret right off the bat. But where I like to start is down here in this graph here. What this is showing us is how much JavaScript memory has been used, how many nodes are in the DOM, and how many event listeners have been attached. And these are all very important things because they impact memory of the browser, how much memory is being used. And so what we can see here at the macro view is that there is a step function going up where periodically a whole bunch of JavaScript is allocated, several nodes are created, several listeners are attached, and it just keeps happening over and over and over again. And it never gets released. So that's a pretty strong indication we have a leak here because the application itself, I believe, should have reset to its normal state. And so we should have dropped to some normal baseline again. So we can understand more about this by starting to zoom in. And this is beautiful. They used classic first person shooter gamer controls. So you get onto your keyboard and with WASD on your keyboard, you just start, I need to show something. Sorry, I need to rearrange this a little bit. Oh, all right, thank you. All right, so now with WASD style controls, we can just start zooming in and navigating this timeline all the way down to microscopic levels of detail about what's actually happening here. And this is so fun. Like I have literally lost hours just playing with applications that don't even have memory leaks. I just want to see what they do. And so you can go in and see what's going on. And so let me back up a little bit. Here's one of those steps that we saw at the macro level. And so as we start zooming in, we see that it's not one step. It's a whole lot of little steps happening in rapid succession. And so we keep zooming in on this and we can see what's actually going on. We see that a timer is going off. So we see timer seven, timer eight, timer nine, like a whole series of timers have gone off. In response to those timers, we're calling this function here in sidebar ad provider. Sidebar ad provider is calling this anonymous function, calling this thing called placing an ad and doing a bunch of stuff. And when this all happens, we're stepping up our memory usage. So if we go in here and look at sidebar ad provider, which we can just navigate to it right from here, we can see exactly what's going on. So we have, let me zoom back in maybe. So we have a set interval. This is that interval we saw going off at the top of our timeline function. We have an interval going off a set number of times. And every time that interval goes off, we have this ad container, which looks like it's just a dom node. And we're clearing out the contents of this dom node. And then we're getting a new advertisement and placing that advertisement. So let's look at what place ad is doing. Place ad is making an Ajax call. It's going out and getting the contents for an ad. And it's pulling it down. And once we have the contents of it, it's sticking it, it's sticking that text inside of the element and attaching a click handler to it. And so it's just doing this over and over again. This is the sidebar ads right here. If we wait a few seconds, you can see them refresh. And every time those refresh, it's just taking all of the nodes that were there and just putting them aside, still with click handlers on them, and putting a new set of ads. And so all of the ads that we've ever loaded on this page are still in this page. They're just not present in the dom at this point. So another way to think about that is over the course of a day, there are literally thousands of advertisements for David Hasselhoff on this page waiting for you to click on them. That's terrifying. So leaky memory. Memory leaks are most often caused when you attach events to parts of the dom that don't always exist there. If you attach an event to a piece of dom that gets replaced at some point later, that event is a pointer to that whole part of the dom that will be kept around until you release that event. There are tons of other ways memory leaks can happen. This, however, is the most common. So there's no good way to monitor for this in general terms that we've found so far. However, periodically checking any sort of apps for kidnapping of your memory, any of your apps that you're doing major development on, I encourage you to tell your boss that you get to play a little game for a little while and just play around the timeline for a couple hours. And you'll be able to find pretty quickly if there are any sort of leaks happening. So this was JavaScript forensics and the six most common kinds of errors that I see affecting production web applications. Script errors are obfuscating the real problems that are happening in your app. Third-party errors, whenever you introduce a script from a third party, you're introducing risk that they can break your app for reasons beyond your control. Context errors, losing what the value of this is as you pass it around into callback functions. Data errors, changing the underlying data contracts, either communications between different parts of your organization or in relying on third parties. Loading errors, way more common than I ever thought. Part of your application failing to load in certain circumstances. Think about what would happen when only part of your asset fails. And finally, memory errors, using two attaching events to detachable DOM nodes and causing the browser memory to balloon and balloon up until it eventually crashes. I encourage you to all look at your app and monitor it. Watch out for these errors. Watch out for these villains and monitor for them. And if you need a new sheriff to help you monitor them, TrackJS can help. So this was JavaScript forensics. If you have questions for me about JavaScript or monitoring or TrackJS, you can get me on Twitter or there's my email. I've also, if you fancy a sticker, I've left stickers on both of the tables here. All I have to do is push a green button and you can take a sticker. Other than that, thank you very much and enjoy the rest of NDC.
 My clock says it's time to start, so should we start?  So I'm super excited.  This is my third NDC, but it's my first on one of these really cool suspended stages.  And you might have noticed, but I am American.  And so I really hope these chains are going to, I'm a little nervous about that.  This talk is called JavaScript Forensics, and it's about diagnosing and debugging our  web applications.  It's about seeing how it all goes wrong when our customers are actually using it, not when  it's in our local development environments.  My name is Todd H. Gardner.  This is what I look like on the Internet.  I spend a lot of time thinking about debugging JavaScript, so much so that about two years  ago now, I started a little company called TrackJS.  And with TrackJS, I spend, I help my customers understand how their web applications are  failing in their customers' hands.  I see all of the common things that they run into, and just about every customer has a  very common set of problems that they talk about.  And so that's what we're going to talk about here today.  That's what I'd like to share, is six of the most common web application errors that affect  production JavaScript applications.  And they can affect your applications maybe without you even knowing about it.  Maybe they're only occurring in production, and only for a certain small percentage of  your users.  So that's what we're going to talk about today, how to see them, how to understand them, and  how to fix them.  So let's get started.  These are our six errors.  I've attached awesome mug shots to each of them so that we can better identify with them.  Because a JavaScript error is kind of this weird abstract thing, we can't dislike a message.  So I put caricatures on these people so that we can start thinking about them.  And here's the first one.  Script error.  If you've ever tried to understand how your production JavaScript fails, you've probably  run into this right away.  Because this is by far the most common message that we ever capture from our production JavaScript.  But it doesn't really tell you much.  Script error doesn't talk a lot.  It just says something broke.  But you don't get to know why, and you don't get to know where, and you don't really even  get to know who.  Something is broke.  It masks and drowns out all other useful information that we get from the page.  So this is the first error that we have to capture.  So now I'd like to show you our crime scene.  This is a little app I built called Soliloquy.  Has anybody used Soliloquy?  I'm sure not.  We're going to be huge next year.  Soliloquy is kind of like Twitter.  It's got a timeline like Twitter.  You can share your thoughts like Twitter.  It's got inline ads like Twitter for awesome things like Hasselhoff and Bacon and Bears.  But it leaves out the most irritating part of Twitter, other people.  Because with Soliloquy, it's all about you.  Like I said, this is going to be huge.  Next year, giant booth out here.  So this is our crime scene.  But Soliloquy has some errors.  And so this is our crime scene that I'm going to use today to start debugging into what's  going on here.  So let's pop open our console.  So here, when I refresh the page, the Chrome debugging tools immediately show me there's  an error.  But the error that Chrome is able to report here is different than the error that is actually  available to us in the JavaScript layer.  When I go and attach an error handler to Window on Error or I add error listeners, what I'm  able to capture programmatically isn't this good information here.  It's simply Script Air.  And Script Air doesn't tell us anything at all.  I know that, hey, Window blew up with the message Script Air.  And I don't have a stack trace.  And I don't really know what file it came from.  And I don't really know what line it came from.  And I don't know what column it came from.  And I don't know anything at all.  Something broke.  So this is the first thing that we really need to dig into.  And Script Air happens a lot across all browsers.  When you first start tracking what errors are coming from your client-side application,  this is going to totally flood in.  And it's going to happen on all browsers.  Here I have a list of modern ones that's happening too.  Happens on all versions of I of Internet Explorer, going back as far as it's supported  JavaScript.  Oops, too far.  So how do we fix this?  Because there is an underlying error that Chrome is actually willing to tell us about.  We'll talk about that one later.  But we need to figure out why is the browser obfuscating this Script Air.  And so if we look at the file this is coming from, this sidebar ad provider, what's important  to note here is not that what's in the file itself, but where it's coming from.  Soliloquy right now is being hosted off my local host.  Sidebar ad provider is being hosted off of this other domain.  I'm pulling it in from my advertising network, which on this case I'm hosting at ads.local  at port 3000.  And so I'm pulling this script in from a different domain, a different origin.  The browsers are set up with an origin policy that is trying to isolate information from  leaking between different origins.  You might have run into this with dealing with iframes or making Ajax's on to it or  trying to make an Ajax request to a different origin, like trying to post to a different  domain and you have to add cores headers.  The same problem applies to JavaScript that you load from other domains and errors that  attempt to leak out from a JavaScript file loaded separately are obfuscated this way.  But fortunately, new browsers, we can teach it to trust it.  The first thing is that the script files themselves need to be served with the cores  headers.  The cores are the cross origin request headers.  You serve those.  It's a header that you put on the request.  You can easily configure Apache and IIS and Nginx and CloudFront and Azure and all the  different things to serve with those.  But that's half the problem.  The other half is actually teaching our application to look for them.  So let's dig into a little bit of code.  This is the main index file for Soliloquy.  There's just a little bit of bootstrap here to make it not look disgusting.  At the bottom here is where I'm loading up all of my different scripts.  Here I'm pulling in some basic libraries from a CDN.  I'm pulling in jQuery and moment and underscore and backbone.  And then I'm pulling in the minified version of my own JavaScript file.  Then I'm loading advertising providers from this other network.  Now, being that all of these are other domains, even my own scripts here are from a different  domain.  They're all hosted on a CDN.  I need to tell the browser to trust all of these for me.  So we do that by populating a new attribute on this, not anonymous, called cross origin  anonymous.  You can also put a different value other than anonymous if you want to do some authentication  stuff.  But for most cases, we just do cross origin anonymous.  And what that's going to educate the browser to do is that when it's downloading this file,  it's going to check for the course headers.  And if they're there, it's not going to use its isolation mechanisms.  It's going to let real errors through.  Caution though, if you put this attribute on these scripts and you don't serve with  course headers, the script fails to load entirely.  It just will not load.  It treats it as a bad request.  So you need to make sure that you've added the headers appropriately before decorating  this and not just do this blankedly to all scripts.  So I put this on all of our scripts and reloaded.  And now if I rebuild the page, you'll notice that Chrome still has this error.  We haven't actually solved the underlying issue.  But programmatically, we're actually able to capture some real details.  I don't just know script error anymore.  I know getRandomAd was not defined.  And I know that this came from this particular sidebar ad provider file on line 20 with this  message and a full-on stack trace.  And I can understand all of these details programmatically so that I could pull them  back to my network, log them, aggregate them, understand what's really going wrong for my  customers.  That is script error.  Script error, the primary problem this is causing is browser obfuscation.  The browser is trying to protect information from leaking between origins.  But it's really noisy.  And chances are you own both origins.  This happens whenever you're using a third-party script or loading your stuff from CDNs, which  is really popular right now and I'm sure a lot of you are doing that.  But it conceals all the other errors.  So when you're doing it, be sure to put the course headers on and use the cross origin  attribute.  So now let's move on to some real errors.  Get rid of just the masking.  And this one, getRandomAds is not defined.  This is the error we saw on console earlier.  Now this error happened really suddenly.  Now it's a little quiz been launched for months.  And you know, we haven't really made a real change to the application recently.  But all of a sudden yesterday, bam, we just started getting hammered with errors.  They were just flowing into our system faster than we could handle.  And we don't really know why.  So let's take a look.  So the root issue here, or so the error that we're capturing here in Chrome, is this uncaught  reference error.  GetRandomAd is not defined.  Now this JavaScript is again being loaded from our advertising provider.  We don't own this script.  We're just referencing it.  I pay a monthly service to get awesome ads for bacon and Hasselhoff on my page.  And in exchange, they just tell me to use this script.  But now it's broken.  If we dig into it, the stack trace tells me that, hey, something happened here in this  sidebar ad provider and then a bunch of jQuery stuff that I don't really need to understand.  Let's dig into the code and see if we can understand what's actually happening.  Here in here, we're calling this function called getRandomAd.  And getRandomAd isn't a thing.  So if we reload with a breakpoint, we try to inspect this.  GetRandomAd.  It's definitely not a thing.  So let's take a look at their source code and see if we can figure out what's going  wrong.  Let's hold a copy of the sidebar ad provider down.  And here's the line in question, this getRandomAd.  Now down here at the bottom, there's a function called getRandomAds, which is obviously just  a typo.  So if we just correct this typo for them and reload it, hey, we get a bunch more ads.  Now I have a whole sidebar full of advertisements.  The problem here, I mean, it was just a simple developer typo like I've made a thousand times.  But the problem is that I didn't know that it was going to happen.  I wasn't planning a change cycle.  I wasn't prepared to get a 3AM phone call saying my site was down.  I relied on a third party.  And that third party can break your application whenever they want.  And so when we're building modern web apps, we are leveraging probably dozens of third  party services today.  Social providers, payment processors, social networks, I've probably already said that,  analytics tools, all kinds of maybe content injection, all kinds of different things can  come into our page.  But every time we trade that away, we're paying for value that somebody else is building,  we're introducing some risk into our application of what happens when they change their code,  which can break our application without us knowing, because the entire application is  assembled in the user's browser.  So this can introduce all kinds of uncontrolled changes.  Any time you are using an externally hosted script, something you don't control, this  is something you lose.  And so the performance and benefits you get with third party scripts come with this risk  that you need to recognize.  So now let's move into some errors that are actually ours.  So what we started getting in our error logging was that a small percentage of customers would  get this error, that they cannot read property destroy of undefined.  And the profile of this one was a little bit different.  We when we would get this error, and we would only get it like maybe 10, 20 percent of users,  but every time we got it, it looked like they were trying to delete a statement out of their  timeline.  And we had a really direct correlation.  So we understand some context of how the user got to this problem.  And so let's try and recreate that.  So here's my timeline.  Whole lot of interesting lorem if some statements in mind.  But if we go ahead and try and delete one of our statements out of here, we see an error  pop up in the console right away.  And it does appear to be the error that we were logging on our servers.  This cannot read property destroy of undefined.  Here in statement view JS line 30.  Now here, I want to just take a second, because this is my production version of soliloquy.  And in my production version, I don't think I remember referencing statement view JS directly.  My code here is only referencing one file that composes my custom code.  And that's the soliloquy.min.js.  Yet Chrome here is able to recognize that this error is coming from statement view JS,  which is one of my base level source files.  If we open this up a little bigger.  The reason for this, because you'll see that I'm only loading soliloquy.min.js, and it  is a minified one line file that would produce nasty error messages like C is not defined.  But Chrome sees that in this soliloquy.min.js, I have this source map attribution.  Source map is a newer concept.  It's been around for probably the last three years.  It is still being formalized and it is still changing.  But source maps allow you to bundle together your JavaScript for your production level  application to reduce it to make it smaller, put it on one line, obfuscate your variables.  So when you debug stuff to have a pointer file back to where the original files came  from.  And if everything is publicly accessible, if you make your map publicly accessible and  your original source file is publicly accessible, then Chrome does some awesome stuff and interprets  it live on the fly on your behalf.  And so Chrome caught the message that cannot read property, destroy, undefined.  And the actual stack trace that was generated, see if I can put this in here.  The actual stack trace that was generated was not that nice thing.  It was this one thing, cannot read property, destroy, undefined, at soliloquy.min.js,  line one, column 2338, which is not very helpful.  I don't know if you've ever tried to understand what's going wrong by interpreting the column  in a JavaScript file.  But sometimes that is not fun.  Chrome, when we loaded this file, instead interpreted this source map live for us and  tracked down what was at that column, where did, what was the file that originally came  from and even lets us navigate to the original source file, which is pretty awesome.  So now let's actually take a look at solving this bug.  Sometimes we have source maps because we have the original files, it becomes way easier  to debug our apps in production.  So here's the underlying problem.  When we try and delete a statement out of our timeline, we cannot read property, destroy,  of undefined.  Now destroy is being called right here on this.model.  And right away, this, this is the most common JavaScript error I still run into.  And you can see a smell right away.  Because we've passed a function into a callback, and in that function we're using the word  this.  And so this is not what you think it is.  This is something else.  And you don't know what it is.  So when we call this, this is probably something like window or document or an event or something  we don't even know.  And so that, this doesn't have a thing called model and it probably doesn't have a thing  called destroy.  And so the whole thing breaks down.  Now a really common way, statement view, a really common way to solve this a few years  ago would have been to do something like var that equals this or var self or maybe even  var model equals this.model.  So we store the value out in the outer closure so that when we reference it in here, we don't  reference it by this.  And this would totally work.  This is, this is totally cool.  However, if you are using newer browsers, I really like bind.  Bind allows us to set, to hard code to force the value of this when a particular function  gets called.  And so I am saying, I'm declaring this function here and then I am binding its context to  the, to this, to the outer context when it gets called.  And when I execute it now, this is the correct value.  If you do need to support older browsers, I'm sorry, you should probably take a look  at a utility library like lodash or underscore or hopefully very soon they will be the same  thing which provides a whole bunch of utility functions that allow you to make this really  safe to use everywhere.  So rather than specifying.bind this in the underscore or lodash way, you would say underscore  bind this.  Which is the safe will work in all browsers.  You don't have to worry about it way and will fall back to the native implementation if  it exists.  And it still works.  One of the other things you might notice as we are debugging through as long as we have  a clean console here is I print a lot of console messages out.  And I do that for a couple of reasons.  One is this is a demo and it's a lot easier for y'all to understand what's going on if  I print out like a bunch of console messages.  But I also do this in my real applications because I'll build soliloquy and then I might  not touch it for weeks or months at a time.  And when I come back to it, it's really helpful to see what are the underlying actions that  my application is taking when things are happening.  It is an invaluable debugging tool and it is really easy to turn off if you don't want  them in production and it's really easy to polyfill if you need to worry about older  browsers.  And we can talk about that later if you're really interested.  All right.  So now we have a functioning application.  The user can delete all of the statements they want out of their timeline.  I don't know why they'd want to.  Clearly this content is gold.  Context airs.  This is still the most common JavaScript RIC.  Any time that you are using a callback or a promise or you are passing a function into  some asynchronous event, be it an Ajax event, a timeout, an interval, a click handler, whatever  and you reference the word this, this has a high potential of happening.  You can usually discover this through testing.  There's nothing magic about production in this happening.  However, because JavaScript is so asynchronous and dynamic, there are so many edge cases  through any respectively sized application that you probably will not be able to test  for every potential edge case.  And if you haven't thought through every potential order that your callbacks could come back  in, you could run into this problem.  And in fact, many of my customers do.  So text substring is not a function.  Something's gone wrong again.  We got a call from a particular customer saying that they were having a really bad time.  They were getting this error and it showed up in our logging.  We saw it happen a lot.  Over 2,000 times last week this happened.  A.text.substring is not a function.  But it only happened to one dude.  Like one person just had a really bad time of it.  So this isn't a widespread error.  This isn't like something that we just failed to test for in all circumstances.  But there's definitely some edge case, either with our code, their data, or their browser  that is causing this to break down.  So let's see if we can troubleshoot this and understand what's happening.  So let's load up that customer's data.  So when we load soliloquy for them, they still have ads.  So that's good.  Getting a little revenue.  But they don't have anything at all to look at.  They don't have any status updates.  Here I see the error that we've been recording, this a.text.substring is not a function.  If we try and type in a status for them, well, that shows up.  But nothing else does.  So maybe let's dig in and see if we can understand what's happening here.  Let's take a look at this error here in the console.  So with Chrome, we can expand an error that happens in the console just by dropping it  down.  And we can see all kinds of stuff in it.  So we have an anonymous function, a function that wasn't named here in statement model  blowing up.  Looks like we have a bunch of underscore functions happening that I don't really know  what's going on there.  Everything from a jQuery callback is down here at the bottom.  And a jQuery Ajax transport.  So that's kind of interesting.  We see that the root cause of this whole thing is probably an Ajax callback of some kind.  Now if I wanted to go even farther, like what started this Ajax?  What are we requesting here?  In the latest couple of versions of Chrome, they introduced this async button right here,  which is, in my opinion, the coolest thing that they have ever done.  So if we go ahead and turn that on and reload this, now we can take a look at this stack  trace again.  And we see the same stuff is on the top.  It's a text substring is not a function and a bunch of other stuff.  But here, where the stack trace used to end, we see that this was separated by an XMLHP  request send event, an Ajax call.  That Ajax call was started by this file right here, or was last touched by this file here.  It was initiated through a jQuery Ajax transport by Ajax send from backbone, from underscore,  all the way to a jQuery ready.  So there is something happening in jQuery ready that's firing this thing off when we  first start, an Ajax goes off, some period of time later it comes back, blows up, and  we end up with this function.  So if we look at the network, and filter this down to XHR, it's probably one of these two  first requests that are happening right at page load.  So let's take a look at the code itself and see if we can figure out what's happening.  So because we clicked on this, Chrome interpreted our source maps and gave us the actual file.  You'll notice that some of these lines are grayed out here, and that's because there's  no corresponding line in the original source map to this file.  And so we could not put a break point on certain lines, because Chrome doesn't know how to  handle that.  But it renders it so we can see what's going on, and it points as to where this error is  happening.  So this is a parse function, and a parse function that takes a response.  So based on that stack trace, this seems like we are trying to parse our XMLHP request,  or parsing the server response.  And this has some expectations about what this is supposed to look like.  Response is something that we can enumerate across.  We are looping over it.  And for each thing in this list, it's expecting those things to be objects of some kind, and  those objects to have a field called text.  And text, we are calling a substring on.  And so all of those expectations about what our data looks like is baked into this.  So if we go and look at what's actually in this request, we can expand this out, and  the response is indeed something we can enumerate over.  We do have an array, and we do have an array of objects, and those objects do have a text  property, but that text property is not a string.  And so what the server is actually giving us back, and what our JavaScript is expecting,  fell out of sync.  And so this can happen for all kinds of reasons.  In big enterprises, it's really common for the team responsible for building the API,  and the team responsible for building the front-end code, or might be different teams, and might  have different release schedules.  There you have the possibility that somebody changes a type or changes a shape of an object.  If this is a third-party API, perhaps I'm calling into Twitter's API to do something,  Twitter could change its API out from under me.  Twitter probably wouldn't, but there's all kinds of other vendors that could.  Let's take a look at how we could actually deal with this particular issue so that we  could let this customer have a little bit better time.  So here we have our parse function, and all of the expectations in it.  Let's see if we can reduce those expectations or make it safer.  So the first thing I'm doing is underscore map, which should handle non-enumerable responses  just fine.  We can't do anything, and we'd get an empty array.  And so that should be okay.  The next thing is that we expect item to be an object.  And so we can fake that out.  We could say, you know, item is an item, or it's an object.  That's just a little handy piece of syntax that you can force an object to be at least  something, and it won't be undefined.  And then we can do the same thing here where, you know what, first we should really check  to see if text.item is a string.  So I'm going to use another underscore function, but there's all kinds of ways to do it.  I'm just going to use the helper isString, item.text.  If it's not a string, we should do something different.  If it's not a string, like it's a number in this case, we should assign it some value.  We should say item.text is item.text or an empty string.  And this is just a truthy value.  So when text is 42, which is the case we're dealing with, text will still be a truthy  value.  But the truthy values in JavaScript tend to have a two-string method.  And the empty string case allows us to solve for maybe an edge case where we have item.text  is undefined or item.text is not a number or other weird things like that that might  not have two-string implemented on them.  And so with all of those things set, we should have normalized this data down to item.text  should at this point be a string and have the function substring on it.  So if we save this and reload, take off that breakpoint, we can now render all of their  old, maltyped data.  Maybe they don't want the number 42 printed in over and over again.  But the client side is trying to be a little bit more tolerant of changes in their API.  If you wanted to be a little bit more strict, we could also just make a simple change and  say, hey, if this isn't a string, return empty string, and we should probably log something,  console.error.  Got bad data.  That might be the little bit more stricter way to handle this problem.  Where the new message that I put in is still rendered, however, all the old messages are  not, the page continues to function, and we have an error that's a little bit more descriptive  about what's happening, actually getting recorded.  Apparently that causes another error downstream, though.  So I'm going to go back to the way it was.  All right, so now they can see all of their data, they can see their awesome inline ads,  and everything is happy.  So this is a dirty data error.  And it can be caused by a couple of different reasons.  It can either be caused by unvalidated input, which I talk about here, where somehow data  got into the system in a bad form.  Somebody was able to post a number value rather than a string value, and we had all kinds of  downstream problems from it.  In this case, it was a simple error, but this can cause all kinds of issues like cross-site  scripting vulnerabilities, major production faults, all kinds of things like that.  This can also be caused just by simple lack of communication between teams.  Like I mentioned earlier, the server-side people not talking to the client-side people  when they change an API and a contract gets malformed.  This can be really difficult to test for.  I have struggled with validation problems on every single client-side app I've ever worked  on, and it's always a really tricky problem to solve.  Window.inline ads is not a function.  So let's dig into this error.  Now, this error has a little bit different profile for us.  More normal traffic, because Soliloquy is amazing, is 20,000, 30,000 hits a day.  We don't have a lot of these errors.  This isn't a super big deal.  But 134 people, or 129 people encountered this error 134 times.  It's not insignificant.  This is a real problem that is affecting some small segment of our user population.  Let's try and understand how inline ads could be an error.  I'm going to go back to our other account.  Cool.  So, when I load our default account here, I'm not seeing that error.  I see inline ads, so I'm not really sure how this could be a problem.  So if we search our code for references to inline ads, the only place it shows up is  in our main bootstrapper code, right here.  So we are setting up our application on the original state, and we're calling a function  called window.inline ads.  So why wouldn't that ever be a function?  So window.inline ads is the API given to us by one of our advertising providers.  The advertising provider that I load right here.  So what would happen if that failed to load?  If I just, you know, fake this out and comment it out, well, there's our error.  A.inline ads is not a function.  I mean, that appears to be what's happening, but look at the UI when this happens.  That's a terrible user experience, right?  Not only did I not load any inline ads, I didn't load any sidebar ads, I didn't load  any of their content, I just gave them a big empty screen, and they think all their data  is gone.  It would be terrible if a user ran into this problem.  So, so inline ads appears to be failing some of the time, and that is really, really, really  likely.  So, based on our metrics, about one request out of every about five to ten thousand somewhere  in that range, depending on the customer, has some asset loading problem.  One JavaScript file from your page will fail to load.  Not all of them.  Just one.  What would happen if one file failed to load?  So, I had a project where we loaded Stripe.  Stripe is a credit card payment processor, and Stripe works on the page by basically capturing  the form input and stealing the credit card, not stealing, taking the credit card data,  and forwarding it to the Stripe servers and getting a token back and then returning that.  And so, we had a form set up that, you know, took the credit card data and then would just,  you know, send on the token when it was all done.  But some percentage of the time, for a lot of our customers in Australia for some reason,  some of the time Stripe.js would just fail to load.  It could have been due to a bad network connection.  Maybe Stripe CDN was having a problem.  Maybe they had some sort of corporate firewall that was blocking it.  There could be all kinds of reasons.  But when Stripe failed to load, the behavior of that page was to take the credit card details  and post them in clear text to my server.  Because I wasn't expecting that to ever go wrong.  I thought Stripe had that handled.  But in those rare occasions where Stripe.js fails to load, I was making a major security  vulnerability.  Think about your applications.  What would happen if one of your scripts fails to load?  It's probably not a big deal if you bundle everything together, or it's not a big deal  if there's a catastrophic failure and nothing loads.  If you're building a single page app, the user probably gets a big white screen.  Or hopefully you're doing progressive enhancement and everything works great out of the box.  But what if a major piece of your content fails to load?  What if your transaction processor fails to load?  So that's what Soliloquy failed to do here.  Is window.inlineads was in my core pipeline.  I called this inlineads function and then I called the sidebar add function and then  higher up this file I called my main start the application function.  So when inlineads failed, everything broke.  We just didn't proceed any farther.  So what I should really do here is be a little bit safer.  Inlineads isn't a critical piece of my infrastructure.  If it fails, it's not the end of the world for me.  So what we should really just do is just check to see if window.inlineads is a thing.  Or rather not, let's check if it's not a thing.  And if it's not a thing, let's write a warning message.  This way, window.inlineads will only start itself up if it has successfully loaded.  And on times when it doesn't load, I will capture some logging statements back to my  own systems to give me some indication of what percentage of the time is this happening.  So I can go back and complain to them or figure out what exactly is going wrong.  So now with this in place and we load up the page, you know, the user, they don't have  inlineads, but they at least see their content.  And that's what's important here.  So loading errors.  This happens way more frequently than I ever thought was likely.  But all kinds of times in poor network connections, corporate proxies, long distance things,  edge networks, American mobile networks on Norwegian cellular networks, you get sporadic  times packets drop.  And sometimes those drops will happen when you're loading a critical piece of your web  application.  All right.  This is the last thing I'm going to talk about today.  But it's a doozy.  This is when the whole browser crashes out from underneath you.  And unfortunately, I don't have a good automated way to capture this because the entire browser  is crashing.  When you're monitoring a runtime and that runtime crashes, you really have no way to  report that.  But luckily, if you crash a browser enough time, people complain, usually on Twitter,  complaining about Soliloquy on Twitter.  And so we heard a few people complaining that, you know, they would use Soliloquy all day.  And by the end of the day, it was just so slow, they could barely type into it.  And then Chrome would crash on them.  And they'd have to restart and they would, you know, lose all of their tabs.  And so that sounds like a memory leak, getting slower and slower and slower all day.  So let's see if we can find a memory leak in Soliloquy.  And there are some really fun tools to find memory leaks in Chrome today.  So I'm going to pop over into the timeline tab.  And the timeline tab lets you record what's going on in your application or what is the  state of the memory of your web application while you're taking certain actions.  And so what I'm going to do is I'm going to record a bunch of actions and then I'm  going to write a few statements into Soliloquy and delete those statements back out, supposedly  putting us back into the exact same state that we were when we started.  And then just see how the different parts of the DOM, how the different parts of memory  have changed while I'm doing this.  So I'm going to restart this one, make sure it's clean.  And then we're going to record.  And I'm going to say, hey, NDC, some other statement.  Let's go drinking.  And then we'll delete those and stop.  And so now what we can do is we can start digging into this timeline here and understand  what's going on.  And there's a lot of data here.  It's kind of hard to interpret right off the bat.  But where I like to start is down here in this graph here.  What this is showing us is how much JavaScript memory has been used, how many nodes are in  the DOM, and how many event listeners have been attached.  And these are all very important things because they impact memory of the browser, how much  memory is being used.  And so what we can see here at the macro view is that there is a step function going up  where periodically a whole bunch of JavaScript is allocated, several nodes are created, several  listeners are attached, and it just keeps happening over and over and over again.  And it never gets released.  So that's a pretty strong indication we have a leak here because the application itself,  I believe, should have reset to its normal state.  And so we should have dropped to some normal baseline again.  So we can understand more about this by starting to zoom in.  And this is beautiful.  They used classic first person shooter gamer controls.  So you get onto your keyboard and with WASD on your keyboard, you just start, I need to  show something.  Sorry, I need to rearrange this a little bit.  Oh, all right, thank you.  All right, so now with WASD style controls, we can just start zooming in and navigating  this timeline all the way down to microscopic levels of detail about what's actually happening  here.  And this is so fun.  Like I have literally lost hours just playing with applications that don't even have memory  leaks.  I just want to see what they do.  And so you can go in and see what's going on.  And so let me back up a little bit.  Here's one of those steps that we saw at the macro level.  And so as we start zooming in, we see that it's not one step.  It's a whole lot of little steps happening in rapid succession.  And so we keep zooming in on this and we can see what's actually going on.  We see that a timer is going off.  So we see timer seven, timer eight, timer nine, like a whole series of timers have gone  off.  In response to those timers, we're calling this function here in sidebar ad provider.  Sidebar ad provider is calling this anonymous function, calling this thing called placing  an ad and doing a bunch of stuff.  And when this all happens, we're stepping up our memory usage.  So if we go in here and look at sidebar ad provider, which we can just navigate to it  right from here, we can see exactly what's going on.  So we have, let me zoom back in maybe.  So we have a set interval.  This is that interval we saw going off at the top of our timeline function.  We have an interval going off a set number of times.  And every time that interval goes off, we have this ad container, which looks like it's  just a dom node.  And we're clearing out the contents of this dom node.  And then we're getting a new advertisement and placing that advertisement.  So let's look at what place ad is doing.  Place ad is making an Ajax call.  It's going out and getting the contents for an ad.  And it's pulling it down.  And once we have the contents of it, it's sticking it, it's sticking that text inside  of the element and attaching a click handler to it.  And so it's just doing this over and over again.  This is the sidebar ads right here.  If we wait a few seconds, you can see them refresh.  And every time those refresh, it's just taking all of the nodes that were there and just  putting them aside, still with click handlers on them, and putting a new set of ads.  And so all of the ads that we've ever loaded on this page are still in this page.  They're just not present in the dom at this point.  So another way to think about that is over the course of a day, there are literally thousands  of advertisements for David Hasselhoff on this page waiting for you to click on them.  That's terrifying.  So leaky memory.  Memory leaks are most often caused when you attach events to parts of the dom that don't  always exist there.  If you attach an event to a piece of dom that gets replaced at some point later, that event  is a pointer to that whole part of the dom that will be kept around until you release  that event.  There are tons of other ways memory leaks can happen.  This, however, is the most common.  So there's no good way to monitor for this in general terms that we've found so far.  However, periodically checking any sort of apps for kidnapping of your memory, any of  your apps that you're doing major development on, I encourage you to tell your boss that  you get to play a little game for a little while and just play around the timeline for  a couple hours.  And you'll be able to find pretty quickly if there are any sort of leaks happening.  So this was JavaScript forensics and the six most common kinds of errors that I see  affecting production web applications.  Script errors are obfuscating the real problems that are happening in your app.  Third-party errors, whenever you introduce a script from a third party, you're introducing  risk that they can break your app for reasons beyond your control.  Context errors, losing what the value of this is as you pass it around into callback functions.  Data errors, changing the underlying data contracts, either communications between different parts  of your organization or in relying on third parties.  Loading errors, way more common than I ever thought.  Part of your application failing to load in certain circumstances.  Think about what would happen when only part of your asset fails.  And finally, memory errors, using two attaching events to detachable DOM nodes and causing  the browser memory to balloon and balloon up until it eventually crashes.  I encourage you to all look at your app and monitor it.  Watch out for these errors.  Watch out for these villains and monitor for them.  And if you need a new sheriff to help you monitor them, TrackJS can help.  So this was JavaScript forensics.  If you have questions for me about JavaScript or monitoring or TrackJS, you can get me on  Twitter or there's my email.  I've also, if you fancy a sticker, I've left stickers on both of the tables here.  All I have to do is push a green button and you can take a sticker.  Other than that, thank you very much and enjoy the rest of NDC.
JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_0_2963.png JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_2964_5495.png JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_5496_6863.png JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_6864_7571.png JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_7572_10619.png JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_10620_15611.png JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_15612_17183.png JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_17184_18035.png JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_18036_18131.png JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_18132_18395.png JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_18396_20543.png JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_20544_21011.png JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_21012_21491.png JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_21492_21539.png JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_21540_21695.png JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_21696_21779.png JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_21780_22847.png JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_22848_25187.png JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_25188_25787.png JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_25788_25871.png JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_25872_26639.png JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_26640_28163.png JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_28164_28223.png JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_28224_29027.png JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_29028_29639.png JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_29640_29747.png JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_29748_30347.png JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_30348_30707.png JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_30708_35651.png JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_35652_35771.png JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_35772_36887.png JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_36888_39887.png JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_39888_43919.png JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_43920_44147.png JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_44148_44231.png JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_44232_44627.png JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_44628_45947.png JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_45948_46091.png JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_46092_46823.png JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_46824_47099.png JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_47100_47291.png JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_47292_48203.png JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_48204_48803.png JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_48804_48983.png JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_48984_49283.png JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_49284_50699.png JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_50700_51899.png JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_51900_51995.png JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_51996_52043.png JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_52044_52223.png JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_52224_52403.png JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_52404_52643.png JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_52644_53159.png JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_53160_54419.png JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_54420_57755.png JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_57756_57899.png JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_57900_59567.png JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_59568_59675.png JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_59676_59855.png JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_59856_60899.png JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_60900_62399.png JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_62400_64079.png JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_64080_71075.png JavaScript_Forensics_-_Todd_H_Gardner_ORIGINAL_DOWNLOAD_71076_75318.png