Deterministic Solutions to Intermittent Failures
CC Attribution - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Mertens, Tim
Monday mornings are bad when the coffee runs out, but they’re even worse when all of your impossible to reproduce, “flaky” tests decide to detonate at the same time. Join us to learn a systematic, proven workflow for expertly debugging any test failure including consistent, order-related, and non-deterministic failures. We’ll discuss the importance of fixing intermittent failures before they reach critical mass and why you should eradicate the term “flaky tests” from your vocabulary. Don’t wait for an emergency to prioritize fixing the ticking time bomb of intermittent test failure.
 I'm really excited to be here talking to you about this at RubyConf this year. I'd just like to say thank you to all the organizers who have put in a lot of work to put this together for us so we could be here today. So again, my name is Tim Mertens. I am a software architect at a fintech company based out of Chicago called Avant. Prior to my current role there, prior to my current role there, I led our test engineering team for about three years and I've been working in the software industry for about ten. You can find me on GitHub and on Twitter, sort of, I'm not a big Twitter user, but you can hit me up on there. And I'm going to be posting some links later on my Twitter account, some resources for you to look at. So I'm going to be talking today about deterministic solutions to intermittent test failures. I think we've, well, maybe not all of us, but a lot of us have experienced this problem where you have a really large test suite and eventually things start to fail irregularly and it can be a really big pain in the neck. So just to reiterate that fact, my wife made fun of me for making this slide. She says I'm a little bit of a fanboy for Elon Musk, but, you know, Elon, he's sitting in his chair and he gets his newspaper and he's reading it and it's an article about how there was another gravitational wave detected and he says, you see, it's happened again. And his reasonable friend comes over to him and says, Elon, this is base reality. It's not a simulation. And meanwhile, somewhere else in the universe, there is this old guy sitting at a computer and he says, Jesus, the build failed again. Can you rerun it? And in the other room, this younger guy says, sorry, I'm kind of busy. He's playing games on his phone. And the old guy says to him, don't make me send you back down there. And he responds, yes, father. All right, so this is something we all deal with. I'm going to be using some code samples in my presentation today. These samples are all based on RSpec. You can take the workflow, though, that I'm going to share with you and this can be applied to any testing framework. But my primary experience is in RSpec and it provides a lot of tooling that helps us to do what I'm going to show you today. There are also some source code examples that I put together that are available. I'll have this link again at the end of my slide and I'll put it on Twitter. But this has examples of some of the test failures that I'm going to talk about today. And the build in this repository is purposely failing so that you can go familiarize yourself with these things and solve the failures that are happening in that repository. So this is a common thing that happens. You feel like the build is passing consistently for a while, depending on the size of your repository and how many developers you have. It might be a while for you. It could be a day. It could be three days. It could be a week. And inevitably, somebody says, ah, this is awesome. The build has been passing for so long. This is great. And then what happens, the next build always, always fails. And people say, oh, well, that's just a flaky test. And before we get into some of the workflow part of this, I want to talk about the myth of flaky tests. And I want to deconstruct this for us. Because there's this common notion that there are things in our test suite that are unsolvable. And I've been doing this for long enough that I just don't buy it. So I want to remind everybody. Your tests are software too. If you have an API call that is critical to your business, and it's failing every once in a while, and the CEO or a business person comes to you and says, hey, why is this failing? Do you say to them, oh, well, it's just a flaky endpoint. And there's nothing we can do about it. No, you don't do that. It's software. You make it do what you need it to do. Test code does exactly what you tell it to do. It won't do more than you tell it to do, and it won't do anything less. It's going to do exactly what you say. When we use the term flaky, we're implying that there is some kind of unsolvable problem. We have this failure that we just can't deal with. There's nothing we can do about it. I like to take that term and replace it instead with the term non-deterministic. Because then what we're saying is we have behavior that's happening in our system that can be accounted for. Once we understand the root cause of a failure, we can modify our test, or maybe it's a production bug and we can fix the bug, but we can account for that non-determinism so that our tests themselves are deterministic. I also want to remind everybody that when you're ignoring failures, they could be real defects. You could have legitimate defects in your production code that people are simply ignoring because it's difficult to troubleshoot. So I want to talk a little bit about how I got here and how we came up with this. It's a story that is fraught with peril. There were a lot of failing tests that I've had to deal with in the past. This is where what I'm going to tell you about today came from. For those of you who have not been in the development community for a long time, I'm going to be talking a lot about continuous integration. This is where you push your code up to, let's say, GitHub and some service is going to automatically run your tests for you. Some examples of that are Travis or Code Ship or Circle CI or Jenkins. This is what we're talking about when we say continuous integration. So back in the day when Avant was small, we had a small number of tests and code that we could load up and put into our CI tool and it would take our build and it would run it and hopefully things would go smoothly and everybody was happy. But over time, our code base grew and our test suite grew and so our test suite was taking a lot longer. So we said, all right, we need to paralyze this and manage our build times. And so when I say paralyze builds, that means we're taking, let's say, 1,000 tests, 10,000 tests, however many tests you have in your test suite and you're splitting them across a bunch of different containers or workers or whatever you want to call them. And so we did this and we split our build across a bunch of containers and every once in a while something would explode. But we were focused on building features and when you're focused on building features, you can lose sight of the fact that this is a bug that's existing in your test suite. And people like to say, oh, well, that's just a flaky test and master. Run your test suite or run the test locally and if it passes, go ahead and merge your code or rerun master and then deploy as long as it passes. And so these things tend to just get left behind. And so as time went on, we kept building more and more things. We had much more complex dependencies. And so we started using more complex things in our test suite like factories to help us build out these large graphs of dependencies for some complicated business logic that was getting tested. And eventually, people just don't really know everything that is going into what's happening when they run a test. And so we were still running a lot of builds and running a lot of tests and we had a lot more code and things were still blowing up sometimes and people would complain about it, but they wouldn't always do something about it unless it was a critical failure where it was failing all the time. Now eventually, we got to the point where we realized, you know what? Our provider that we're using isn't really cutting anymore. We need more capacity. And so we decided to search for a new CI provider. And so we were looking for a CI provider that could give us these big machines where we could run a lot of tests on them and it would shorten our build times for us. And so we did that. And we tried it. We put our test suite on this new CI provider that would run things more efficiently and all of a sudden, everything is exploding all the time. And what we found was that our old CI provider was breaking down our tests into these very small groups and there were some things that they were doing that just made it so there were some things that we just never even saw because certain tests would never run together and there's a couple of other different things they were doing. But in any case, when we put everything together and we randomized our test suite, we had a really big problem. A colleague of mine and I spent the next eight to 10 months pretty much just fixing tests and then we were finally able to switch to a new provider. So today what I'm going to share with you is a workflow for troubleshooting these types of failures that came out of this process. I fixed thousands of tests. And so when something fails, I usually already know what probably is happening. And so I'm going to give you this map to fix your test failures so that you can go back and you can do it yourself. So all right, master fails. A test fails and master, woohoo, we're all really excited, right? This is the best day ever because whenever master fails, you have something to do. So what do we do, though, really, to debug this? First thing we want to do is we want to run that test by itself. Does it fail? If it fails, then okay, does it fail every time? This is the best possible solution. We have a reproducible failure. We have something that fails every time and is going to be really easy to fix because I can just run it on my local machine and debug it until I can figure out what's going on. This is probably one of the number one challenges in solving test failures when you have an intermittent test failure is you can't reproduce it sometimes. So how do we debug reproducible failures? I'm just going to touch on a couple of common failures that occur, but I'm not going to linger on this because this is something that is generally relatively easy to fix. So a couple of things. Stale branches. You might have branches in, I'm sorry, you might have someone that is working on a branch and they don't back merge for a week and then they come back from vacation and they're like, oh, I got signed off on my PR, boom, merge, and all of a sudden, master fails. But what happens is in that last week, maybe a logical change was made that's incompatible with their branch and now the tests are failing. Business dates and times. You might have logic that depends on the day and time and sometimes you can get to a weekend or a holiday or maybe the next year and all of a sudden your test suite is failing. Mock time and system time. If you're using a gem like TimeCop or TimeOrp or others to modify the Ruby system time, it's important to remember that if you are relying on something, if the code that you're testing is relying on some external process, which is also doing something with date and time, all of the external processes from your Ruby process are not going to share the mocked Ruby time. Missing preconditions. This is another common one. If you're distributing tests across a bunch of different containers and you have a test that or the test code or the production code is not requiring something that it needs or maybe it's not doing the data setup that it needs and it's been relying on some other test to do that for it, you can run into situations where when that test is run on some node in your CI environment, it fails because the other test is no longer running before it. Then finally, of course, real bugs. Make sure when you're fixing a test you're not just putting a band-aid over it where you don't understand what the problem is and so you just put a band-aid on it to fix it. Make sure you understand the root cause and that you're not just putting a band-aid over legitimate defects. All right. So let's go back. Let's say we ran the test by itself and it doesn't fail every time. It doesn't fail. Or maybe it doesn't fail every time. The next thing we want to do is we want to take the group of tests that ran on the CI container that failed and we want to take the seed that was used to randomize the test and run it again. So what is a test group? A test group is a subset of tests from our test suite which run on a specific node in your paralyzed build. So here are some examples. If you're just using the RSpec command directly, this means you don't have a test group. You're just running all of the tests. Maybe you have a list of files that are being piped into the RSpec command. This is your test group. Or maybe you're using a tag or some other command line option of RSpec to specify your test group. Test seed is a value, it's an integer in RSpec that you can pass into RSpec and what it'll do is it'll reproduce exactly the order that the tests were run in in your CI environment. So here we have output from RSpec. The output's truncated. And so at the end of the RSpec output when you run your tests, you'll see this line where it says randomized with seed. This is your test seed. So you're going to take that. And what we're going to do is we're going to take our group of files and we're going to take our test seed and we're going to give that to the RSpec command. And so you can see here we have our test groups, a couple of different examples. We're using the seed option to pass in the test seed. And I also like to use the fail-fast flag because when you're running a large group of tests that take a long time to run and something fails in the middle, it's really nice to have RSpec just automatically stop when it hits that failure. Okay. So we rerun our test with our group and our seed. What do we do next? All right. So if it fails, the next thing we want to do is we want to bisect that failure. In RSpec 3.3, I believe, they added a feature that will automatically do this for you. So when we talk about bisecting tests, what we're saying is we're going to take the full group of tests that ran with this failing test and we're going to cut them in half and cut them in half and in half again until we get to the minimal command required to reproduce the failure. So ideally you end up with one test that's passing and then your failing test. And when those two tests run together in a particular order, you'll see that other test fail. So here we have, this is essentially the same command as we ran before. The only difference here is that instead of the fail-fast flag, we're replacing it with the bisect flag. And this is going to take a long time if your test suite, if the group of tests that you're running, take a long time to run. But it will reduce that set to a minimal command. All right. So now we've bisected our test failure. If our bisect succeeds, then we know that we now have a test failure that is of the classification test pollution. What is test pollution? Test pollution is when you have one or more tests where the side effects of those tests are causing another test to fail. And so how do we debug that? There are a set of very common failures that we see, which usually these can be attributed to. And so I'm going to talk through some of those. So data pollution is a really common type of failure that can occur in a test suite. And this is when you have one test that is, let's say, creating a record outside of the usual test transaction. It's persisting something to the database, or maybe it's persisting something to Redis or some other store, and it's not resetting it afterwards. So I'm a big proponent of isolation testing and making sure that your tests are cleaning up after themselves. But what we've found and what we practice now at Avant is we practice defensive testing. So the thing, what that means is it means that your tests shouldn't expect pristine conditions. So they should clean up after themselves as much as possible, but they should not expect a clean slate. And let me give you some examples. So these are things that put really hard constraints on your whole test suite because of a single test. So you might have a test like this where it performs some action that has a side effect that creates a user. And after that action is performed, it sets an expectation for the user count to be one. What does this mean? This means now that no other tests in our test suite, none of our test support code, none of our fixtures that we have can create a user without breaking this test. That's a really hard constraint to put on an entire test suite for the sake of one test. So what we can do is we can rewrite this test instead to expect change so that we're not putting that global constraint on our test, and our test will still pass even in the presence of other records. Similarly, don't expect global scopes to only return the records you created in your test. Here we have an example where we're calling an active scope on a user model, and we're saying expect the results of that to match this array. And what we do is when we say match array, we say, all right, expect the results of the scope to have exactly these users in it. But what if another active user was created by another test or by a fixture? This will fail. What we prefer to do instead, and the way we can solve this, is we can instead expect inclusion and exclusion. So this allows us to set the assertions that we want to, sorry, it allows us to assert the things that we want to assert in our test without having this reliance on a super clean database when it runs. Another common failure that we see is caching. Caching is really hard. It's a very hard problem in computer science. And so one of the things here that's important to do is that if you have a test that's mutating global state or a global singleton in a breaking way, oh, it is going to break other tests, ideally you want to reset that cache after that test mutates it. Better yet, if you can, use stubs or mocks to avoid modifying the cache at all. Now if you're testing a singleton, what this means is if you are implementing singleton behavior using class methods explicitly and you don't have a singleton instance, then it's going to be really hard not to mutate your global cache. So whenever we're implementing singleton behavior, what we do instead is we implement it using a singleton instance. And then you can see here, the top example here is testing a singleton class directly. So we're adding a key to our singleton class, and then we're checking that the set of keys that it's storing contains that value. Now what I prefer to do when I'm testing a singleton so that I don't mutate the global state is create a separate instance of it for the purpose of testing the singleton's behavior and perform my test on that instance. That way I'm not mutating global state. You can still test that your class level methods are properly delegating to the singleton instance, but you don't want to mutate it directly. Mutated constants. Sometimes you see this, someone overwrites a constant in their test because maybe there's a timeout value that they want to reduce so that their test runs faster, or maybe there's a feature flag that depends on a constant that has a data sign to it that turns that feature on or off. You don't want to modify this constant directly because that value could change in the future. Even if you reset it back, maybe that value gets updated in the future and it's no longer the same thing, you're going to be resetting it back to the wrong value. So instead, learn the tools that your testing framework gives you. OurSpecMox has a very comprehensive mocking framework. So you can use things like stub const to replace a constant value with something else, or you can allow a class method to be received and return a stubbed value. These are things you can do to avoid constant mutation, and what it's going to do is at the end of your test, when it gets torn down, it's going to replace it with the original value. Don't rewrite class, don't rewrite production class implementations. This is something that I've seen before. I don't know what else to say. It's not a good practice. It's much better to just create an instance of the thing that you want to modify and then put a stub on it. You don't want to modify the implementation of your classes because what happens is now you've modified the implementation of that class, and that's going to carry through to any test that run after this one. Don't mutate, sorry. Test constants is another interesting one. Sometimes people do this. They think, oh, I need this variable in my test so that I can assert on it in a bunch of different places, and so they'll define a constant in their test and assign a value to it, and then they'll use it in their assertion. And what happens is three months later, somebody comes along and they copy the production class because it does something close to what they want to do. They modify it a little bit. They copy the original test. They modify it a little bit to reflect the new behavior, but they don't update this constant name. And what happens is now you have two constants that are defined in your test, and people think that, oh, well, this is a scope to my describe block or a scope to my context block, but your describe block is not a namespace. It's not a module in Ruby. And so what happens when you declare a constant here, it is going to go up to the global namespace, and so when these two tests run together, whichever one gets loaded second is going to win, and so the first test that runs is going to fail. And so it's preferable to use let variables or use local variables in your tests so that you're not mutating global state. And of course, always look out for real bugs, especially as you start getting into intermittent failures, things can get a lot more challenging, and a lot of times it can come down to a production issue. All right, so let's go back. We re-ran our test with our group in seed. Let's say it doesn't fail, or maybe we go and we try to bisect it, and the bisect is unable to come to a minimal command to reproduce or can't reproduce it at all. What do we do next? So the next thing we do, and this may sound really stupid and easy, because it is stupid and easy, but run your test in a loop. This is an example. Take your test inside of your outer level describe block, put 100 times do block, and then run that test by itself. A lot of intermittent failures can be reproduced this way. When I do this, I like to use the fail fast flag again because it'll stop execution immediately when a failure occurs, and I can start taking a look at it. All right, so we run our test in a loop. Does it fail sometimes? If it does, we know we have a non-deterministic failure. What that means is we have a test that is, either the test itself is doing something that's non-deterministic and making an assertion on that behavior as if it is deterministic, or you have production code that is non-deterministic, and the test is asserting that it is. So non-deterministic failures are failures that occur seemingly randomly in our test suite for no apparent reason. And so I'm going to give you a couple of common types of failures that we see that fall into this bucket. One of the most common failures that I've seen that this happens with is unordered queries. People assume that when they make a database query, it's going to give them back the results in the order that they created the records. But this is simply not true depending on what type of database you're using. If you're using Postgres, which a lot of Rails users do, if you look at the Postgres documentation, it explicitly states that if you do not specify the ordering of a query, it is going to return the results in the order that they are found. And usually that's going to be the order they're created in a test, but not always. And so what we want to do here, there's an example on the top. Let's say we have a scope that's unordered that we're testing the results of. If we say expect the results to equal some set of records, what our spec is going to do is going to say, okay, oh, your results, they have the right records in them. But what it's also checking for when we check for a quality on an array is it's checking that the order of the elements in that array are exactly the same. So when we're testing unordered queries, what we want to do instead is we want to check that the result set contains exactly the records that we expect or that it matches an array. But what these two matchers do is they perform that expectation without a dependency on the order that the results are in. Frozen time is a similar issue that's pretty interesting. So if you, again, use TimeCop or some other gem and you're freezing time, what happens is all of the records that are created are going to have the same created at. And so if you perform a query that is ordered, and it's ordered by the created at time, the sort order is going to be non-deterministic because the creation time for all of them is going to be exactly the same. So we prefer TimeCop travel over TimeCop freeze. When you say TimeCop.travel, what it does is it travels to the time you give it, and then time continues to roll forward. So if you create a set of records when you're traveling through time, they're going to have different created at timestamps. But if you're freezing time, they're not. Only freeze time when precise time is needed. This is almost never in my experience. All right. Another common case that we see is randomized test data. If you're using the Faker gem or maybe you have a set of test data that you are sampling for the purpose of randomizing your test data, you can end up with unexpected failures. Maybe it's returning names in a format that your code doesn't handle properly. Or maybe the test code is asserting the results incorrectly. Maybe there's phone numbers that don't match validations in your production code or zip codes that you don't support. Or maybe you're sampling from all of the United States, but you only support the continental United States. Sometimes you're going to have a failure. One of the things you can do to help with this is if you are writing tests that rely on faked data like this, is to output that data with your test errors. So when the test fails, you have more context that you can use to figure out what's going wrong. All right. So we go back. And what if we ran our test in a loop? Because that test still doesn't fail. This is what we call unreproducible failures. And I put quotation marks around it because you can't reproduce it, but it's breaking CI. So obviously it's reproducible somewhere. These are the most difficult ones sometimes to figure out. And so we'll talk about a few common cases here. So data in time is a really common issue. Data in time is really hard to deal with, especially when you're dealing with time zones. So some of the things you want to look for in your CI builds that are failing is do your tests only fail on weekends or holidays? Do they fail at certain times of day? Maybe you have some code that relies on business time or business hours. Time cop your test to the data in time when CI ran and see if it fails. We have a time cop RSpec gem that we open sourced. I would say go check it out. It makes it really easy to work with time cop in RSpec using RSpec metadata. And we also have a cool little feature where you can set an NVAR and it'll time cop all of your tests. And we use that for running time travel builds so we can catch failures in advance. So another common failure here is UTC and local date time differences. A lot of people don't know this. Date.today uses your system time zone. Date.current, which is part of active support, uses your application time zone. And so what happens? Here's an example. This is available in that repository that I shared earlier. On the top line, this is mimicking a system like a continuous integration system where the system time zone is set to UTC and our application time zone is set to central time. And what you'll see is if you run this test, this test is going to fail because it's asserting date.current is equal to date.today. And what happens is when you are for certain times of day when the UTC date and the local date don't match, your test is going to fail. And a lot of times this can be a production bug. SQL date comparisons. This is another really interesting one. And this is pretty much always a production bug. So it's really important to know that when you are doing SQL date comparisons, let's say you're looking for records after a certain date or before a certain date, when you pass a SQL string to active record. What it does is it takes your value that you give it. And because it's injecting it into a string, it doesn't know what column you're comparing against. And so it can't infer the type of that column. And so what it does is it's going to coerce the value that you give it to the default database string that it would normally generate for it. So if this is a value that has time, like a date time object or a time object or a time with zone, what it does is it coerces it to a date with time string without zone in UTC. And so if you are passing a time object into your query and you're querying against a date column where the date columns are stored in as local dates, there are going to be certain times of day where the UTC date does not match your local date. And Postgres or whatever other database you're using is not going to know to coerce it to the current local date. Timeouts in asynchronous JavaScript. This is another really common source of errors. I know a lot of people hate doing acceptance or integration testing, whatever you want to call it, browser testing. So when you have a failure in a browser test, it's important to keep in mind that your performance in your continuous integration environment is usually worse than your local machine. So you might never reproduce this failure locally, and if it happens, if you have a timeout occasionally in CI and you can never reproduce it locally, oftentimes you'll find it's because of differences in performance. Also performance can vary widely depending on what tests run before a particular test or how your application is set up. Maybe a test runs before it sometimes that renders some of the views that a page needs in order to finish rendering. So sometimes it'll take less time to render. When you run into these types of situations, you want to increase your timeouts for CI as needed. You don't want to have a fit. It's most important for you not to have failures in CI. You don't want to use your browser test for performance testing. If you're setting low timeout values for your browser test because you say, oh, well, this page needs to load in three seconds, you should not do that in CI. You should always be testing performance against production or production-like environment. CI is not a production-like environment. So also if you have JavaScript that's executing, sometimes you will see failures that happen because your test tries to interact with elements on the page before the page is fully loaded. So I added a feature to the Site Prism Gem. If you're not familiar with it, the Site Prism Gem is an implementation of the page object pattern, and if you're not familiar with that, I would just recommend googling it so that you can learn more about it because I don't have time to talk about it right now. But what the load validation functionality does is it allows you to specify a set of preconditions that must be fulfilled before the block that you're executing in your test is actually executed. So you can say, wait for my spinner to disappear from the page or wait for this form to appear. And then when those preconditions are met, it will continue to test. And if those preconditions aren't met, it will give you back a reasonable error. So if you still can't reproduce it, what do you do then? You need to start digging. You need to start looking for environmental differences. Compare your CI configuration to your local environment. Are there environment variables that are set in one place but not in the other? Are there differences in how your tests are being executed? Maybe you're executing the RSpec or MiniTest or whatever other command differently than you do it on your local machine. Try what you're doing in CI and see if it fails for you locally. Maybe there's an issue with how the database is being set up in CI. There might be seeds that are missing or present that you don't have on your local machine. Or maybe someone committed a migration but in CI, a lot of times you might load from a schema or a structure file because it's faster than running migrations. And so if someone forgets to commit a migration to the schema or structure file, you might have some failures. Some other environmental differences that you might run into is library versions or inconsistencies in how specific libraries behave. There's a lot of differences between OS X and Linux commands where they don't always behave the same. If you can, use Docker for consistency. If you're using Docker in CI and you're using Docker on your local machine, you know that you are running your test in the same virtual environment. And so that can ensure consistency. If you still don't know what's happening, some other things you can do is SSH into your CI environment. If you can't do this, try to find a CI environment that allows you to do it because sometimes this is the only way to continue. I did this, I actually was doing this on Friday. Use common sense. Think about the failure that's happening. Look at the stack traces you're getting. Can you come up with a hypothesis about what is possibly causing it? There was an interesting talk earlier today about using the scientific method when you're debugging. Go listen to that talk and think about how you can apply that to your test suite. If you think that there's a third-party gem that might be causing this problem, go look at the repository. See if other people have had this issue. Maybe there have been bug fixes that you don't have in your application yet. And all you need to do is upgrade your gem. And if it's not there, maybe you should file a ticket and contribute back to the open source community if you do find a defect. Learn to use pry and buy bug. These are really, really useful tools that can assist you in debugging your software. And finally, incrementally narrow the scope of the defect. If you're just throwing darts against a wall and you're not tracking what you're doing and doing it systematically, there is a lower likelihood of finding the root cause of the defect. You want to be continually narrowing the scope so that you're looking at finer and finer grained possibilities of what could be causing the issue. Rule things out. Make sure you know your test support code in and out. This is important. Because we have a test suite that has over 30,000 tests in it, and so we have a lot of test support code. And sometimes there can be conflicts in what the test support code is doing that sometimes can cause issues with our tests. So make sure you know what your tests are doing. Look at failure trends over time. If this test is failing for a long period of time, maybe you can find, maybe there's other tests that are failing in a similar way, and you can find correlations between them. Or you can look at the builds that have failed with this failure and look for correlations between them. You can add logging. If the test is still failing, it doesn't hurt to add some logging to that test to give you more information about it so that then the next time it fails, you now have more context. Thank you all for coming. I'm really happy that I was able to talk to you today. Just some quick takeaways. So keep your builds green to avoid sadness. It's really disheartening when you come in on Monday and the build is failing, and it's not just one thing. It's 10 things that are failing, and you don't even know where to start. Remember that tests are code two. They do exactly what you tell them to do and nothing more. Set realistic goals for yourselves. If you have a test suite currently that suffers from this problem, you have to keep in mind that it's not necessarily going to change overnight. It takes time to root these problems out and solve them. And finally, celebrate success. Set milestones for yourself and celebrate them. This is a picture of cupcakes that our software engineering manager got us when we hit a 95% pass rate on our build for a week straight. That was a really big accomplishment. So again, please get in touch with me. I'm going to be posting a couple things on Twitter for you, and I also have a larger version of the workflow that was in my slide deck that I'm going to share. Thank you for your time.
 I'm really excited to be here talking to you about this at RubyConf this year.  I'd just like to say thank you to all the organizers who have put in a lot of work to  put this together for us so we could be here today.  So again, my name is Tim Mertens.  I am a software architect at a fintech company based out of Chicago called Avant.  Prior to my current role there, prior to my current role there, I led our test engineering  team for about three years and I've been working in the software industry for about  ten.  You can find me on GitHub and on Twitter, sort of, I'm not a big Twitter user, but you  can hit me up on there.  And I'm going to be posting some links later on my Twitter account, some resources for  you to look at.  So I'm going to be talking today about deterministic solutions to intermittent test failures.  I think we've, well, maybe not all of us, but a lot of us have experienced this problem  where you have a really large test suite and eventually things start to fail irregularly  and it can be a really big pain in the neck.  So just to reiterate that fact, my wife made fun of me for making this slide.  She says I'm a little bit of a fanboy for Elon Musk, but, you know, Elon, he's sitting  in his chair and he gets his newspaper and he's reading it and it's an article about  how there was another gravitational wave detected and he says, you see, it's happened  again.  And his reasonable friend comes over to him and says, Elon, this is base reality.  It's not a simulation.  And meanwhile, somewhere else in the universe, there is this old guy sitting at a computer  and he says, Jesus, the build failed again.  Can you rerun it?  And in the other room, this younger guy says, sorry, I'm kind of busy.  He's playing games on his phone.  And the old guy says to him, don't make me send you back down there.  And he responds, yes, father.  All right, so this is something we all deal with.  I'm going to be using some code samples in my presentation today.  These samples are all based on RSpec.  You can take the workflow, though, that I'm going to share with you and this can be applied  to any testing framework.  But my primary experience is in RSpec and it provides a lot of tooling that helps us  to do what I'm going to show you today.  There are also some source code examples that I put together that are available.  I'll have this link again at the end of my slide and I'll put it on Twitter.  But this has examples of some of the test failures that I'm going to talk about today.  And the build in this repository is purposely failing so that you can go familiarize yourself  with these things and solve the failures that are happening in that repository.  So this is a common thing that happens.  You feel like the build is passing consistently for a while, depending on the size of your  repository and how many developers you have.  It might be a while for you.  It could be a day.  It could be three days.  It could be a week.  And inevitably, somebody says, ah, this is awesome.  The build has been passing for so long.  This is great.  And then what happens, the next build always, always fails.  And people say, oh, well, that's just a flaky test.  And before we get into some of the workflow part of this, I want to talk about the myth  of flaky tests.  And I want to deconstruct this for us.  Because there's this common notion that there are things in our test suite that are unsolvable.  And I've been doing this for long enough that I just don't buy it.  So I want to remind everybody.  Your tests are software too.  If you have an API call that is critical to your business, and it's failing every once  in a while, and the CEO or a business person comes to you and says, hey, why is this failing?  Do you say to them, oh, well, it's just a flaky endpoint.  And there's nothing we can do about it.  No, you don't do that.  It's software.  You make it do what you need it to do.  Test code does exactly what you tell it to do.  It won't do more than you tell it to do, and it won't do anything less.  It's going to do exactly what you say.  When we use the term flaky, we're implying that there is some kind of unsolvable problem.  We have this failure that we just can't deal with.  There's nothing we can do about it.  I like to take that term and replace it instead with the term non-deterministic.  Because then what we're saying is we have behavior that's happening in our system that  can be accounted for.  Once we understand the root cause of a failure, we can modify our test, or maybe it's a production  bug and we can fix the bug, but we can account for that non-determinism so that our tests  themselves are deterministic.  I also want to remind everybody that when you're ignoring failures, they could be real  defects.  You could have legitimate defects in your production code that people are simply ignoring because  it's difficult to troubleshoot.  So I want to talk a little bit about how I got here and how we came up with this.  It's a story that is fraught with peril.  There were a lot of failing tests that I've had to deal with in the past.  This is where what I'm going to tell you about today came from.  For those of you who have not been in the development community for a long time, I'm  going to be talking a lot about continuous integration.  This is where you push your code up to, let's say, GitHub and some service is going to automatically  run your tests for you.  Some examples of that are Travis or Code Ship or Circle CI or Jenkins.  This is what we're talking about when we say continuous integration.  So back in the day when Avant was small, we had a small number of tests and code that  we could load up and put into our CI tool and it would take our build and it would run  it and hopefully things would go smoothly and everybody was happy.  But over time, our code base grew and our test suite grew and so our test suite was  taking a lot longer.  So we said, all right, we need to paralyze this and manage our build times.  And so when I say paralyze builds, that means we're taking, let's say, 1,000 tests, 10,000  tests, however many tests you have in your test suite and you're splitting them across  a bunch of different containers or workers or whatever you want to call them.  And so we did this and we split our build across a bunch of containers and every once  in a while something would explode.  But we were focused on building features and when you're focused on building features,  you can lose sight of the fact that this is a bug that's existing in your test suite.  And people like to say, oh, well, that's just a flaky test and master.  Run your test suite or run the test locally and if it passes, go ahead and merge your  code or rerun master and then deploy as long as it passes.  And so these things tend to just get left behind.  And so as time went on, we kept building more and more things.  We had much more complex dependencies.  And so we started using more complex things in our test suite like factories to help us  build out these large graphs of dependencies for some complicated business logic that was  getting tested.  And eventually, people just don't really know everything that is going into what's happening  when they run a test.  And so we were still running a lot of builds and running a lot of tests and we had a lot  more code and things were still blowing up sometimes and people would complain about  it, but they wouldn't always do something about it unless it was a critical failure  where it was failing all the time.  Now eventually, we got to the point where we realized, you know what?  Our provider that we're using isn't really cutting anymore.  We need more capacity.  And so we decided to search for a new CI provider.  And so we were looking for a CI provider that could give us these big machines where we  could run a lot of tests on them and it would shorten our build times for us.  And so we did that.  And we tried it.  We put our test suite on this new CI provider that would run things more efficiently and  all of a sudden, everything is exploding all the time.  And what we found was that our old CI provider was breaking down our tests into these very  small groups and there were some things that they were doing that just made it so there  were some things that we just never even saw because certain tests would never run together  and there's a couple of other different things they were doing.  But in any case, when we put everything together and we randomized our test suite, we had a  really big problem.  A colleague of mine and I spent the next eight to 10 months pretty much just fixing tests  and then we were finally able to switch to a new provider.  So today what I'm going to share with you is a workflow for troubleshooting these types  of failures that came out of this process.  I fixed thousands of tests.  And so when something fails, I usually already know what probably is happening.  And so I'm going to give you this map to fix your test failures so that you can go back  and you can do it yourself.  So all right, master fails.  A test fails and master, woohoo, we're all really excited, right?  This is the best day ever because whenever master fails, you have something to do.  So what do we do, though, really, to debug this?  First thing we want to do is we want to run that test by itself.  Does it fail?  If it fails, then okay, does it fail every time?  This is the best possible solution.  We have a reproducible failure.  We have something that fails every time and is going to be really easy to fix because  I can just run it on my local machine and debug it until I can figure out what's going  on.  This is probably one of the number one challenges in solving test failures when you have an  intermittent test failure is you can't reproduce it sometimes.  So how do we debug reproducible failures?  I'm just going to touch on a couple of common failures that occur, but I'm not going to  linger on this because this is something that is generally relatively easy to fix.  So a couple of things.  Stale branches.  You might have branches in, I'm sorry, you might have someone that is working on a branch  and they don't back merge for a week and then they come back from vacation and they're  like, oh, I got signed off on my PR, boom, merge, and all of a sudden, master fails.  But what happens is in that last week, maybe a logical change was made that's incompatible  with their branch and now the tests are failing.  Business dates and times.  You might have logic that depends on the day and time and sometimes you can get to a weekend  or a holiday or maybe the next year and all of a sudden your test suite is failing.  Mock time and system time.  If you're using a gem like TimeCop or TimeOrp or others to modify the Ruby system time,  it's important to remember that if you are relying on something, if the code that you're  testing is relying on some external process, which is also doing something with date and  time, all of the external processes from your Ruby process are not going to share the mocked  Ruby time.  Missing preconditions.  This is another common one.  If you're distributing tests across a bunch of different containers and you have a test  that or the test code or the production code is not requiring something that it needs or  maybe it's not doing the data setup that it needs and it's been relying on some other  test to do that for it, you can run into situations where when that test is run on some node in  your CI environment, it fails because the other test is no longer running before it.  Then finally, of course, real bugs.  Make sure when you're fixing a test you're not just putting a band-aid over it where  you don't understand what the problem is and so you just put a band-aid on it to fix it.  Make sure you understand the root cause and that you're not just putting a band-aid over  legitimate defects.  All right.  So let's go back.  Let's say we ran the test by itself and it doesn't fail every time.  It doesn't fail.  Or maybe it doesn't fail every time.  The next thing we want to do is we want to take the group of tests that ran on the CI  container that failed and we want to take the seed that was used to randomize the test  and run it again.  So what is a test group?  A test group is a subset of tests from our test suite which run on a specific node in  your paralyzed build.  So here are some examples.  If you're just using the RSpec command directly, this means you don't have a test group.  You're just running all of the tests.  Maybe you have a list of files that are being piped into the RSpec command.  This is your test group.  Or maybe you're using a tag or some other command line option of RSpec to specify your  test group.  Test seed is a value, it's an integer in RSpec that you can pass into RSpec and what it'll  do is it'll reproduce exactly the order that the tests were run in in your CI environment.  So here we have output from RSpec.  The output's truncated.  And so at the end of the RSpec output when you run your tests, you'll see this line where  it says randomized with seed.  This is your test seed.  So you're going to take that.  And what we're going to do is we're going to take our group of files and we're going  to take our test seed and we're going to give that to the RSpec command.  And so you can see here we have our test groups, a couple of different examples.  We're using the seed option to pass in the test seed.  And I also like to use the fail-fast flag because when you're running a large group  of tests that take a long time to run and something fails in the middle, it's really  nice to have RSpec just automatically stop when it hits that failure.  Okay.  So we rerun our test with our group and our seed.  What do we do next?  All right.  So if it fails, the next thing we want to do is we want to bisect that failure.  In RSpec 3.3, I believe, they added a feature that will automatically do this for you.  So when we talk about bisecting tests, what we're saying is we're going to take the full  group of tests that ran with this failing test and we're going to cut them in half and  cut them in half and in half again until we get to the minimal command required to reproduce  the failure.  So ideally you end up with one test that's passing and then your failing test.  And when those two tests run together in a particular order, you'll see that other test  fail.  So here we have, this is essentially the same command as we ran before.  The only difference here is that instead of the fail-fast flag, we're replacing it with  the bisect flag.  And this is going to take a long time if your test suite, if the group of tests that you're  running, take a long time to run.  But it will reduce that set to a minimal command.  All right.  So now we've bisected our test failure.  If our bisect succeeds, then we know that we now have a test failure that is of the classification  test pollution.  What is test pollution?  Test pollution is when you have one or more tests where the side effects of those tests  are causing another test to fail.  And so how do we debug that?  There are a set of very common failures that we see, which usually these can be attributed  to.  And so I'm going to talk through some of those.  So data pollution is a really common type of failure that can occur in a test suite.  And this is when you have one test that is, let's say, creating a record outside of the  usual test transaction.  It's persisting something to the database, or maybe it's persisting something to Redis  or some other store, and it's not resetting it afterwards.  So I'm a big proponent of isolation testing and making sure that your tests are cleaning  up after themselves.  But what we've found and what we practice now at Avant is we practice defensive testing.  So the thing, what that means is it means that your tests shouldn't expect pristine  conditions.  So they should clean up after themselves as much as possible, but they should not expect  a clean slate.  And let me give you some examples.  So these are things that put really hard constraints on your whole test suite because  of a single test.  So you might have a test like this where it performs some action that has a side effect  that creates a user.  And after that action is performed, it sets an expectation for the user count to be one.  What does this mean?  This means now that no other tests in our test suite, none of our test support code,  none of our fixtures that we have can create a user without breaking this test.  That's a really hard constraint to put on an entire test suite for the sake of one test.  So what we can do is we can rewrite this test instead to expect change so that we're not  putting that global constraint on our test, and our test will still pass even in the presence  of other records.  Similarly, don't expect global scopes to only return the records you created in your  test.  Here we have an example where we're calling an active scope on a user model, and we're  saying expect the results of that to match this array.  And what we do is when we say match array, we say, all right, expect the results of the  scope to have exactly these users in it.  But what if another active user was created by another test or by a fixture?  This will fail.  What we prefer to do instead, and the way we can solve this, is we can instead expect  inclusion and exclusion.  So this allows us to set the assertions that we want to, sorry, it allows us to assert  the things that we want to assert in our test without having this reliance on a super clean  database when it runs.  Another common failure that we see is caching.  Caching is really hard.  It's a very hard problem in computer science.  And so one of the things here that's important to do is that if you have a test that's mutating  global state or a global singleton in a breaking way, oh, it is going to break other tests,  ideally you want to reset that cache after that test mutates it.  Better yet, if you can, use stubs or mocks to avoid modifying the cache at all.  Now if you're testing a singleton, what this means is if you are implementing singleton  behavior using class methods explicitly and you don't have a singleton instance, then  it's going to be really hard not to mutate your global cache.  So whenever we're implementing singleton behavior, what we do instead is we implement it using  a singleton instance.  And then you can see here, the top example here is testing a singleton class directly.  So we're adding a key to our singleton class, and then we're checking that the set of keys  that it's storing contains that value.  Now what I prefer to do when I'm testing a singleton so that I don't mutate the global  state is create a separate instance of it for the purpose of testing the singleton's behavior  and perform my test on that instance.  That way I'm not mutating global state.  You can still test that your class level methods are properly delegating to the singleton instance,  but you don't want to mutate it directly.  Mutated constants.  Sometimes you see this, someone overwrites a constant in their test because maybe there's  a timeout value that they want to reduce so that their test runs faster, or maybe there's  a feature flag that depends on a constant that has a data sign to it that turns that  feature on or off.  You don't want to modify this constant directly because that value could change in the future.  Even if you reset it back, maybe that value gets updated in the future and it's no longer  the same thing, you're going to be resetting it back to the wrong value.  So instead, learn the tools that your testing framework gives you.  OurSpecMox has a very comprehensive mocking framework.  So you can use things like stub const to replace a constant value with something else, or you  can allow a class method to be received and return a stubbed value.  These are things you can do to avoid constant mutation, and what it's going to do is at  the end of your test, when it gets torn down, it's going to replace it with the original  value.  Don't rewrite class, don't rewrite production class implementations.  This is something that I've seen before.  I don't know what else to say.  It's not a good practice.  It's much better to just create an instance of the thing that you want to modify and then  put a stub on it.  You don't want to modify the implementation of your classes because what happens is now  you've modified the implementation of that class, and that's going to carry through to  any test that run after this one.  Don't mutate, sorry.  Test constants is another interesting one.  Sometimes people do this.  They think, oh, I need this variable in my test so that I can assert on it in a bunch  of different places, and so they'll define a constant in their test and assign a value  to it, and then they'll use it in their assertion.  And what happens is three months later, somebody comes along and they copy the production class  because it does something close to what they want to do.  They modify it a little bit.  They copy the original test.  They modify it a little bit to reflect the new behavior, but they don't update this constant  name.  And what happens is now you have two constants that are defined in your test, and people  think that, oh, well, this is a scope to my describe block or a scope to my context block,  but your describe block is not a namespace.  It's not a module in Ruby.  And so what happens when you declare a constant here, it is going to go up to the global namespace,  and so when these two tests run together, whichever one gets loaded second is going to  win, and so the first test that runs is going to fail.  And so it's preferable to use let variables or use local variables in your tests so that  you're not mutating global state.  And of course, always look out for real bugs, especially as you start getting into intermittent  failures, things can get a lot more challenging, and a lot of times it can come down to a production  issue.  All right, so let's go back.  We re-ran our test with our group in seed.  Let's say it doesn't fail, or maybe we go and we try to bisect it, and the bisect is  unable to come to a minimal command to reproduce or can't reproduce it at all.  What do we do next?  So the next thing we do, and this may sound really stupid and easy, because it is stupid  and easy, but run your test in a loop.  This is an example.  Take your test inside of your outer level describe block, put 100 times do block, and  then run that test by itself.  A lot of intermittent failures can be reproduced this way.  When I do this, I like to use the fail fast flag again because it'll stop execution immediately  when a failure occurs, and I can start taking a look at it.  All right, so we run our test in a loop.  Does it fail sometimes?  If it does, we know we have a non-deterministic failure.  What that means is we have a test that is, either the test itself is doing something  that's non-deterministic and making an assertion on that behavior as if it is deterministic,  or you have production code that is non-deterministic, and the test is asserting that it is.  So non-deterministic failures are failures that occur seemingly randomly in our test  suite for no apparent reason.  And so I'm going to give you a couple of common types of failures that we see that fall into  this bucket.  One of the most common failures that I've seen that this happens with is unordered  queries.  People assume that when they make a database query, it's going to give them back the results  in the order that they created the records.  But this is simply not true depending on what type of database you're using.  If you're using Postgres, which a lot of Rails users do, if you look at the Postgres documentation,  it explicitly states that if you do not specify the ordering of a query, it is going to return  the results in the order that they are found.  And usually that's going to be the order they're created in a test, but not always.  And so what we want to do here, there's an example on the top.  Let's say we have a scope that's unordered that we're testing the results of.  If we say expect the results to equal some set of records, what our spec is going to  do is going to say, okay, oh, your results, they have the right records in them.  But what it's also checking for when we check for a quality on an array is it's checking  that the order of the elements in that array are exactly the same.  So when we're testing unordered queries, what we want to do instead is we want to check  that the result set contains exactly the records that we expect or that it matches an array.  But what these two matchers do is they perform that expectation without a dependency on the  order that the results are in.  Frozen time is a similar issue that's pretty interesting.  So if you, again, use TimeCop or some other gem and you're freezing time, what happens  is all of the records that are created are going to have the same created at.  And so if you perform a query that is ordered, and it's ordered by the created at time, the  sort order is going to be non-deterministic because the creation time for all of them  is going to be exactly the same.  So we prefer TimeCop travel over TimeCop freeze.  When you say TimeCop.travel, what it does is it travels to the time you give it, and  then time continues to roll forward.  So if you create a set of records when you're traveling through time, they're going to have  different created at timestamps.  But if you're freezing time, they're not.  Only freeze time when precise time is needed.  This is almost never in my experience.  All right.  Another common case that we see is randomized test data.  If you're using the Faker gem or maybe you have a set of test data that you are sampling  for the purpose of randomizing your test data, you can end up with unexpected failures.  Maybe it's returning names in a format that your code doesn't handle properly.  Or maybe the test code is asserting the results incorrectly.  Maybe there's phone numbers that don't match validations in your production code or zip  codes that you don't support.  Or maybe you're sampling from all of the United States, but you only support the continental  United States.  Sometimes you're going to have a failure.  One of the things you can do to help with this is if you are writing tests that rely  on faked data like this, is to output that data with your test errors.  So when the test fails, you have more context that you can use to figure out what's going  wrong.  All right.  So we go back.  And what if we ran our test in a loop?  Because that test still doesn't fail.  This is what we call unreproducible failures.  And I put quotation marks around it because you can't reproduce it, but it's breaking  CI.  So obviously it's reproducible somewhere.  These are the most difficult ones sometimes to figure out.  And so we'll talk about a few common cases here.  So data in time is a really common issue.  Data in time is really hard to deal with, especially when you're dealing with time zones.  So some of the things you want to look for in your CI builds that are failing is do your  tests only fail on weekends or holidays?  Do they fail at certain times of day?  Maybe you have some code that relies on business time or business hours.  Time cop your test to the data in time when CI ran and see if it fails.  We have a time cop RSpec gem that we open sourced.  I would say go check it out.  It makes it really easy to work with time cop in RSpec using RSpec metadata.  And we also have a cool little feature where you can set an NVAR and it'll time cop all  of your tests.  And we use that for running time travel builds so we can catch failures in advance.  So another common failure here is UTC and local date time differences.  A lot of people don't know this.  Date.today uses your system time zone.  Date.current, which is part of active support, uses your application time zone.  And so what happens?  Here's an example.  This is available in that repository that I shared earlier.  On the top line, this is mimicking a system like a continuous integration system where  the system time zone is set to UTC and our application time zone is set to central time.  And what you'll see is if you run this test, this test is going to fail because it's asserting  date.current is equal to date.today.  And what happens is when you are for certain times of day when the UTC date and the local  date don't match, your test is going to fail.  And a lot of times this can be a production bug.  SQL date comparisons.  This is another really interesting one.  And this is pretty much always a production bug.  So it's really important to know that when you are doing SQL date comparisons, let's  say you're looking for records after a certain date or before a certain date, when you pass  a SQL string to active record.  What it does is it takes your value that you give it.  And because it's injecting it into a string, it doesn't know what column you're comparing  against.  And so it can't infer the type of that column.  And so what it does is it's going to coerce the value that you give it to the default  database string that it would normally generate for it.  So if this is a value that has time, like a date time object or a time object or a time  with zone, what it does is it coerces it to a date with time string without zone in UTC.  And so if you are passing a time object into your query and you're querying against a date  column where the date columns are stored in as local dates, there are going to be certain  times of day where the UTC date does not match your local date.  And Postgres or whatever other database you're using is not going to know to coerce it to  the current local date.  Timeouts in asynchronous JavaScript.  This is another really common source of errors.  I know a lot of people hate doing acceptance or integration testing, whatever you want  to call it, browser testing.  So when you have a failure in a browser test, it's important to keep in mind that your performance  in your continuous integration environment is usually worse than your local machine.  So you might never reproduce this failure locally, and if it happens, if you have a  timeout occasionally in CI and you can never reproduce it locally, oftentimes you'll find  it's because of differences in performance.  Also performance can vary widely depending on what tests run before a particular test  or how your application is set up.  Maybe a test runs before it sometimes that renders some of the views that a page needs  in order to finish rendering.  So sometimes it'll take less time to render.  When you run into these types of situations, you want to increase your timeouts for CI as  needed.  You don't want to have a fit.  It's most important for you not to have failures in CI.  You don't want to use your browser test for performance testing.  If you're setting low timeout values for your browser test because you say, oh, well, this  page needs to load in three seconds, you should not do that in CI.  You should always be testing performance against production or production-like environment.  CI is not a production-like environment.  So also if you have JavaScript that's executing, sometimes you will see failures that happen  because your test tries to interact with elements on the page before the page is fully loaded.  So I added a feature to the Site Prism Gem.  If you're not familiar with it, the Site Prism Gem is an implementation of the page  object pattern, and if you're not familiar with that, I would just recommend googling  it so that you can learn more about it because I don't have time to talk about it right now.  But what the load validation functionality does is it allows you to specify a set of preconditions  that must be fulfilled before the block that you're executing in your test is actually  executed.  So you can say, wait for my spinner to disappear from the page or wait for this form to appear.  And then when those preconditions are met, it will continue to test.  And if those preconditions aren't met, it will give you back a reasonable error.  So if you still can't reproduce it, what do you do then?  You need to start digging.  You need to start looking for environmental differences.  Compare your CI configuration to your local environment.  Are there environment variables that are set in one place but not in the other?  Are there differences in how your tests are being executed?  Maybe you're executing the RSpec or MiniTest or whatever other command differently than  you do it on your local machine.  Try what you're doing in CI and see if it fails for you locally.  Maybe there's an issue with how the database is being set up in CI.  There might be seeds that are missing or present that you don't have on your local machine.  Or maybe someone committed a migration but in CI, a lot of times you might load from  a schema or a structure file because it's faster than running migrations.  And so if someone forgets to commit a migration to the schema or structure file, you might  have some failures.  Some other environmental differences that you might run into is library versions or  inconsistencies in how specific libraries behave.  There's a lot of differences between OS X and Linux commands where they don't always  behave the same.  If you can, use Docker for consistency.  If you're using Docker in CI and you're using Docker on your local machine, you know that  you are running your test in the same virtual environment.  And so that can ensure consistency.  If you still don't know what's happening, some other things you can do is SSH into your  CI environment.  If you can't do this, try to find a CI environment that allows you to do it because sometimes  this is the only way to continue.  I did this, I actually was doing this on Friday.  Use common sense.  Think about the failure that's happening.  Look at the stack traces you're getting.  Can you come up with a hypothesis about what is possibly causing it?  There was an interesting talk earlier today about using the scientific method when you're  debugging.  Go listen to that talk and think about how you can apply that to your test suite.  If you think that there's a third-party gem that might be causing this problem, go look  at the repository.  See if other people have had this issue.  Maybe there have been bug fixes that you don't have in your application yet.  And all you need to do is upgrade your gem.  And if it's not there, maybe you should file a ticket and contribute back to the open source  community if you do find a defect.  Learn to use pry and buy bug.  These are really, really useful tools that can assist you in debugging your software.  And finally, incrementally narrow the scope of the defect.  If you're just throwing darts against a wall and you're not tracking what you're doing  and doing it systematically, there is a lower likelihood of finding the root cause of the  defect.  You want to be continually narrowing the scope so that you're looking at finer and finer  grained possibilities of what could be causing the issue.  Rule things out.  Make sure you know your test support code in and out.  This is important.  Because we have a test suite that has over 30,000 tests in it, and so we have a lot of  test support code.  And sometimes there can be conflicts in what the test support code is doing that sometimes  can cause issues with our tests.  So make sure you know what your tests are doing.  Look at failure trends over time.  If this test is failing for a long period of time, maybe you can find, maybe there's  other tests that are failing in a similar way, and you can find correlations between them.  Or you can look at the builds that have failed with this failure and look for correlations  between them.  You can add logging.  If the test is still failing, it doesn't hurt to add some logging to that test to give you  more information about it so that then the next time it fails, you now have more context.  Thank you all for coming.  I'm really happy that I was able to talk to you today.  Just some quick takeaways.  So keep your builds green to avoid sadness.  It's really disheartening when you come in on Monday and the build is failing, and it's  not just one thing.  It's 10 things that are failing, and you don't even know where to start.  Remember that tests are code two.  They do exactly what you tell them to do and nothing more.  Set realistic goals for yourselves.  If you have a test suite currently that suffers from this problem, you have to keep in mind  that it's not necessarily going to change overnight.  It takes time to root these problems out and solve them.  And finally, celebrate success.  Set milestones for yourself and celebrate them.  This is a picture of cupcakes that our software engineering manager got us when we hit a 95%  pass rate on our build for a week straight.  That was a really big accomplishment.  So again, please get in touch with me.  I'm going to be posting a couple things on Twitter for you, and I also have a larger  version of the workflow that was in my slide deck that I'm going to share.  Thank you for your time.
Deterministic_0_47.png Deterministic_48_95.png Deterministic_96_287.png Deterministic_288_647.png Deterministic_648_1295.png Deterministic_1296_1871.png Deterministic_1872_3227.png Deterministic_3228_3743.png Deterministic_3744_4427.png Deterministic_4428_5267.png Deterministic_5268_5675.png Deterministic_5676_7523.png Deterministic_7524_7883.png Deterministic_7884_9635.png Deterministic_9636_9983.png Deterministic_9984_10859.png Deterministic_10860_11519.png Deterministic_11520_11987.png Deterministic_11988_12311.png Deterministic_12312_13847.png Deterministic_13848_15635.png Deterministic_15636_15803.png Deterministic_15804_19559.png Deterministic_19560_19787.png Deterministic_19788_20351.png Deterministic_20352_20723.png Deterministic_20724_21071.png Deterministic_21072_21851.png Deterministic_21852_22991.png Deterministic_22992_23699.png Deterministic_23700_23771.png Deterministic_23772_23831.png Deterministic_23832_23879.png Deterministic_23880_24035.png Deterministic_24036_24563.png Deterministic_24564_25067.png Deterministic_25068_25727.png Deterministic_25728_28295.png Deterministic_28296_29183.png Deterministic_29184_30827.png Deterministic_30828_32399.png Deterministic_32400_33347.png Deterministic_33348_35267.png Deterministic_35268_36179.png Deterministic_36180_36767.png Deterministic_36768_37595.png Deterministic_37596_37787.png Deterministic_37788_40043.png Deterministic_40044_43331.png Deterministic_43332_43691.png Deterministic_43692_43943.png Deterministic_43944_45599.png Deterministic_45600_46019.png Deterministic_46020_47075.png Deterministic_47076_49139.png Deterministic_49140_52979.png Deterministic_52980_59903.png Deterministic_59904_60935.png Deterministic_60936_61271.png Deterministic_61272_61787.png Deterministic_61788_61931.png Deterministic_61932_61974.png