I have a lot of strong opinions about this issue, but had a hard time finding official documents today to back up my position. A lot of times we say it is bad practice to use seeAllData=true, but we don't really explain why. Maybe we can generate a more definitive and thorough answer here. So I want to ask the broader community:

Why is it important to create our own data when we write unit tests?


I think the answer is quite simply one of consistency. It's also one of minimizing variations between environments we test in. This makes our test more consistent and repeatable. These are all things that are helpful should we need to diagnose or troubleshoot a problem during deployment.

  • When we generate our own data, that eliminates a variable between what happens in any sandbox vs a production environment. We KNOW we've used the same test data. We know what values we can expect to see at any point in our unit test.

  • We can create not only the database records, but also our User records. This allows us to control the context for running our unit tests with RunAs().

  • Similarly, we can create users who become the owners of the records in our test data. Doing this creates even more consistency and reduces additional variances between environments (development org to development org or vs production). Again, this allows us to eliminate another variable should we need to troubleshoot a deployment from one environment to another.

  • We don't always know that the kind of data we need to run our tests will exist in an org if we don't create it. This is especially true when testing in a sandbox, a fresh org, or one where records of the type our code uses have never been created. How an our test class possibly grab test data from the org if there are no records records that meet the criteria to test all of the conditions our code uses? Clearly we can't do that. It's incumbent on us to ensure the data we need is available which means we need to create valid data to test with.

  • Our code must also be bulk safe. In my view, testing using only one record per method doesn't constitute an adequate unit test. All it does is allow code to "pass the minimum requirements" needed to deploy it. The latter doesn't ensure our code will function reliably. That's what system.asserts, positive and negative test cases, bulk test cases along with other test methods are for. I strongly believe that our unit tests need to include those methods as well.

  • I want to add that what I'm speaking of doesn't prohibit a developer from creating randomness in their data. It still allows for that. The randomness is created in a controlled and consistent manner so the behavior of the code can be predicted and results still asserted. Validity of the data will be known provided the environment doesn't "do something" to it that's unexpected.

  • It's this consistency during creation that allows the diagnostics and troubleshooting of what's "different" between two environments when there's an issue with deployment or functionality of the code is not as expected.

This goes beyond the question that's been asked, but I'm a big believer in Test Driven Development. These help me test functionality as I build my code. When I'm finished, my test class is also completed and it's not an afterthought or chore. I like to create tests using lists of records where I can change a variable in my class that initially sets the list size to a length of 1. When I'm ready to bulk test, I'm then able to set it to a larger number, including raising it to 200 (or even more if appropriate), to ensure my code is bulk safe.

During deployment, I can lower that value to create 1 record/list to test with so time isn't wasted. But the method is there in my unit test and I KNOW that my data has been bulk tested and can easily be retested at any time in production. If I want to, the value of that list size can easily be controlled through a custom setting via a test environment utility/helper class. This allows me to retest in bulk at any point in time I want to. I'll add that I generally create a test utility class that creates all of my objects. This is great way to reuse my code.

So, my answer to the question is we do it so the developer of the code can achieve the maximum amount of control over the consistency of test conditions as he or she possibly can; regardless of the environment it runs in, each and every time the test method is executed. This is a discipline I was taught in engineering school for designing and creating tests. It was also expected when I was an engineer in the automotive industry (see this article by Charles Deming famous for "The Deming Way"). It only seems logical to me that it would apply to software as well since it applies to any other kind of testing, including tests of humans using software and electronics running it as well.

|improve this answer|||||
  • 1
    This. Awesome writeup! I too include in every test class static final Integer RECORD_COUNT. I actually set it to 101 for all sandbox tests and 5 for production. – Adrian Larson May 26 '16 at 16:24
  • 1
    You couldn't really use a Custom Setting to control test size without using seeAllData=true though. Perhaps an advantage of Custom Metadata Types? – Adrian Larson May 26 '16 at 16:26
  • 2
    @AdrianLarson Actually, you could put it in a separate class that's used for test class settings which your test methods look at to get the value without needing to use seeAllData=true. Feel free to edit this with bullets however you'd like. This is a collaborative forum and all improvements to posts are encouraged. – crmprogdev May 26 '16 at 17:00
  • I'm surprised this answer hasn't gotten more attention. You definitely filled in what I was looking for. I guess when you throw your hat in the ring late, you don't get as much love. Leaving the bounty open for now to draw more attention, but I consider this canon. Thanks again! – Adrian Larson May 27 '16 at 19:29
  • 2
    @AdrianLarson You're welcome and thanks for the compliments. I recall being extremely busy at the time you initially posted the question. When I saw it had an open bounty on it, and thought about how I'd have answered it, I realized why there was a bounty and decided to post the answer above. – crmprogdev May 27 '16 at 21:14

In two words: "bitter experience"

I will admit that if you don't have any asserts in your code (another topic) you run a lesser risk but, the reasons all have to do with repeatability/predictability and the advantages therein.

Every deployment of the same test class should run the same environment every time. Today, tomorrow, and 5 years from now. Here are some examples that have happened to me over the many years:

  1. Your SOQL might return way more sobjects than you found in sandbox due to poorly constructed where clauses. Hence, you run into governor limits when deploying.
  2. As a variation of #1, you might have an assert expecting n objects but the test returns 20n. Hence, deployment fails
  3. As another variation of #1, you might have an assert fetch n objects but they don't match the n objects you mocked. The deployment fails. In fact, this happened to me today with some unmanaged package class I had been using that was at V16.0 (seeAllData=true was implicit back then). The deployment occurred when users were actively adding Accounts and an "interloper" appeared in the assertion error message.
  4. You might be expecting custom settings or other sobjects used as configuration data to have the "right" values but there is no guarantee PROD will match sandbox. Hence, deployment fails.

So, for argument's sake, let's assume you are a good developer and strongly believe in system asserts to "prove" the correctness of your implementation with positive/negative/bulk tests. By the time you have done all this the last thing you want is a deployment error -- On the next deployment or on some deployment far in the future. You are under time pressure when you get to deployments, you may have limited deployment windows, and finding errors in deployments is more difficult than in sandbox.

And lastly, professional pride. You are leaving a legacy of software for some other developer/admin and why not treat them to repeatable/reliable code just as you would want to be treated.

|improve this answer|||||
  • @adrianlarson - thanks a million (or at least 200) :-) – cropredy Jan 13 '17 at 0:40

Another point to add to cropredy's excellent list is that tests have significant value as a specification of what the code being tested does. But that only works if you can see the inputs (often the database rows) and expected outputs (the asserted values) in the test. Inserting exactly the data that is relevant to what is being tested in the test makes those inputs visible.

Somewhat related is that it is actually easier to write assertions where you have a code reference to the input. Though if you take advantage of @TestSetup to speed up the tests by setting the data up once, you lose the simple references to the data (as you have to query to get them in each test method).

|improve this answer|||||
  • 1
    Yes! Awesome point about input/output matching. You can set up test data outside of @TestSetup just as well as within it, though. Can you clarify what you're getting at there? – Adrian Larson May 19 '16 at 13:21
  • 1
    @AdrianLarson Was really just an FYI point: sure you don't have to use TestSetup but using it has the disadvantage of you not being able to keep variable references to the data AFAIK. – Keith C May 19 '16 at 13:34
  • Ah, but that's just one more argument to use static final constants. IMO it's cleaner anyway. – Adrian Larson May 19 '16 at 13:35
  • 3
    Keith - I agree with you on the limitations of @TestSetup; it is annoying to have to requery for the objects. – cropredy May 19 '16 at 16:27
  • 1
    @KeithC A bit late chiming in here. Like you, I'm not fond of @TestSetup. I prefer using a common utility class to create records within each method at run time. I write them in a way that allows me to create related records by passing parameters within my utility classes so I can return lists with varying sizes, param names, etc. It still allows me to make consistent & predictable assertions. – crmprogdev Jun 10 '16 at 13:12

The Hammer Execution team explicitly asks us to use data silos in our tests (emphasis mine):

Our Commitment To You

This complex juggling act happens three times a year. We make a commitment to you that your customizations will continue to run as you expect, no matter how much we do to change the system. In all my years of being in enterprise software, I have never heard of such a commitment. I’ve never heard of such a smooth upgrade path: you go to bed on Friday, you wake up Saturday, and your system has been upgraded. I still have flashbacks to lenghty, painful upgrade projects for other companies I have worked for – those were never easy times. To save you from these flashbacks, we do the complex ballet described here (which still makes it sound easier than it is) three times a year.

You can help us fulfill this commitment by using data silo. Tests that utilize the new data silo paradigm (seeAllData=false) create their own data, and don’t actually need access to the production data stores. These tests are much faster for us to run, and much more reliable. We don’t need to worry about reading your debug statements, because they’re outputting fake data. We don’t need to worry about having a copy of your production data, because data silo tests do not rely on that data to succeed. Org-independent tests make The Hammer smile.

I consider that compelling in its own right! I'm also happy to learn that following best practice in this case makes us good citizens of the platform.

|improve this answer|||||

Your Answer

