Unit testing sounds great to me, but I'm not sure I should spend any time really learning it unless I can convince others that is has significant value. I have to convince the other programmers and, more importantly, the bean-counters in management, that all the extra time spent learning the testing framework, writing tests, keeping them updated, etc.. will pay for itself, and then some.

What proof is there? Has anyone actually developed the same software with two separate teams, one using unit testing and the other not, and compared the results? I doubt it. Am I just supposed to justify it with, "Look it up on the Internet, everybody's talking about it, so it must be the right thing to do"?

Where is the hard evidence that will convince the laymen that unit testing is worth the effort?

11 Answers 11


Yes. This is a link to a study by Boby George and Laurie Williams at NCST and a another by Nagappan et al. I'm sure there are more. Dr. Williams publications on testing may provide a good starting point for finding them.

[EDIT] The two papers above specifically reference TDD and show 15-35% increase in initial development time after adopting TDD, but a 40-90% decrease in pre-release defects. If you can't get at the full text versions, I suggest using Google Scholar to see if you can find a publicly available version.

| improve this answer | |
  • 13
    The first study compares agile+TDD against waterfall projects, it's results would be more relevant if it had compared two agile teams. The second study mentions other studies that found little to no quality bonus for TDD projects. And when you compare management's estimates about needed extra time for TDD it's significantly estimated higher for the two teams with a high domain expertise, yet they also have a 20% lower test coverage. This confirms my own experience, I find assurance much more important in systems I haven't worked with yet, while testing is a hindrance for everything else. – LearnCocos2D Oct 14 '13 at 17:45
  • Neither of the studies compares comparable process model with only the testmethofology change. That is is spending the time used on UT actually better spent on eg. system testing. As it stands it might as well be " if we test smarter does that help" study. – Rune FS Mar 23 '14 at 8:40
  • 1
    So what if the cost of fixing the post release bugs is 0.01% of total development? TDD would be a terrible investment in that case. And if the bugs are few? These %s mean nothing without context. To be fair I am yet to read the whole study. But as it stands your post is useful (good links) but does not answer the question regarding ROI, IMO. – Instine Jun 13 '14 at 10:56
  • 1
    @Instine Luckily (?) there is good evidence that this is not the case. Fixing post-release bugs is exponentially more expensive than bugs found early in development (which is what TDD does). In that context, a cost of 0.01% of total development for all post-release bugs seems unlikely. (For details see Code Complete, in particular Boehm & al., “Understanding and Controlling Software Costs”, IEEE Trans Softw Eng (1988)). – Konrad Rudolph Nov 4 '14 at 10:52
  • It's probably worth noting that the first study has a sample size of 24 programmers (working in pairs, so 12 teams). I'm not sure what a statistically valid sample size would be, but these seem low. Perhaps someone else knows? – Zachary Yates Jun 14 '18 at 16:31



Why not just do it, quietly and discretely. You don't have to do it all at once. You can do this in little tiny pieces.

The framework learning takes very little time.

Writing one test, just one, takes very little time.

Without unit testing, all you have is some confidence in your software. With one unit test, you still have your confidence, plus proof that at least one test passes.

That's all it takes. No one needs to know you're doing it. Just do it.

| improve this answer | |
  • 9
    The bean counters couldn't tell a unit test from the rest of the code if their lives depended on it. I support the suggestion to just do it. There's one caveat, though: If you are not alone, you need your fellow developers to embrace this practice. If not, they will unintentionally break your tests. – Thomas Eyde Oct 25 '08 at 23:54
  • Just do it and don't tell them, and sell the idea to your colleges at the coffee break ;-) – Johan Feb 17 '09 at 20:06
  • 3
    Because you'd get fired when you consistently didn't hit your deadlines? – Andrew Jan 16 '10 at 5:09
  • 3
    @Neko: Unit tests don't add a "bit of overhead". They reduce the overall workload by preventing a whole flood of dumb mistakes. The work does not grow; it simply shifts in nature from bad code to good unit tests and good code. – S.Lott Jan 29 '12 at 14:19
  • 1
    The bean counters want their engineers to provide sound solutions to the domain problems. You can just write tests as part of your solution. They won't even notice. If they ask you can just tell them you are spending more time on it to make sure it is robust and won't require reworking. If you SUGGEST writing unit tests to them you are asking their approval on something they know nothing about. – Yorkshireman Nov 1 '17 at 16:44

I take a different approach to this:

What assurance do you have that your code is correct? Or that it doesn't break assumption X when someone on your team changes func1()? Without unit tests keeping you 'honest', I'm not sure you have much assurance.

The notion of keeping tests updated is interesting. The tests themselves don't often have to change. I've got 3x the test code compared to the production code, and the test code has been changed very little. It is, however, what lets me sleep well at night and the thing that allows me to tell the customer that I have confidence that I can implement the Y functionality without breaking the system.

Perhaps in academia there is evidence, but I've never worked anywhere in the commercial world where anyone would pay for such a test. I can tell you, however, that it has worked well for me, took little time to get accustomed to the testing framework and writing test made me really think about my requirements and the design, far more than I ever did when working on teams that wrote no tests.

Here's where it pays for itself: 1) You have confidence in your code and 2) You catch problems earlier than you would otherwise. You don't have the QA guy say "hey, you didn't bother bounds-checking the xyz() function, did you? He doesn't get to find that bug because you found it a month ago. That is good for him, good for you, good for the company and good for the customer.

Clearly this is anecdotal, but it has worked wonders for me. Not sure I can provide you with spreadsheets, but my customer is happy and that is the end goal.

| improve this answer | |
  • My QA guy was pretty sharp but he wasn't looking at code, but it was easy to tell the bounds weren't checked. – itsmatt Oct 25 '08 at 22:04
  • Totally agreed about unit testing forcing you to think more about your design and correctness rather than code recklessly – chakrit Oct 25 '08 at 22:28
  • 7
    Customers don't pay us to write tests. Then again, they don't pay us to write code, either. They pay us to solve their problems, and when confronted, I bet they also want the problems to stay solved. Given the evidence, it's unbelievable customers don't want to secure their investment. – Thomas Eyde Oct 25 '08 at 23:47

We've demonstrated with hard evidence that it's possible to write crappy software without Unit Testing. I believe there's even evidence for crappy software with Unit Testing. But this is not the point.

Unit Testing or Test Driven Development (TDD) is a Design technique, not a test technique. Code that's written test driven looks completely different from code that is not.

Even though this is not your question, I wonder if it's really the easiest way to go down the road and answer questions (and bring evidence that might be challenged by other reports) that might be asked wrong. Even if you find hard evidence for your case - somebody else might find hard evidence against.

Is it the business of the bean counters to determine how the technical people should work? Are they providing the cheapest tools in all cases because they believe you don't need more expensive ones?

This argument is either won based on trust (one of the fundamental values of agile teams) or lost based on role power of the winning party. Even if the TDD-proponents win based on role power I'd count it as lost.

| improve this answer | |
  • 13
    hear, hear :) A lot of the hard evidence for TDD also comes from very experienced teams that were already getting good results without it. TDD just improved their results rather than creating them out of thin air. The real ROI is hiring decent coders and letting them decide how to do things. – workmad3 Oct 25 '08 at 21:17
  • "Is it the business of the bean counters to determine how the technical people should work?" --> all business decisions come down to money. Still, good answer, +1 – jcollum Feb 17 '09 at 17:23
  • @jcollum but how you perform your job has nothing to do with money and if you wish dome one to be accountable you let them decide HOW they do the WHAT you asked of them – Rune FS Mar 23 '14 at 8:50
  • TDD is not a design technique, it is just a coding technique. blog.ploeh.dk/2010/12/22/TheTDDApostate Many commenters disagree that TDD involves refactoring (which is a design technique) but refactoring does not imply TDD. One can refactor without tests, large complex refactoring affects unit tests anyway i.e. tests need to be refactored too so can become invalid / false green just as well; simpler refactorings many not affect tests but the risk of error is lower - because refactoring is simple. – KolA Mar 7 '19 at 3:08
  • @KolA well, with the reflection of 10.5 years after this answer, I might phrase it a bit more defensive today, but still: I don't argue that TDD is the only design technique you'll ever need and Mark opens with it being a good design technique before concluding that it isn't one at all. I'd weaken his opinion and say it must not be the only design technique. Every code that I've ever written TDD looks different from code that I've written without. I'd call that a result of design. I work best with whiteboard, discussions, and other tools, in addition to TDD. But thanks for the link – Olaf Kock Mar 7 '19 at 6:24

If you are also interested in evidence against unit testing here is one well researched and thought out article:

Why Most Unit Testing is Waste By James O Coplien (lean and agile guru)

| improve this answer | |

More about TDD than strictly unit testing, here is a link to the Realizing quality improvement through test driven development: results and experiences of four industrial teams paper, by Nagappan, E. Michael Maximilien, Thirumalesh Bhat, and Laurie Williams. paper published by Microsoft Empirical Software Engineering and Measurement (ESM) group and already mentionned here.

The team found was that the TDD teams produced code that is between 60% and 90% percent better (in terms of defect density) than non-TDD teams. However TDD teams took between 15% and 35% longer to complete their projects.

| improve this answer | |

Here's a great and entertaining read of a guy changing his company from within. It's not limited to TDD. http://jamesshore.com/Change-Diary/ Note that he didn't persuade the "bean counters" for quite some time and did "guerilla tactics" instead.

| improve this answer | |
  • the link looks interesting... worth checking out re: changing organisations work processes... – nasty pasty Feb 18 '09 at 5:13

Just to add more information to these answers, there are two meta-analysis resources that may help out figuring out productivity & quality effects on academic and industry background:

Guest Editors' Introduction: TDD—The Art of Fearless Programming [link]

All researchers seem to agree that TDD encourages better task focus and test coverage. The mere fact of more tests doesn't necessarily mean that software quality will be better, but the increased programmer attention to test design is nevertheless encouraging. If we view testing as sampling a very large population of potential behaviors, more tests mean a more thorough sample. To the extent that each test can find an important problem that none of the others can find, the tests are useful, especially if you can run them cheaply.

Table 1. A summary of selected empirical studies of test-driven development: industry participants*


Table 2. A summary of selected empirical studies of TDD: academic participants*

enter image description here

The Effects of Test-Driven Development on External Quality and Productivity: A Meta-Analysis [link]


This paper provides a systematic meta-analysis of 27 studies that investigate the impact of Test-Driven Development (TDD) on external code quality and productivity.

The results indicate that, in general, TDD has a small positive effect on quality but little to no discernible effect on productivity. However, subgroup analysis has found both the quality improvement and the productivity drop to be much larger in industrial studies in comparison with academic studies. A larger drop of productivity was found in studies where the difference in test effort between the TDD and the control group's process was significant. A larger improvement in quality was also found in the academic studies when the difference in test effort is substantial; however, no conclusion could be derived regarding the industrial studies due to the lack of data.

Finally, the influence of developer experience and task size as moderator variables was investigated, and a statistically significant positive correlation was found between task size and the magnitude of the improvement in quality.

| improve this answer | |

Well, there are some large companies that require you to use unit testing but if you are a small company why mimic large ones?

For me when I started with unit testing , many years ago,(today we mostly use behavior model) it was because I could not control all the path in one application.

I was used to bottom first programming and a REPL so when I got Unit Test (One Test for Every Function) it was like bringing back a REPL to languages that where very much compile. It brought the fun back to every line of code I wrote. I felt god. I liked it. I didn't need a report to tell me that I begun writing better code faster. My boss didn't need a report to notice that because we where doing crazy stuff we suddenly never missed a deadline. My boss didn't need a report to notice that the number of "plain" bugs drop from (to many) to nearly nil because of this very strange thing of writing non-productive code.

As another poster already wrote, you don't use TDD to Test (verify). You write it to capture the specification, the behaviour of what your unit(object, module, function, class, server, cluster) works.

There are lot of failures and success stories of switching to a different model of developing software in a lot of companies.

I just started to use it whenever I had something new to write. There is a old saying that goes somewhat hard for me to translate to english but:

Start with something so simple that you don't notice that you do it. When training for a marathon, start by walking 9 meters and run 1 meter, repeat.

| improve this answer | |
  • So, I should just do it? It's guaranteed to work, and it doesn't matter if no one else does it with me? – raven Oct 25 '08 at 21:39
  • Actually, this is a Joel Test: joelonsoftware.com/articles/fog0000000043.html. It sounds to me that you may have more problem than a lack of the Nobel Prize Award Study On Unit Test – Jonke Oct 26 '08 at 0:07

There are statistics that prove that fixing a bug found in the unit/integration test costs many times less than fixing once it's on the live system (they are based on monitoring thousand of real life projects).

Edit: for example, as pointed out, the book "Code Complete" reports on such studies (paragraph 20.3, "Relative Effectiveness of Quality Techniques"). But there is also private research in the consulting field that proves that as well.

| improve this answer | |
  • 1
    This is covered in Steve McConnell's Code Complete, which is a book you probably want to have on your bookshelf for other reasons. – Robert Rossney Oct 25 '08 at 21:27
  • That's not related to the test method but to when in the process a bug is reported and further the time would be better spent finding bugs in the specifications since the cost of fixing them when finding them when developing is reported upto 1000 fold as expensive ( a factor of 10 per development phase) – Rune FS Mar 23 '14 at 8:46
  • OTOH, if you only fix the problems that people actually encounter in real-life situations, you probably end up having to fix far fewer bugs. It's also not clear to me that fixing bugs earlier is really cheaper, since detecting a bug in a specification might require much more effort than detecting the same bug in implementation, and detecting the bug is part of the cost of the bugfix. This is one of these things that everybody just believes because it sounds self-evident, but I've never seen a sound study that showed the effect. – LKM Jul 9 '19 at 9:06

I do have one set of data points for this - from an experience that sold me on unit tests.

Many moons ago I was a fresh graduate working on a large VB6 project and had occasion to write a large body of stored procedure code. Of the subsystem I was writing it made up about 1/4 of the whole code base - around 13,000 LOC out of 50K or so.

I wrote a set of unit tests for the stored procedures but unit testing VB6 UI code is not really feasible without tools like Rational Robot; at least it wasn't back then.

The statistics from QA on the piece were that about 40 or 50 defects were raised on the whole subsystem, of which two originated from the stored procedures. That's one defect per 6,500 lines of code vs. 1 per 1,000-1,200 or so across the whole piece. Bear in mind also, that about 2/3 of the VB6 code was boilerplate code for error handling and logging, identical across all of the procedures.

Without too much handwaving you can ascribe at least an order-of-magnitude improvement in defect rates to the unit testing.

| improve this answer | |

Your Answer

