Take the 2-minute tour ×

I was wondering how big companies of software developers check for bugs in their programs.

Do they just test it on several computers?

share|improve this question

migrated from stackoverflow.com Feb 12 '11 at 1:30

This question came from our site for professional and enthusiast programmers.

Releasing them. (judging by the buggy piles of .... that come out from the major institutions) –  Orbling Feb 12 '11 at 2:24
They have very aggressive sales and marketing campaign, outsource development to other countries and get away with whatever bugs they can ... until they "unexpectedly" lose a huge market share. –  Job Apr 3 '11 at 17:08
Why do you think it is any different for big companies than small ones? –  JohnFx Jan 23 '12 at 15:42
add comment

7 Answers

Here are some of the techniques that Google uses.

  1. Hire good developers who are likely to produce reliable code.
  2. Unit test heavily.
  3. Use code review.
  4. Set up a continuous build to catch integration issues.
  5. Have dedicated QA departments. Meaning both people testing, and automated programs (for instance using Selenium) that simulate end users.
  6. Have monitoring in production to catch evidence of things misbehaving.

I have ranked these in what I suspect is descending order of effectiveness in catching bugs.

share|improve this answer
While not a bad answer, you use lots of terminology like "QA", "unit test", and "continuous build" that are likely to be unknown to the sort of person who would ask a question like this. It would be better if you linked to or gave definitions. –  Gabe Feb 12 '11 at 1:41
@gabe: I've added pointers to the terminology used. –  btilly Feb 12 '11 at 2:16
+1 - Actually that is a pretty good order (1-> 6) of when it is BEST (ie. cheapest, in time and complexity) to catch bugs as well. –  Ozz Feb 12 '11 at 12:22
maybe also usability testing, before the ribbon 90% of word feature requests were for features word already had, users just couldn't find them –  jk. Jan 23 '12 at 13:20
@jk: whose statistic is that? citation please. –  JBRWilkinson Jan 24 '12 at 14:45
show 2 more comments

Bigger companies usually have whole entire Q/A departments which are responsible for testing code and making sure it works the way it is supposed to. Its usually just as you described-- a bunch of people testing lots of machines. Sometimes the tests are automated, sometimes they aren't. See Quality Assurance - Wikipedia

Many times, the developers themselves will find bugs during the development process. Also, customers are frequently the first to find a bug.

Smaller companies, like one I am currently working for, use the Agile Testing practice

share|improve this answer
Yup, and the QA folks make up test plans. –  Job Feb 12 '11 at 4:07
+1: This is exactly how we do it, the test team (which I'm on) writes the test plans and writes all of the test code with the exception the trivial unit tests (devs write those, some of which practice TDD but it's not mandated). We focus on acceptance, integration and regressions. When devs find bugs, they'll log it, fix it and we'll test it and write automation for it. –  Steve Evers Apr 3 '11 at 16:06
add comment

I would say its about the maturity of a company and not the size :) There are large companies that have poor development practices and small companies that are on the bleeding edge.

In general a mature development team will engage in the following activities to 1; minimize introducing new bugs to the system and 2; find bugs in the existing system.

Unit testing: These are 'mini drivers' for individual methods to ensure that a method does what it says it does. These are always automated tests.

Integration testing: These tests aim to check that a larger unit of functionality works within the system. This might involve testing database integration or integration with third party libraries. These are automated tests as well.

Acceptance testing: Acceptance tests are written to test user requirements. These usually just test the 'happy path'. In my team, these tests are designed to show that if the user uses the functionality as it was designed to be used, they will have no trouble. Can be manual or automated.

Functional testing: These tests are similar to acceptance tests, but they also test the 'unhappy path'. These tests mean to test the not so obvious scenarios. Can be manual or automated.

Regression testing: We use this term to do a 'full testing' of the system before its released to the customers. Manual or automated.

Gorilla testing: (Manual only). This is the kind of testing when very smart humans intentionally try to break the application.

Performance testing Aims to make sure that performance is acceptable and does not degrade over time. Usually automated.

Stability testing: These tests are designed to make sure the system remains stable over time. Automated.

Continuous Integration: This is a system that automatically checks out your code, compiles it and runs your automated tests. Your faster tests (unit, integration) will run each time a dev commits code. Some other ones run nightly (acceptance, functional) or weekly (performance, stability).

Code coverage reports: Shows you how much of your code is tested. Code that has no test coverage is more likely to break.

Different tools that analyze the code: These usually show where the code needs to be re-factored to make it less prone to potential bugs.

Pair programming: Two developers working together to deliver functionality. "A cohesive pair is better than the sum of its parts."

The most important to take away is: automation and continuous integration.

share|improve this answer
+1 for comment about maturity. –  ChrisFletcher Jan 23 '12 at 19:33
Disagree about maturity of the company - it is entirely possible that the head of software development cares about quality in a small/young company too, and that message is driven top-down. Maturity of the engineers, yes, that's possible. –  JBRWilkinson Jan 24 '12 at 14:47
@JBRWilkinson: I guess we can start talking about what it means for a company to be 'mature'. I did not mean to suggest it has to do with age, more like 'wisdom'. A startup can be mature/wise even though it is only a year or two old. –  c_maker Jan 25 '12 at 1:42
add comment

It depends on the company and on the products it develops.

First, many companies enforce coding practices like code reviews and mandatory linting (automated bug detection tools) to reduce the amount of errors going in to the repository. Many companies also adopted unit testing. This is the case where I work (Google). When code is checked in, the tests are run against everything, to make sure no new errors are introduced.

Second, many companies have QA departments that are responsible for validating behavior. This is particularly common in Finance (where mistakes can be expensive and validation rules are complex), but also exists in companies that sell products to users where recalls can be expensive (e.g., Intel, Microsoft, etc.).

Third, whenever possible companies do Dogfooding (have their own users use the product internally) and then release limited betas. Many errors are caught at this stage. For example, folks working at Microsoft use newer internal versions of Office and Windows and DevStudio than what you have outside. Then limited groups of users or contracted companies get to sample it. Similarly, at Google we use internal versions of GMail and Docs prior to release. Game companies organize open betas to test their products and the load on the servers,etc.

share|improve this answer
add comment

Of course the answer is "It dpends", but I will give a sample from my largest project so far, which had at peak time around 50 developers involved.

The basic setup: A backend software for processing large amounts of data with BizTalk.

The first line of defense are the unit tests. In our case these got executed daily for everything checked into source control and usually some of them were executed manually by the developer before check-in. The unit tests were mainly written by the developers but sometimes amended with additional tests by the testers.

Next step was a weekly Virtual PC build, where the testers ran a series of mainly automated end-to-end tests on the data-flow based on the specification documents for each component.

After that the same Virtual PC was enriched with some business data quite close to the real thing and tested again with some specific use cases.

Then the Virtual PC was put together with other system components (also mostly virtual) from other departments to an integration test cycle based on end-to-end testing from the user entering the data to the end of the data flow.

On another track the installation packets were tested by the systems provider to see if they installed correctly on a production-like environment and if they could be rolled back if something failed.

After the installation on the production-like environment we ran load and stress tests there to test the overall stability (not something to be taken lightly when you run on 10 BizTalk servers, 8 SQL Servers and a bunch of other specialized hardware like an XML accelerator and a dedicated Archive - all clustered of course).

When we were satisfied with all the tests, the code was put into production. You get a pretty big latency to fix bugs in the code (like 4-6 weeks for the whole test cycle), and it's expensive to do all these tests, but the overall stability was pretty good. In fact the best I have seen so far. Again that's quite important on a system that processes several million dollars worth each day. Your requirements may vary, but that's how we have done it and it worked.

share|improve this answer
add comment

The original question seems more conceptually generic, than most of the highly detailed answers that were provided.

Let´s look it from a higher level (less detailed). Software is developed to attend to specific needs from someone (person, company, whatever).

Those needs need to be mapped into the individual stories/requirements that would be lately (in a construction phase) be implemented in source code.

Having the stories/requirements well defined is essential for the Quality Assurance (QA) team (the actual software testers) to validate the final code, when executed, attends to the demands of those stories and requirements. So for that purposee, the QA team creates "testcases" to do that validation.

The code, once released to the QA team, will then be tested and bugs will be identified. Bugs of differente types and severities. Those bugs are tracked and developers get them assigned to finally get them fixed.

The usage of virtual machines, nowadays, allows for one tester to run different environments in one same real hardware. But sometimes you end up needing some computers dedicated to the QA phase.

I hope that helps you understanding (roughly) the overall process.

share|improve this answer
add comment

Well, I hate to be cynical, but with the number of open bugs in a certain 'device' operating system it seems that the bigger and richer the company, the more bugs they are capable of creating and delivering to the end user. If the software kinda works and looks cool then they just release it anyway. If the managers think its ready, then its ready. That's when the really nasty bugs start coming out of the woodwork, and the end users get to be guinea pigs. Ten or so years later, most of the bugs will have been worked out (and a few added for good measure) and the company will be ready to move on to the next big idea.

share|improve this answer
add comment

Your Answer


