perhaps because of ingrained cultural beliefs about the infallibility of computation 
 , people show a level of trust in computed outputs that is completely at odds with the reality that nearly zero provably error - free computer programs have ever been written 
 2 , 
 3 .
it has been estimated that the industry average rate of programming errors is  about 15  50 errors per 1000 lines of delivered code  
 .
that estimate describes the work of professional software engineers  not of the graduate students who write most scientific data analysis programs , usually without the benefit of training in software engineering and testing 
 5 , 
 6 .
the recent increase in attention to such training is a welcome and essential development 
 7 
 11 .
nonetheless , even the most careful software engineering practices in industry rarely achieve an error rate better than 1 per 1000 lines .
since software programs commonly have many thousands of lines of code ( 
 table 1 ) , it follows that many defects remain in delivered code  even after all testing and debugging is complete .
defects occur not only in the top - level program being run but also in compilers , system libraries , and even firmware and hardware  and errors in such underlying components are extremely difficult to detect 
 .
of course , not every error in a program will affect the outcome of a specific analysis .
for a simple single - purpose program , it is entirely possible that every line executes on every run . in general
, however , the code path taken for a given run of a program executes only a subset of the lines in it , because there may be command - line options that enable or disable certain features , blocks of code that execute conditionally depending on the input data , etc .
furthermore , even if an erroneous line executes , it may not in fact manifest the error ( i.e. , it may give the correct output for some inputs but not others ) . finally
: many errors may cause a program to simply crash or to report an obviously implausible result , but we are really only concerned with errors that propagate downstream and are reported . in combination , then , we can estimate the number of errors that actually affect the result of a single run of a program , as follows : 
 number of errors per program execution = 
 total lines of code ( loc ) 
 * proportion executed 
 * probability of error per line 
 * probability that the error 
 meaningfully affects the result 
 * probability that an erroneous result 
 appears plausible to the scientist . 
  for these purposes , using a formula to compute a value in excel counts as a  line of code  , and a spreadsheet as a whole counts as a 
programso many scientists who may not consider themselves coders may still suffer from bugs 
 .
all of these values may vary widely depending on the field and the source of the software . for a typical analysis in bioinformatics
, i ll speculate at some plausible values : 
 100,000 total loc ( neglecting trusted components such as the linux kernel).20% executed10 errors per 1000 lines10% chance that a given error meaningfully changes the outcome10% chance that a consequent erroneous result is plausible 
  100,000 total loc ( neglecting trusted components such as the linux kernel ) .
10 errors per 1000 lines 10% chance that a given error meaningfully changes the outcome 10% chance that a consequent erroneous result is plausible so , we expect that two errors changed the output of this program run , so 
 the probability of a wrong output is effectively 100% .
let s imagine a more optimistic scenario , in which we write a simple , short program , and we go to great lengths to test and debug it .
in such a case , any output that is produced is in fact more likely to be plausible , because bugs producing implausible outputs are more likely to have been eliminated in testing . 1 error per 1000 lines 10% chance that a given error meaningfully changes the outcome 50% chance that a consequent erroneous result is plausible here 
 the probability of a wrong output is 5% .
the factors going into the above estimates are rank speculation , and the conclusion varies widely depending on the guessed values . measuring such values rigorously in different contexts would be valuable but also tremendously difficult .
regardless , it is sobering that some plausible values indicate total wrongness all the time , and that even conservative values suggest that an appreciable proportion of results are erroneous due to software defects  above and beyond those that are erroneous for more widely appreciated reasons .
all of these values may vary widely depending on the field and the source of the software . for a typical analysis in bioinformatics
, i ll speculate at some plausible values : 
 100,000 total loc ( neglecting trusted components such as the linux kernel).20% executed10 errors per 1000 lines10% chance that a given error meaningfully changes the outcome10% chance that a consequent erroneous result is plausible 
  100,000 total loc ( neglecting trusted components such as the linux kernel ) .
10 errors per 1000 lines 10% chance that a given error meaningfully changes the outcome 10% chance that a consequent erroneous result is plausible so , we expect that two errors changed the output of this program run , so 
 the probability of a wrong output is effectively 100% .
let s imagine a more optimistic scenario , in which we write a simple , short program , and we go to great lengths to test and debug it .
in such a case , any output that is produced is in fact more likely to be plausible , because bugs producing implausible outputs are more likely to have been eliminated in testing .
1 error per 1000 lines 10% chance that a given error meaningfully changes the outcome 50% chance that a consequent erroneous result is plausible here 
 the probability of a wrong output is 5% . the factors going into the above estimates are rank speculation , and the conclusion varies widely depending on the guessed values .
regardless , it is sobering that some plausible values indicate total wrongness all the time , and that even conservative values suggest that an appreciable proportion of results are erroneous due to software defects  above and beyond those that are erroneous for more widely appreciated reasons .
a response to concerns about software quality that i have heard frequently  particularly from wet - lab biologists  is that errors may occur but have little impact on the outcome .
this may be because only a few data points are affected , or because values are altered by a small amount ( so the error is  in the noise  ) .
the above estimates account for this by including terms for  meaningful changes to the result  and  the outcome is plausible  .
nonetheless , in the context of physical experiments , it is tempting to believe that small errors tend to reduce 
 precision but have less effect on 
 accuracy  i.e .
if the concentration of some reagent is a bit off then the results will also be just a bit off , but not completely unrelated to the correct result .
we can not apply our physical intuitions , because software is profoundly brittle :  small  bugs commonly have unbounded error propagation . a sign error , a missing semicolon , an off - by - one error in matching up two columns of data , etc
it is rare that a software bug would alter a small proportion of the data by a small amount .
more likely , it systematically alters every data point , or occurs in some downstream aggregate step with effectively global consequences . 

in general , software errors produce outcomes that are 
 inaccurate , not merely 
 imprecise .
bugs that produce program crashes or completely implausible results are more likely to be discovered during development , before a program becomes  delivered code  ( the state of code on which the above errors - per - line estimates are based ) . consequently , published scientific code often has the property that nearly every possible output is plausible .
when the code is a black box , situations such as these may easily produce outputs that are simply accepted at face value : 
 an indexing off - by - one error associates the wrong pairs of x s and y s 
 .a correlation is found between two variables where in fact none exists , or vice versa.a sequence aligner reports the 
best  match to a sequence in a genome , but actually provides a lower - scoring match.a protein structure produced from x - ray crystallography is wrong , but it still looks like a protein 
 .a classifier reports that only 60% of the data points are classifiable , when in fact 90% of the points should have been classified ( and worse , there is a bias in which points were classified , so those 60% are not representative).all measured values are multiplied by a constant factor , but remain within a reasonable range . 
 
an indexing off - by - one error associates the wrong pairs of x s and y s 
 .
a correlation is found between two variables where in fact none exists , or vice versa .
best  match to a sequence in a genome , but actually provides a lower - scoring match .
a protein structure produced from x - ray crystallography is wrong , but it still looks like a protein 
 .
a classifier reports that only 60% of the data points are classifiable , when in fact 90% of the points should have been classified ( and worse , there is a bias in which points were classified , so those 60% are not representative ) .
all measured values are multiplied by a constant factor , but remain within a reasonable range .
a software error may produce a spurious result that appears significant , or may mask a significant result . if the error occurs early in an analysis pipeline , then it may be considered a form of measurement error ( i.e. , if it systematically or randomly alters the values of individual measurements ) , and so may be taken into account by common statistical methods .
however : typically the computed portion of a study comes after data collection , so its contribution to wrongness may easily be independent of sample size , replication of earlier steps , and other techniques for improving significance .
for instance , a software error may occur near the end of the pipeline , e.g. in the computation of a significance value or of other statistics , or in the preparation of summary tables and plots .
the diversity of the types and magnitudes of errors that may occur 
 16 
 19 makes it difficult to make a general statement about the effects of such errors on apparent significance .
however it seems clear that , a substantial proportion of the time ( based on the above scenarios , anywhere from 5% to 100% ) , a result is simply wrong  rendering moot any claims about its significance .
all hope is not lost ; we must simply take the opportunity to use technology to bring about a new era of collaborative , reproducible science 
 20 
 22 .
open availability of all data and source code used to produce scientific results is an incontestable foundation 
 23 
 27 . beyond that
, we must redouble our commitment to replicating and reproducing results , and in particular we must insist that a result can be trusted only when it has been observed on multiple occasions using 
 completely different software packages and methods .
this in turn requires a flexible and open system for describing and sharing computational workflows 
 .
projects such as galaxy 
 , kepler 
 , and taverna 
  have made inroads towards this goal , but much more work is needed to provide widespread access to comprehensive provenance of computational results . perhaps ironically , a shared workflow system must itself qualify as a  trusted componentlike the linux kernel  in order to provide a neutral platform for comparisons , and so must be held to the very highest standards of software quality . crucially , any shared workflow system must be widely used to be effective , and gaining adoption is more a sociological and economic problem than a technical one 
 . 

 errors in scientific results due to software bugs are not limited to a few high - profile cases that lead to retractions and are widely reported . 
 here i estimate that in fact most scientific results are probably wrong if data have passed through a computer , and that these errors may remain largely undetected . 
 the opportunities for both subtle and profound errors in software and data management are boundless , yet they remain surprisingly underappreciated . 
Computational results are particularly prone to misplaced trust
How frequently are published results wrong due to software bugs?
Scenario 1: A typical medium-scale bioinformatics analysis
Scenario 2: A small focused analysis, rigorously executed
Software is exceptionally brittle
Many erroneous results are plausible
Software errors and statistical significance are orthogonal issues
What can be done?