<p>Just been looking at a similar issue (google: "testing scientific software") and came up with a few papers that may be of interest. These cover both the mundane coding errors and the bigger issues of knowing if the result is even right (depth of the Earth's mantle?) </p>

<p><a href="http://http.icsi.berkeley.edu/ftp/pub/speech/papers/wikipapers/cox_harris_testing_numerical_software.pdf" rel="nofollow noreferrer">http://http.icsi.berkeley.edu/ftp/pub/speech/papers/wikipapers/cox_harris_testing_numerical_software.pdf</a></p>

<p><a href="http://www.cs.ua.edu/~SECSE09/Presentations/09_Hook.pdf" rel="nofollow noreferrer">http://www.cs.ua.edu/~SECSE09/Presentations/09_Hook.pdf</a> (broken link; new link is <a href="http://www.se4science.org/workshops/secse09/Presentations/09_Hook.pdf" rel="nofollow noreferrer">http://www.se4science.org/workshops/secse09/Presentations/09_Hook.pdf</a>)</p>

<p><a href="http://www.associationforsoftwaretesting.org/?dl_name=DianeKellyRebeccaSanders_TheChallengeOfTestingScientificSoftware_paper.pdf" rel="nofollow noreferrer">http://www.associationforsoftwaretesting.org/?dl_name=DianeKellyRebeccaSanders_TheChallengeOfTestingScientificSoftware_paper.pdf</a></p>

<p>I thought the idea of mutation testing described in 09_Hook.pdf (see also matmute.sourceforge.net) is particularly interesting as it mimics the simple mistakes we all make. The hardest part is to learn to use statistical analysis for confidence levels, rather than single pass code reviews (man or machine).</p>

<p>The problem is not new. I'm sure I have an original copy of "How accurate is scientific software?" by Hatton et al Oct 1994, that even then showed how different implementations of the same theories (as algorithms) diverged rather rapidly (It's also ref 8 in Kelly &amp; Sanders paper)</p>

<p>--- (Oct 2019)
More recently <a href="https://arxiv.org/pdf/1804.01954.pdf" rel="nofollow noreferrer">Testing Scientific Software:  A Systematic Literature Review</a> </p>
How to test scientific software?
<p>I'm convinced that software testing indeed is very important, especially in science. However, over the last 6 years, I never have come across any scientific software project which was under regular tests (and most of them were not even version controlled). </p>

<p>Now I'm wondering how you deal with software tests for scientific codes (numerical computations). </p>

<p>From my point of view, standard unit tests often miss the point, since there is no exact result, so using <code>assert(a == b)</code> might prove a bit difficult due to "normal" numerical errors. </p>

<p>So I'm looking forward to reading your thoughts about this. </p>