Register Log in

Scientific Method / Science & Exploration

Keeping computers from ending science’s reproducibility

Software updates, changing databases, and poor documentation of computational …



Computation and the reproducibility crisis





Putting the reproducibility into computational code





Putting the reproducibility into computational code





Expand full story

46 Reader Comments

Post a reply
  1. Portability and Archivability - I didn't see either of these concepts mentioned. But these should be factored into the issue of scientific reproducibility of experiments. If at all possible, your peers should require no special platform-specific tools to reproduce your results. Likewise, you yourself should consider the likelihood that your own tools could become irrelevant in say 20 years. Plan accordingly.

    Published data (input and output) should be in a plain-text "export" format. If source code is included, it should be clearly written and commented in a high-order procedural language, e.g. - Fortran, C, Perl. And "black box" applications and functions should definitely be avoided. But if such tools are unavoidable, the algorithmic assumptions should at least be clearly stated in the paper.

    Unix and its built-in tool set will be around for a very long time, and should be your platform of choice. Avoid using object-oriented languages as your source code unless you eschew the libraries and methods that are the major rationale for OOP. Do your peers have those same libraries, and will you 20 years hence?

    You wouldn't dream of archiving your most precious documents in MS Word, or you tax records in TurboTax format. In 20 years, both types would be unreadable on a future platform. Why treat scientific reproducibility any differently?
    130 posts | registered
  2. I don't think this is a new problem. I just finished a 100 hour + MSc project reimplementing Lindenmayer Systems from the mid 70s and mid 80s. All my problems were to do with interpreting the rules of a 2L parser and at least 1/3 of my time was spent determining exactly what was meant by the rules they defined for the parser (by trialling the alternative interpretations of the rules until my results match theirs), which in some respects turned out to be wrongly described.

    This could have been cleared up by my sources publishing their code, which is not something they would have been keen on doing, as one of my main sources worked in the nascent film CGI industry and he was writing about the advances he made working at the company.

    What I really needed was the work to be clearly described in the first place but there will obviously always be problems. So having access to the code might have saved me time and would have been a nice option but then what if the algorithm is commercially sensitive and has been patented and your work is part of another commercial effort?

    This comment was edited by shazoom on January 23, 2010 22:42
    1 post | registered
  3. It's a good article, enough to get me off my butt and write something myself! I've put up a few thoughts for this issue in computational biology on my blog:

    (These aren't meant to directly "reply" to the article, but add additional points.)
    59 posts | registered
  4. As an undergrad, I thought all simulation was voodoo and witchcraft. Having taken a graduate course in the methods and worked on them for a little while, I can now appreciate how critical simulation is to the modern scientific method. A giant creative commons for code would be amazingly productive and useful. However, the biggest problem is understanding the limitations of the code, ie. how strongly the limiting assumptions affect the result
    525 posts | registered
  5. quote:
    Originally posted by xtofe:
    Reproducible Research has been around for a while; I suggest you take a look at following pages where you can find many interesting ideas, papers and news around reproducible research.

    [linkspam removed]

    I can guarantee you that dumping me a link to your company's website without telling me anything about it is going to make me put those links into the memory hole, and never, ever do business with you. it's also a violation of the ToS of this site to do it.

    Now, if you give me a description of what's behind each link, and a bit of information about why it's relevant to this thread, then it's not a ToS violation, and I may very well click through and do some reading, giving you precious hits and ad dollars.

    edit: removed quoted links.
    5507 posts | registered
  6. What many scientists wouldn't like, is a). after spending months on creating results and writing a paper, someone picking up their code, notice a few places to improve and publish a paper on it in 2 weeks, causing major pain in the behind, b). many people publish a number of papers during the development of an idea, and later following up on the idea, branching the idea, and so on, many of which publications wouldn't happen if everything would've been made openly available with the first paper, causing even more pain in the behind, since at many research institutes there are high publication quotas to meet, and loosing any number of those publications would be really painful, c). many scientists are not good coders and they only create proof of concept code in matlab or something else, and many of them would feel somewhat ashamed to show such code to the world, d). time is precious, many times a lot of work is being done in parallel and sometimes one is just happy to push a paper out of the pipeline and never think of it again, let alone prepare lots of additional marerial for making it public, e). in a lot of cases a good idea implies and is followed by a patenting process which is then followed by a publication, and this order can also hinder fully public disclosure.

    There are also a lot of other issues, I just don't have time for listing them all, I have papers to write :P
    107 posts | registered
  7. I absolutely have first-hand experience with this.

    But meh, I simply made a LiveCD (a remastered KNOPPIX) of my entire master's thesis. During my defense, I actually brought it up in qemu and ran through some scenarios, and delivered copies of the CD to my committee.

    It already has come in handy. The python accelerator I used (psyco) only works on 32-bit architectures (one of the first things I rewarded myself with after my defense was a new AMD64 machine... doh!). The 100x speedup was pretty critical in getting my simulation runs to finish in a reasonable amount of time.

    I also had the entire build environment, dev tools, libraries, even the latex build and all the fonts used to generate my paper packaged into it. So I can (and have) pretty much go back and run the entire thing over again if I found a small mistake or something like that.
    35 posts | registered
Post a reply

You must to comment.

Need to register for a new account?

If you don't have an account yet it's free and easy.