Update: July 26, 2019

Why does call caching miss even when inputs are identical?

(Version 36)

I'm stuck on a problem where call caching reproducibly works early in one of my workflows, and then consistently fails to find a hit for a specific task. The hashes in the call info from the metadata show that it's one particular input file whose hash always changes. I don't understand how this is possible considering that the file contents are identical between runs - it's just a symlinked output of a previous task that was successfully retrieved from the cache. This occurs with multiple configurations of duplication-strategy and hashing-strategy, and so far no matter what I try it always changes the hash for this file. Is there some subtlety to how repeated calls are detected and I'm screwing it up?


  • natecholsnatechols Member

    PS. I'm using MySQL, and running Cromwell in server mode.

  • ChrisLChrisL Cambridge, MAMember, Broadie, Moderator, Dev admin

    Hey @natechols, I'd suggest having a look at the call cache diff endpoint, that's how Cromwell can tell you what's preventing it from caching.

  • natecholsnatechols Member

    The cache diff endpoint told me what I already knew: the hashes for one of my inputs do not match. What I don't understand is how this is possible when it's just symlinks to the same file. diff, md5sum, and readlink -f all report the same thing; only Cromwell thinks they're different.

  • ChrisLChrisL Cambridge, MAMember, Broadie, Moderator, Dev admin

    When you look at the cache miss info, it should tell you what Cromwell thinks the hash for the file input was when the job ran the first time, and when it ran the second time. Presumably the "when it ran the second time" hash is correct and the "when it ran the first time" hash is wrong (ie if you run md5sum against the file manually)?

    If so, could anything have changed the value of the contents of that file between the first run and the second run, to cause the hash to change? And if not, if you try to run the workflow again, does the hash change again (ie is the "when it ran the third time" hash different from the "when it ran the second time" hash)?

  • natecholsnatechols Member

    Yes, every time it runs I get a different hash. It appeared to be inconsistent with md5sum too. It's definitely possible the file changed without me knowing about it - I'll check up on that. (I ended up working around the problem in the short term by removing one of the offending tasks that wasn't immediately necessary, but I'd like to make sure I understand what went wrong in case this happens again.)

Sign In or Register to comment.