text: user15166: I'm trying to adapt an existing workflow to make it more robust to failures. Below is a simplified example of the workflow.
  workflow main {
     File filelist="list_of_input_files.txt"
     Array[String] files=read_lines(filelist)
     scatter(file in files) {
        call task1 { input: in=file }
        if(defined(task1.out)) { call task2 { input: in=task1.out } }
        if(defined(task2.out)) { call task2 { input: in=task2.out }
     }
  }
  
  task task1 {
     File in
     String out1 = basename(in,'.txt') + '_1.txt'
     command { if(...) { ./script1.sh ${in} > ${out1} } }
     output { File? out = out1 }
     runtime { ... }
  }
  
  task task2 {
     File in
     String out2 = basename(in,'.txt') + '_2.txt'
     command { if(...) {./script2.sh ${in} > ${out2} } }
     output { File? out = out2 }
     runtime { ... }
  }
  
  task task3 {
     File in
     String out3 = basename(in,'.txt') + '_3.txt'
     command { if(...) {./script3.sh ${in} > ${out3} } }
     output { File? out = out3 }
     runtime {...}
  }
  
  with my list_of_input_files.txt looking like
  firstfile.txt
  secondfile.txt
  thirdfile.txt
  ...
  
  Obviously this is oversimplified, but I am trying to catch failed blocks by making the output of each task optional (using File?) and then having an if(defined(taskX.out)) condition to call the next task. My reasoning is that if something fails (represented by the if(...) within the command block of each task), then the out file will never be created in that task and so won't exist as input for the next task and thus if(defined(taskX.out)) will resolve as false and skip the next task (and therefore all the subsequent tasks).
  With current testing, the first file in the list satisfies the if(...) conditions of all the tasks and succeeds in giving me the final output file firstfile_1_2_3.txt. However, I am getting a Required file output 'secondfile_1.txt' does not exist' error message because it fails the if(...) condition in task1 and so the output file is never created.
  From what I've read in the WDL v1.1 spec, optional File outputs are allowed and are undefined if the files they point to don't exist. However, after much confused googling, I've realised it's probably because I'm using Cromwell, which is stuck with WDL v1.0.
  My question is whether this should be working in WDL v1.0 and, if not, is my logic correct and should this (psuedo-)code work with WDL v1.1? Is there a way to achieve what I'm trying to do using pure WDL v1.0?
  
  user6955: Optional outputs should work in WDL 1.0. Here is an example that runs without issue through cromwell (version 74).
  Note that for the second call in the scatter the input file needs to be converted to from an optional to an non optional value. Since this only gets run if this value is defined, this conversion can safely be done using the select_first function (which returns the first defined value in a given list of optionals).
  version 1.0
  
  workflow test {
      input {
          File filelist = "files.txt"
      }
      Array[String] files = read_lines(filelist)
  
      scatter (file in  files) {
           call blah as b1 {input: in=file}
           if (defined(b1.out)) {
               call blah as b2 {input: in=select_first([b1.out])}
           }
      }
  }
  
  
  task blah {
      input {
          File in
      }
      String outPath = basename(in,'.txt') + '_blah.txt'
  
      command {
          if $(grep -q "yay" ~{in})
            then
               echo yay > ~{outPath}
          fi
      }
  
      output {
          File? out = outPath
      }
  }
  
  files.txt looks like this:
  test1.txt
  test2.txt
  
  With test1.txt containing yay and test2.txt containing nay.
  
original_text: 15166: <p>I'm trying to adapt an existing workflow to make it more robust to failures. Below is a simplified example of the workflow.</p>
  <pre><code>workflow main {
     File filelist=&quot;list_of_input_files.txt&quot;
     Array[String] files=read_lines(filelist)
     scatter(file in files) {
        call task1 { input: in=file }
        if(defined(task1.out)) { call task2 { input: in=task1.out } }
        if(defined(task2.out)) { call task2 { input: in=task2.out }
     }
  }
  
  task task1 {
     File in
     String out1 = basename(in,'.txt') + '_1.txt'
     command { if(...) { ./script1.sh <span class="math-container">${in} &gt; $</span>{out1} } }
     output { File? out = out1 }
     runtime { ... }
  }
  
  task task2 {
     File in
     String out2 = basename(in,'.txt') + '_2.txt'
     command { if(...) {./script2.sh <span class="math-container">${in} &gt; $</span>{out2} } }
     output { File? out = out2 }
     runtime { ... }
  }
  
  task task3 {
     File in
     String out3 = basename(in,'.txt') + '_3.txt'
     command { if(...) {./script3.sh <span class="math-container">${in} &gt; $</span>{out3} } }
     output { File? out = out3 }
     runtime {...}
  }
  </code></pre>
  <p>with my list_of_input_files.txt looking like</p>
  <pre><code>firstfile.txt
  secondfile.txt
  thirdfile.txt
  ...
  </code></pre>
  <p>Obviously this is oversimplified, but I am trying to catch failed blocks by making the output of each task optional (using <code>File?</code>) and then having an <code>if(defined(taskX.out))</code> condition to call the next task. My reasoning is that if something fails (represented by the <code>if(...)</code> within the command block of each task), then the <code>out</code> file will never be created in that task and so won't exist as input for the next task and thus <code>if(defined(taskX.out))</code> will resolve as false and skip the next task (and therefore all the subsequent tasks).</p>
  <p>With current testing, the first file in the list satisfies the <code>if(...)</code> conditions of all the tasks and succeeds in giving me the final output file <code>firstfile_1_2_3.txt</code>. However, I am getting a <code>Required file output 'secondfile_1.txt' does not exist'</code> error message because it fails the <code>if(...)</code> condition in <code>task1</code> and so the output file is never created.</p>
  <p>From what I've read in the <a href="https://github.com/openwdl/wdl/blob/main/versions/development/SPEC.md#declarations" rel="nofollow noreferrer">WDL v1.1 spec</a>, optional <code>File</code> outputs are allowed and <a href="https://github.com/openwdl/wdl/blob/main/versions/development/SPEC.md#file-directory-and-optional-outputs" rel="nofollow noreferrer">are undefined if the files they point to don't exist</a>. However, after much confused googling, I've realised it's probably because I'm using <a href="https://github.com/broadinstitute/cromwell" rel="nofollow noreferrer">Cromwell</a>, which is stuck with WDL v1.0.</p>
  <p>My question is whether this should be working in WDL v1.0 and, if not, is my logic correct and should this (psuedo-)code work with WDL v1.1? Is there a way to achieve what I'm trying to do using pure WDL v1.0?</p>
   6955: <p>Optional outputs should work in WDL 1.0. Here is an example that runs without issue through cromwell (version 74).</p>
  <p>Note that for the second call in the scatter the input file needs to be converted to from an optional to an non optional value. Since this only gets run if this value is defined, this conversion can safely be done using the <code>select_first</code> function (which returns the first defined value in a given list of optionals).</p>
  <pre><code>version 1.0
  
  workflow test {
      input {
          File filelist = &quot;files.txt&quot;
      }
      Array[String] files = read_lines(filelist)
  
      scatter (file in  files) {
           call blah as b1 {input: in=file}
           if (defined(b1.out)) {
               call blah as b2 {input: in=select_first([b1.out])}
           }
      }
  }
  
  
  task blah {
      input {
          File in
      }
      String outPath = basename(in,'.txt') + '_blah.txt'
  
      command {
          if $(grep -q &quot;yay&quot; ~{in})
            then
               echo yay &gt; ~{outPath}
          fi
      }
  
      output {
          File? out = outPath
      }
  }
  </code></pre>
  <p>files.txt looks like this:</p>
  <pre><code>test1.txt
  test2.txt
  </code></pre>
  <p>With test1.txt containing <code>yay</code> and test2.txt containing <code>nay</code>.</p>
  