text: user136: Bug fixed
  This issue appears to have been fixed in latest Nextflow: https://github.com/nextflow-io/nextflow/issues/2693
  
  I've been using Nextflow for various workflows on a cluster using the SLURM scheduler. I've noticed that the 'parent' job does not shut down when the Nextflow workflow finishes successfully. I can cancel the job, but then Nextflow doesn't get a chance to clean things up and also doesn't write out the trace, report, etc. The parent job does end properly if the workflow ends with an error.
  Question: Why doesn't the parent Nextflow job end upon successful completion?
  I saw a post on the Nextflow GitHub where the original poster describes various acrobatics to get the parent job to exit cleanly, but this is something that Nextflow should handle.
  Here's my SLURM submission script that I submit using sbatch run_workflow.slurm
  run_workflow.slurm
  #!/bin/bash
  #SBATCH --time=60:15:00                             # Time limit for the job (REQUIRED).
  #SBATCH --job-name=NF_Parent_Identify_D&C_regions   # Job name
  #SBATCH --ntasks=1                                  # Number of cores for the job. Same as SBATCH
  #SBATCH --mem=5G                                    # Total memory requested
  #SBATCH --partition=<partition>                     # Partition/queue to run the job in. (REQUIRED
  #SBATCH -e slurm-%j.err                             # Error file for this job.
  #SBATCH -o slurm-%j.out                             # Output file for this job.
  #SBATCH -A <account>                                # Project allocation account name (REQUIRED)
  
  module load ccs/java/jdk1.8.0_202
  nextflow run identify_dark_and_camouflaged_regions.nf \
      -with-report \
      -with-trace \
      -with-dag idcr-workflow.png \
      -with-timeline \
      -resume
  
  Update 1
  I'm adding example Nextflow output from a different workflow (I don't have the SLURM output from the one I mention above) where the workflow has completed, but the parent job is still running. In this case, there were errors, but I added errorStrategy 'ignore' to all of the processes listed:
  executor >  slurm (876)
  [06/c5672b] process > samtools_csort_proc (A-CUHS... [100%] 304 of 304, faile...
  [45/26a046] process > Lsamtools_csort_proc (A-CUH... [100%] 304 of 304, faile...
  [b5/0f2d7c] process > sambamba_csort_proc (A-CUHS... [100%] 304 of 304, faile...
  
  So, regardless of whether the workflow completed successfully, Nextflow has nothing left to do, but it's still running.
  Update 2 (adding partial .nextflow.log file)
  Per @Steve's point, I checked the .nextflow.log file and am including part of the file, here. In this case, I don't think Nextflow has any tasks related to publishDir because I'm not using that in this workflow. The job that's still running is from the Nextflow workflow in my GitHub repository for my answer posted in my related question Samtools sort: most efficient memory and thread settings for many samples on a cluster. Whether it's reproducible, I haven't checked. Not sure how common this issue is.
  In either case, my interpretation of the .nextflow.log file below is that Nextflow thinks it's hung, too. I just don't know why, or how to exit cleanly.
  Partial view of .nextflow.log
  ...
  
  Feb-28 05:29:04.607 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor slurm > tasks to be completed: 1 -- submitted tasks are shown below
  ~> TaskHandler[jobId: 662274; id: 876; name: sambamba_csort_proc (A-CUHS-CU003128-BL-COL-49696BL1.unsorted;CPUs:2;MEM_PER_THREAD:100GB); status: RUNNING; exit: -; error: -; workDir: /path/to/Samtools_sort_optimization_test/SORT_COMPARISON/work/b5/0f2d7c5d7f81a6e62dedb08bf6694a started: 1646038174334; exited: -; ]
  Feb-28 05:30:54.613 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[jobId: 662274; id: 876; name: sambamba_csort_proc (A-CUHS-CU003128-BL-COL-49696BL1.unsorted;CPUs:2;MEM_PER_THREAD:100GB); status: COMPLETED; exit: 0; error: -; workDir: /path/to/Samtools_sort_optimization_test/SORT_COMPARISON/work/b5/0f2d7c5d7f81a6e62dedb08bf6694a started: 1646038174334; exited: 2022-02-28T10:30:53.516015Z; ]
  Feb-28 05:34:04.641 [Task monitor] DEBUG n.processor.TaskPollingMonitor - No more task to compute -- Execution may be stalled
  Feb-28 05:39:04.647 [Task monitor] DEBUG n.processor.TaskPollingMonitor - No more task to compute -- Execution may be stalled
  Feb-28 05:44:04.653 [Task monitor] DEBUG n.processor.TaskPollingMonitor - No more task to compute -- Execution may be stalled
  Feb-28 05:49:04.658 [Task monitor] DEBUG n.processor.TaskPollingMonitor - No more task to compute -- Execution may be stalled
  Feb-28 05:54:04.662 [Task monitor] DEBUG n.processor.TaskPollingMonitor - No more task to compute -- Execution may be stalled
  
  < above line repeats hundreds of times >
  
  ...
  
  
  user10775: How do you know your workflow actually finishes successfully? One way to check the workflow termination status is to use the workflow.onComplete and workflow.onError event handlers. Consider resuming your workflow with something like the following appended to your script:
  workflow.onComplete {
  
    log.info "Workflow completed at: ${workflow.complete}"
    log.info "Time taken: ${workflow.duration}"
    log.info "Execution status: ${workflow.success ? 'success' : 'failed'}"
  }
  
  workflow.onError {
  
    log.info "Execution halted: ${workflow.errorMessage}"
  }
  
  Note that the workflow event handlers may also be defined inside your nextflow.config if preferred.
  
original_text: 136: <h2>Bug fixed</h2>
  <p>This issue appears to have been fixed in latest <code>Nextflow</code>: <a href="https://github.com/nextflow-io/nextflow/issues/2693" rel="nofollow noreferrer">https://github.com/nextflow-io/nextflow/issues/2693</a></p>
  <hr />
  <p>I've been using <code>Nextflow</code> for various workflows on a cluster using the <code>SLURM</code> scheduler. I've noticed that the 'parent' job does not shut down when the <code>Nextflow</code> workflow finishes successfully. I can cancel the job, but then <code>Nextflow</code> doesn't get a chance to clean things up and also doesn't write out the <code>trace</code>, <code>report</code>, etc. The parent job does end properly if the workflow ends with an error.</p>
  <p><strong>Question</strong>: Why doesn't the parent <code>Nextflow</code> job end upon successful completion?</p>
  <p>I saw <a href="https://github.com/nextflow-io/nextflow/issues/968" rel="nofollow noreferrer">a post on the <code>Nextflow</code> GitHub</a> where the original poster describes various acrobatics to get the parent job to exit cleanly, but this is something that <code>Nextflow</code> should handle.</p>
  <p>Here's my <code>SLURM</code> submission script that I submit using <code>sbatch run_workflow.slurm</code></p>
  <p><strong>run_workflow.slurm</strong></p>
  <pre><code>#!/bin/bash
  #SBATCH --time=60:15:00                             # Time limit for the job (REQUIRED).
  #SBATCH --job-name=NF_Parent_Identify_D&amp;C_regions   # Job name
  #SBATCH --ntasks=1                                  # Number of cores for the job. Same as SBATCH
  #SBATCH --mem=5G                                    # Total memory requested
  #SBATCH --partition=&lt;partition&gt;                     # Partition/queue to run the job in. (REQUIRED
  #SBATCH -e slurm-%j.err                             # Error file for this job.
  #SBATCH -o slurm-%j.out                             # Output file for this job.
  #SBATCH -A &lt;account&gt;                                # Project allocation account name (REQUIRED)
  
  module load ccs/java/jdk1.8.0_202
  nextflow run identify_dark_and_camouflaged_regions.nf \
      -with-report \
      -with-trace \
      -with-dag idcr-workflow.png \
      -with-timeline \
      -resume
  </code></pre>
  <h1>Update 1</h1>
  <p>I'm adding example <code>Nextflow</code> output from a different workflow (I don't have the <code>SLURM</code> output from the one I mention above) where the workflow has completed, but the parent job is still running. In this case, there were errors, but I added <code>errorStrategy 'ignore'</code> to all of the processes listed:</p>
  <pre><code>executor &gt;  slurm (876)
  [06/c5672b] process &gt; samtools_csort_proc (A-CUHS... [100%] 304 of 304, faile...
  [45/26a046] process &gt; Lsamtools_csort_proc (A-CUH... [100%] 304 of 304, faile...
  [b5/0f2d7c] process &gt; sambamba_csort_proc (A-CUHS... [100%] 304 of 304, faile...
  </code></pre>
  <p>So, regardless of whether the workflow completed <code>successfully</code>, <code>Nextflow</code> has nothing left to do, but it's still running.</p>
  <h1>Update 2 (adding partial <code>.nextflow.log</code> file)</h1>
  <p>Per @Steve's point, I checked the <code>.nextflow.log</code> file and am including part of the file, here. In this case, I don't think <code>Nextflow</code> has any tasks related to <code>publishDir</code> because I'm not using that in this workflow. The job that's still running is from the <code>Nextflow</code> workflow in my GitHub repository for my answer posted in my related question <a href="https://bioinformatics.stackexchange.com/questions/18538/samtools-sort-most-efficient-memory-and-thread-settings-for-many-samples-on-a-c">Samtools sort: most efficient memory and thread settings for many samples on a cluster</a>. Whether it's reproducible, I haven't checked. Not sure how common this issue is.</p>
  <p>In either case, my interpretation of the <code>.nextflow.log</code> file below is that <code>Nextflow</code> thinks it's hung, too. I just don't know why, or how to exit cleanly.</p>
  <p><strong>Partial view of <code>.nextflow.log</code></strong></p>
  <pre><code>...
  
  Feb-28 05:29:04.607 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor slurm &gt; tasks to be completed: 1 -- submitted tasks are shown below
  ~&gt; TaskHandler[jobId: 662274; id: 876; name: sambamba_csort_proc (A-CUHS-CU003128-BL-COL-49696BL1.unsorted;CPUs:2;MEM_PER_THREAD:100GB); status: RUNNING; exit: -; error: -; workDir: /path/to/Samtools_sort_optimization_test/SORT_COMPARISON/work/b5/0f2d7c5d7f81a6e62dedb08bf6694a started: 1646038174334; exited: -; ]
  Feb-28 05:30:54.613 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed &gt; TaskHandler[jobId: 662274; id: 876; name: sambamba_csort_proc (A-CUHS-CU003128-BL-COL-49696BL1.unsorted;CPUs:2;MEM_PER_THREAD:100GB); status: COMPLETED; exit: 0; error: -; workDir: /path/to/Samtools_sort_optimization_test/SORT_COMPARISON/work/b5/0f2d7c5d7f81a6e62dedb08bf6694a started: 1646038174334; exited: 2022-02-28T10:30:53.516015Z; ]
  Feb-28 05:34:04.641 [Task monitor] DEBUG n.processor.TaskPollingMonitor - No more task to compute -- Execution may be stalled
  Feb-28 05:39:04.647 [Task monitor] DEBUG n.processor.TaskPollingMonitor - No more task to compute -- Execution may be stalled
  Feb-28 05:44:04.653 [Task monitor] DEBUG n.processor.TaskPollingMonitor - No more task to compute -- Execution may be stalled
  Feb-28 05:49:04.658 [Task monitor] DEBUG n.processor.TaskPollingMonitor - No more task to compute -- Execution may be stalled
  Feb-28 05:54:04.662 [Task monitor] DEBUG n.processor.TaskPollingMonitor - No more task to compute -- Execution may be stalled
  
  &lt; above line repeats hundreds of times &gt;
  
  ...
  </code></pre>
   10775: <p>How do you know your workflow actually finishes successfully? One way to check the workflow termination status is to use the <a href="https://www.nextflow.io/docs/latest/metadata.html#completion-handler" rel="nofollow noreferrer"><code>workflow.onComplete</code></a> and <a href="https://www.nextflow.io/docs/latest/metadata.html#error-handler" rel="nofollow noreferrer"><code>workflow.onError</code></a> event handlers. Consider resuming your workflow with something like the following appended to your script:</p>
  <pre><code>workflow.onComplete {
  
    log.info &quot;Workflow completed at: <span class="math-container">${workflow.complete}"
    log.info "Time taken: $</span>{workflow.duration}&quot;
    log.info &quot;Execution status: ${workflow.success ? 'success' : 'failed'}&quot;
  }
  
  workflow.onError {
  
    log.info &quot;Execution halted: ${workflow.errorMessage}&quot;
  }
  </code></pre>
  <p>Note that the workflow event handlers may also be defined <a href="https://www.nextflow.io/docs/latest/metadata.html#decoupling-metadata" rel="nofollow noreferrer">inside</a> your <code>nextflow.config</code> if preferred.</p>
  