text: user2005: I am running a script file on a cluster with slurm through sbatch. Locally it runs perfectly well on 2 different linux versions (linux mint, ubuntu 12.04), however, it returns -9 error code while running on a cluster which is CentOS 7.3. When we are using Pypiper and run the whole .py script file, it fails when it starts running trimmomatic program
  http://www.usadellab.org/cms/?page=trimmomatic
  However, when running just trimmomatic program step by itself it does not fail and finishes successfully. We have a suspicion that something may be wrong with how Pypiper runs on the cluster, however, feeling rather stuck.
  Any suggestions would be greatly appreciated. 
  Here is the error that is printed out:
  ILLUMINACLIP: Using 2 prefix pairs, 22 forward/reverse sequences, 0 
  forward only sequences, 0 reverse only sequences
  </pre>
  Process 109391 returned: (-9). Elapsed: 0:00:51. Peak memory: (Process: 
  5.211GB; Pipeline: 0.0GB)
  
  ### Pipeline failed at:  (12-14 15:50:47) elapsed:0:23:09 _TIME_
  
  Total time: 0:23:09
  
  Change status from running to failed
  Traceback (most recent call last):
    File "...../salmon_rna_seq.py", line 344, in <module>
  sys.exit(main())
  File "...../salmon_rna_seq.py", line 304, in main
  pipeline.process()      
  File "...../salmon_rna_seq.py", line 277, in process
  self.trim_reads()      
  File "...../salmon_rna_seq.py", line 195, in trim_reads
  self.pipe_manager.run(cmd, self.sample_files['r1']['fastq']['trimmed_paired'])
  File "...../pypiper.py", line 508, in run
  process_return_code, local_maxmem = self.callprint(cmd, shell, nofail, container)  # Run command
  File "...../pypiper.py", line 691, in callprint
  self.fail_pipeline(e)
  File "...../pypiper.py", line 1111, in fail_pipeline
  raise e
  OSError: Subprocess returned nonzero result.
  
  Pypiper terminating spawned child process 108475...
  child process terminated
  slurmstepd: error: Exceeded step memory limit at some point.
  
  The last line may be informative. Although, we are still not sure what we could possibly do.
  
  user77: Slurm is killing the process because it's using more memory per core than the default allocation (that's what slurmstepd: error: Exceeded step memory limit at some point means). I've never used pyPiper (like seemingly most people, I use snakemake), but presuming that you have it running either srun or sbatch at some point then specify something like --mem-per-cpu 10G.
  
original_text: 2005: <p>I am running a script file on a cluster with <code>slurm</code> through <code>sbatch</code>. Locally it runs perfectly well on 2 different linux versions (<code>linux mint</code>, <code>ubuntu 12.04</code>), however, it returns <code>-9</code> error code while running on a cluster which is <code>CentOS 7.3</code>. When we are using <code>Pypiper</code> and run the whole <code>.py</code> script file, it fails when it starts running <code>trimmomatic</code> program</p>
  
  <p><a href="http://www.usadellab.org/cms/?page=trimmomatic" rel="nofollow noreferrer">http://www.usadellab.org/cms/?page=trimmomatic</a></p>
  
  <p>However, when running just <code>trimmomatic</code> program step by itself it does not fail and finishes successfully. We have a suspicion that something may be wrong with how <code>Pypiper</code> runs on the cluster, however, feeling rather stuck.
  Any suggestions would be greatly appreciated. </p>
  
  <p>Here is the error that is printed out:</p>
  
  <pre><code>ILLUMINACLIP: Using 2 prefix pairs, 22 forward/reverse sequences, 0 
  forward only sequences, 0 reverse only sequences
  &lt;/pre&gt;
  Process 109391 returned: (-9). Elapsed: 0:00:51. Peak memory: (Process: 
  5.211GB; Pipeline: 0.0GB)
  
  ### Pipeline failed at:  (12-14 15:50:47) elapsed:0:23:09 _TIME_
  
  Total time: 0:23:09
  
  Change status from running to failed
  Traceback (most recent call last):
    File "...../salmon_rna_seq.py", line 344, in &lt;module&gt;
  sys.exit(main())
  File "...../salmon_rna_seq.py", line 304, in main
  pipeline.process()      
  File "...../salmon_rna_seq.py", line 277, in process
  self.trim_reads()      
  File "...../salmon_rna_seq.py", line 195, in trim_reads
  self.pipe_manager.run(cmd, self.sample_files['r1']['fastq']['trimmed_paired'])
  File "...../pypiper.py", line 508, in run
  process_return_code, local_maxmem = self.callprint(cmd, shell, nofail, container)  # Run command
  File "...../pypiper.py", line 691, in callprint
  self.fail_pipeline(e)
  File "...../pypiper.py", line 1111, in fail_pipeline
  raise e
  OSError: Subprocess returned nonzero result.
  
  Pypiper terminating spawned child process 108475...
  child process terminated
  slurmstepd: error: Exceeded step memory limit at some point.
  </code></pre>
  
  <p>The last line may be informative. Although, we are still not sure what we could possibly do.</p>
   77: <p>Slurm is killing the process because it's using more memory per core than the default allocation (that's what <code>slurmstepd: error: Exceeded step memory limit at some point</code> means). I've never used pyPiper (like seemingly most people, I use snakemake), but presuming that you have it running either <code>srun</code> or <code>sbatch</code> at some point then specify something like <code>--mem-per-cpu 10G</code>.</p>
  