text: user6948: I am trying to perform a HISAT2 indexing using a GCF_000001405.39_GRCh38.p13_genomic.gtf and GCF_000001405.39_GRCh38.p13_genomic.fna files downloaded from https://www.ncbi.nlm.nih.gov/assembly/GCF_000001405.39 using the procedure described in  Pertea et al. (https://www.ncbi.nlm.nih.gov/pubmed/27560171), e.g. first I create extract_splice_sites.py and extract_exons.py and then my code looks like this
  #!/bin/bash
  #SBATCH --time=02:30:00
  #SBATCH --account=def-myusername
  #SBATCH --mem=160000M
  #SBATCH --mail-user=mymail@gmail.com
  #SBATCH --mail-type=All
  
  module load nixpkgs/16.09  intel/2018.3
  module load hisat2/2.1.0
  
  hisat2-build --ss imam.ss --exon imam.exon GCF_000001405.39_GRCh38.p13_genomic.fna imam002
  
  I am trying to receive a set of imam002*.ht2 files but every time the procedure ends with a Ran out of memory message. From here https://github.com/griffithlab/rnaseq_tutorial/wiki/Indexing I learned that I must allocate 160Gb of RAM if I want to index the entire genome. Even when I do that and submit my script via sbatch on a HPC I still got this 
  slurmstepd: error: Detected 1 oom-kill event(s) in step 30827017.batch cgroup. 
  
  Just in case here's the window that I have when I don't sbatch my script, I still have an error
  Inactive Modules:
    1) openmpi/2.1.1
  
  The following have been reloaded with a version change:
    1) gcccore/.5.4.0 => gcccore/.7.3.0     2) icc/.2016.4.258 => icc/.2018.3.222     3) ifort/.2016.4.258 => ifort/.2018.3.222     4) imkl/11.3.4.258 => imkl/2018.3.222     5) intel/2016.4 => intel/2018.3
  
  Settings:
    Output files: "imam002.*.ht2"
    Line rate: 7 (line is 128 bytes)
    Lines per side: 1 (side is 128 bytes)
    Offset rate: 4 (one in 16)
    FTable chars: 10
    Strings: unpacked
    Local offset rate: 3 (one in 8)
    Local fTable chars: 6
    Local sequence length: 57344
    Local sequence overlap between two consecutive indexes: 1024
    Endianness: little
    Actual local endianness: little
    Sanity checking: disabled
    Assertions: disabled
    Random seed: 0
    Sizeofs: void*:8, int:4, long:8, size_t:8
  Input files DNA, FASTA:
    GCF_000001405.39_GRCh38.p13_genomic.fna
  Reading reference sizes
    Time reading reference sizes: 00:00:23
  Calculating joined length
  Writing header
  Reserving space for joined string
  Joining reference sequences
    Time to join reference sequences: 00:00:13
    Time to read SNPs and splice sites: 00:00:15
    Ran out of memory; automatically trying more memory-economical parameters.
  ./hisat2_index.sh: line 24:   943 Killed                  hisat2-build --ss imam.ss --exon imam.exon GCF_000001405.39_GRCh38.p13_genomic.fna imam002
  
  When I use the same script for another genome (not human) - I don't have this problem. 
  Is it really a problem of RAM and, if it is, how is usually human genome gets indexed by HISAT2? Is it even possible?
  
  user961: You haven't allocated the requisite 160GB of RAM required. Or rather, you allocated 160GB of memory, which is what you were told to do, but they most likely meant 160GiB.
  Computers use binary, not decimal, so a gigabyte is 1024 megabytes. This is the same annoying mix-up people run into when their hard drives don't have the space the marketing department says it does. By requesting 160000M in your job script, you actually requested a few gigabytes less than what your source says you need.
  $$
  160000 \text{ megabytes} \times \frac{1 \text{ gigabyte}}{1024 \text{ megabytes}} = 156.25 \text{ gigabytes}
  $$
  $$
  160 \text{ gigabytes} \times \frac{1024 \text{ megabytes}}{1 \text{ gigabyte}} = 163840 \text{ megabytes}
  $$
  If "160GB" is what you need, then 163840M is the correct value for the job script.
  I would honestly suggest cutting the Gordian knot and not dealing with this "gigabytes vs. gibibytes" ordeal by simply using the G suffix.1
  #SBATCH --mem=160G
  
  
original_text: 6948: <p>I am trying to perform a <code>HISAT2</code> indexing using a <code>GCF_000001405.39_GRCh38.p13_genomic.gtf</code> and <code>GCF_000001405.39_GRCh38.p13_genomic.fna</code> files downloaded from <a href="https://www.ncbi.nlm.nih.gov/assembly/GCF_000001405.39" rel="nofollow noreferrer">https://www.ncbi.nlm.nih.gov/assembly/GCF_000001405.39</a> using the procedure described in  Pertea et al. (<a href="https://www.ncbi.nlm.nih.gov/pubmed/27560171" rel="nofollow noreferrer">https://www.ncbi.nlm.nih.gov/pubmed/27560171</a>), e.g. first I create <code>extract_splice_sites.py</code> and <code>extract_exons.py</code> and then my code looks like this</p>
  
  <pre><code>#!/bin/bash
  #SBATCH --time=02:30:00
  #SBATCH --account=def-myusername
  #SBATCH --mem=160000M
  #SBATCH --mail-user=mymail@gmail.com
  #SBATCH --mail-type=All
  
  module load nixpkgs/16.09  intel/2018.3
  module load hisat2/2.1.0
  
  hisat2-build --ss imam.ss --exon imam.exon GCF_000001405.39_GRCh38.p13_genomic.fna imam002
  </code></pre>
  
  <p>I am trying to receive a set of <code>imam002*.ht2</code> files but every time the procedure ends with a <code>Ran out of memory</code> message. From here <a href="https://github.com/griffithlab/rnaseq_tutorial/wiki/Indexing" rel="nofollow noreferrer">https://github.com/griffithlab/rnaseq_tutorial/wiki/Indexing</a> I learned that I must allocate 160Gb of RAM if I want to index the entire genome. Even when I do that and submit my script via sbatch on a HPC I still got this </p>
  
  <pre><code>slurmstepd: error: Detected 1 oom-kill event(s) in step 30827017.batch cgroup. 
  </code></pre>
  
  <p>Just in case here's the window that I have when I don't sbatch my script, I still have an error</p>
  
  <pre><code>Inactive Modules:
    1) openmpi/2.1.1
  
  The following have been reloaded with a version change:
    1) gcccore/.5.4.0 =&gt; gcccore/.7.3.0     2) icc/.2016.4.258 =&gt; icc/.2018.3.222     3) ifort/.2016.4.258 =&gt; ifort/.2018.3.222     4) imkl/11.3.4.258 =&gt; imkl/2018.3.222     5) intel/2016.4 =&gt; intel/2018.3
  
  Settings:
    Output files: "imam002.*.ht2"
    Line rate: 7 (line is 128 bytes)
    Lines per side: 1 (side is 128 bytes)
    Offset rate: 4 (one in 16)
    FTable chars: 10
    Strings: unpacked
    Local offset rate: 3 (one in 8)
    Local fTable chars: 6
    Local sequence length: 57344
    Local sequence overlap between two consecutive indexes: 1024
    Endianness: little
    Actual local endianness: little
    Sanity checking: disabled
    Assertions: disabled
    Random seed: 0
    Sizeofs: void*:8, int:4, long:8, size_t:8
  Input files DNA, FASTA:
    GCF_000001405.39_GRCh38.p13_genomic.fna
  Reading reference sizes
    Time reading reference sizes: 00:00:23
  Calculating joined length
  Writing header
  Reserving space for joined string
  Joining reference sequences
    Time to join reference sequences: 00:00:13
    Time to read SNPs and splice sites: 00:00:15
    Ran out of memory; automatically trying more memory-economical parameters.
  ./hisat2_index.sh: line 24:   943 Killed                  hisat2-build --ss imam.ss --exon imam.exon GCF_000001405.39_GRCh38.p13_genomic.fna imam002
  </code></pre>
  
  <p>When I use the same script for another genome (not human) - I don't have this problem. </p>
  
  <p>Is it really a problem of RAM and, if it is, how is usually human genome gets indexed by HISAT2? Is it even possible?</p>
   961: <p>You haven't allocated the requisite 160GB of RAM required. Or rather, you allocated 160GB of memory, which is what you were told to do, but they most likely meant 160<strong>GiB</strong>.</p>
  
  <p>Computers use binary, not decimal, so a gigabyte is 1024 megabytes. This is the same annoying mix-up people run into when their hard drives don't have the space the marketing department says it does. By requesting 160000M in your job script, you actually requested a few gigabytes less than what your source says you need.</p>
  
  <p><span class="math-container">$$
  160000 \text{ megabytes} \times \frac{1 \text{ gigabyte}}{1024 \text{ megabytes}} = 156.25 \text{ gigabytes}
  $$</span></p>
  
  <p><span class="math-container">$$
  160 \text{ gigabytes} \times \frac{1024 \text{ megabytes}}{1 \text{ gigabyte}} = 163840 \text{ megabytes}
  $$</span></p>
  
  <p>If "160GB" is what you need, then <code>163840M</code> is the correct value for the job script.</p>
  
  <p>I would honestly suggest <a href="https://en.wikipedia.org/wiki/Gordian_Knot" rel="nofollow noreferrer">cutting the Gordian knot</a> and not dealing with this "<a href="https://en.wikipedia.org/wiki/Gigabyte" rel="nofollow noreferrer">gigabytes</a> vs. <a href="https://en.wikipedia.org/wiki/Gordian_Knot" rel="nofollow noreferrer">gibibytes</a>" ordeal by simply using the <code>G</code> suffix.<sup><a href="https://slurm.schedmd.com/sbatch.html" rel="nofollow noreferrer">1</a></sup></p>
  
  <pre><code>#SBATCH --mem=160G
  </code></pre>
  