text: user9024: I have 2 .fa sequenes like:
  >some description of first...
  ACTGACTACGTACGACTACGATCTGACTACACGTAGATAGACTAGTCACTACG
  ACTGACTACGTACGACTGACTACATGACTGAC...
  > some description of second...
  ATCGATAGCTACGATCGATAGTAGATAG....
  
  This file is 461mb (1 chromosome of homo sapiens & 1 chromosome of pan troglodytes) and I want to align it. I am using mafft.
  mafft --globalpair --maxiterate 1000 Homo_sapiens_1_Pan_troglodytes_1.fa > ./mafft_results/Homo_sapiens_1_Pan_troglodytes_1_output.fa
  
  And it returns me
  generating a scoring matrix for nucleotide (dist=200) ... done
  All-to-all alignment.
  /usr/bin/mafft: line 2719:  6030 Killed                  "$prefix/tbfast" _ -u $unalignlevel $localparam -C $numthreads $seqtype $model -g $pgexp -f $pggop -Q $spfactor -h $pgaof -A $usenaivepairscore $focusarg _ -+ $iterate -W $minimumweight -V "-"$gopdist -s $unalignlevel $legacygapopt $mergearg $outnum $addarg $add2ndhalfarg -C $numthreadstb $rnaopt $weightopt $treeinopt $treeoutopt $distoutopt $seqtype $model -f "-"$gop -Q $spfactor -h $aof $param_fft $localparam $algopt $treealg $scoreoutarg $focusarg < infile > /dev/null 2>> "$progressfile"
  
  Why? How could I align it for further comparing?
  EDIT 1
  MAFFT working cool, when I tried to use it on test sequence file
  >test1
  ATCGATCGATGACTACGATCGACTACGTACG
  ACTAGCTAGCTACGCATAGTAGATCGA
  ATCGACTAGCTCAGATCGACTAGATCGATCGT
  >test2
  TTATCCACTGATCGACTCAGATCGACTCAGAC
  ATCGATCAGCTACGATGACTACGTAC
  ACTGACTAGCTAGACTGACTAGCTACGTCA
  
  and after MAFFT mafft test.fa > output.fa it returned
  
  test1
  ------atcgatcgat-------------gactacgatcgactacgtacgactagctagc
  tacgcatagtagatcgaatcgactagctcagatcgactagatcgatcgt
  test2
  ttatccactgatcgactcagatcgactcagacatcgatcagctac---------gatgac
  tacgtac----------actgactagct-agactgactagctacgtca-
  
  So it works.
  I can use all 4 at the example. Now about my large file:
  alex@alex:~/Desktop/test/BOTH/homo_sapiens_pan_troglodytes$ mafft --thread 4 --maxiterate 2 Homo_sapiens_1_Pan_troglodytes_1.fa > ./mafft_results/Homo_sapiens_1_Pan_troglodytes_1_output.fa
  nthread = 4
  nthreadpair = 4
  nthreadtb = 4
  ppenalty_ex = 0
  reallocating...
  done.
  reallocating...
  done.
  reallocating...
      done.
  reallocating...
  done.
  reallocating...
  ...
  stacksize: 8192 kb
  reallocating...
  done.
  reallocating...
  done.
  reallocating...
  done.
  ...
  generating a scoring matrix for nucleotide (dist=200) ... done
  Gap Penalty = -1.53, +0.00, +0.00
  
  
  
  Making a distance matrix ..
      1 / 2 (thread    1)
  done.
  
  Constructing a UPGMA tree (efffree=1) ... 
      0 / 2
  done.
  
  Progressive alignment 1/1... 
  
  len1=230480994, len2=223796828, Switching to the memsave mode
  STEP     1 / 1 (thread    0)m/usr/bin/mafft: line 2719:  3192 Killed                  "$prefix/disttbfast" -q $npickup -E $cycledisttbfast -V "-"$gopdist -s $unalignlevel $legacygapopt $mergearg -W $tuplesize $termgapopt $outnum $addarg $add2ndhalfarg -C $numthreads-$numthreadstb $memopt $weightopt $treeinopt $treeoutopt $distoutopt $seqtype $model -g $gexp -f "-"$gop -Q $spfactor -h $aof $param_fft $algopt $treealg $scoreoutarg $anchoropt < infile > pre 2>> "$progressfile"
  
  Doesnt matter I do not enter --thread 4 or not, --maxiterate 2 or --maxiterate 1000, it still shows me same error. It takes around 2 minute. Ram usage when running at the first minute is 6656mb, after 1 min 11028mb and then it out of ram and error..
  PS
  discussion moved here
  
  user3848: --thread 4 is essential, --maxiterate just try and get it working, not getting it to work optimally. Just make sure there are no other processes running on the machine when --thread 4 is operating, e.g. you are using another machine as your desktop
  The gist is as you spread the calculation across your processors (4) and simplify the calculation (lowering the iterations), you are getting further into the calculation, for example now you are getting into gap penalties.
  If you are agreed then I would try performing the alignment using just one chromosome of the human genome against the chimp genome and I suspect that would work. Then take it from there.
  The objective is to get the calculation working, for example I wouldn't trust a result with max. iterations of 2, but once you have it working you can then start refining the parameters towards optimal results. The i3 processor isn't perfect in context.
  Using --thread 4 is super important and you want to increase maxiterate towards the default if the machine supports it.
  I suspect the solution is to perform the calculation on an 8+-core machine and set --thread 8 (or higher depending on the number of cores on the machine), at least it will get further into the calculation. With 4-cores and an i3 (diagnostics of RAM usage would be nice) in any case it just doesn't seem to like it. The alternative is to perform this across the genome via sampling the chromosomes on your current i3.
  
  Just to explain why this works. When you have a RAM intensive job and you spread it across cores the RAM overhead massively reduces for each core that is added. Don't ask me how this works in the hardware mecahanics of RAM distribution per core. The reason I know this is because of emperically parallelising Bayesian calculation using each chain on a different core.
  
original_text: 9024: <p>I have 2 .fa sequenes like:</p>
  <pre><code>&gt;some description of first...
  ACTGACTACGTACGACTACGATCTGACTACACGTAGATAGACTAGTCACTACG
  ACTGACTACGTACGACTGACTACATGACTGAC...
  &gt; some description of second...
  ATCGATAGCTACGATCGATAGTAGATAG....
  </code></pre>
  <p>This file is 461mb (1 chromosome of homo sapiens &amp; 1 chromosome of pan troglodytes) and I want to align it. I am using mafft.</p>
  <pre><code>mafft --globalpair --maxiterate 1000 Homo_sapiens_1_Pan_troglodytes_1.fa &gt; ./mafft_results/Homo_sapiens_1_Pan_troglodytes_1_output.fa
  </code></pre>
  <p>And it returns me</p>
  <pre><code>generating a scoring matrix for nucleotide (dist=200) ... done
  All-to-all alignment.
  /usr/bin/mafft: line 2719:  6030 Killed                  &quot;<span class="math-container">$prefix/tbfast" _ -u $</span>unalignlevel <span class="math-container">$localparam -C $</span>numthreads <span class="math-container">$seqtype $</span>model -g <span class="math-container">$pgexp -f $</span>pggop -Q <span class="math-container">$spfactor -h $</span>pgaof -A <span class="math-container">$usenaivepairscore $</span>focusarg _ -+ <span class="math-container">$iterate -W $</span>minimumweight -V &quot;-&quot;<span class="math-container">$gopdist -s $</span>unalignlevel <span class="math-container">$legacygapopt $</span>mergearg <span class="math-container">$outnum $</span>addarg <span class="math-container">$add2ndhalfarg -C $</span>numthreadstb <span class="math-container">$rnaopt $</span>weightopt <span class="math-container">$treeinopt $</span>treeoutopt <span class="math-container">$distoutopt $</span>seqtype <span class="math-container">$model -f "-"$</span>gop -Q <span class="math-container">$spfactor -h $</span>aof <span class="math-container">$param_fft $</span>localparam <span class="math-container">$algopt $</span>treealg <span class="math-container">$scoreoutarg $</span>focusarg &lt; infile &gt; /dev/null 2&gt;&gt; &quot;$progressfile&quot;
  </code></pre>
  <p>Why? How could I align it for further comparing?</p>
  <p><strong>EDIT 1</strong></p>
  <p>MAFFT working cool, when I tried to use it on test sequence file</p>
  <pre><code>&gt;test1
  ATCGATCGATGACTACGATCGACTACGTACG
  ACTAGCTAGCTACGCATAGTAGATCGA
  ATCGACTAGCTCAGATCGACTAGATCGATCGT
  &gt;test2
  TTATCCACTGATCGACTCAGATCGACTCAGAC
  ATCGATCAGCTACGATGACTACGTAC
  ACTGACTAGCTAGACTGACTAGCTACGTCA
  </code></pre>
  <p>and after MAFFT <code>mafft test.fa &gt; output.fa</code> it returned</p>
  <blockquote>
  <p>test1
  ------atcgatcgat-------------gactacgatcgactacgtacgactagctagc
  tacgcatagtagatcgaatcgactagctcagatcgactagatcgatcgt
  test2
  ttatccactgatcgactcagatcgactcagacatcgatcagctac---------gatgac
  tacgtac----------actgactagct-agactgactagctacgtca-</p>
  </blockquote>
  <p>So it works.</p>
  <p>I can use all 4 at the example. Now about my large file:</p>
  <pre><code>alex@alex:~/Desktop/test/BOTH/homo_sapiens_pan_troglodytes$ mafft --thread 4 --maxiterate 2 Homo_sapiens_1_Pan_troglodytes_1.fa &gt; ./mafft_results/Homo_sapiens_1_Pan_troglodytes_1_output.fa
  nthread = 4
  nthreadpair = 4
  nthreadtb = 4
  ppenalty_ex = 0
  reallocating...
  done.
  reallocating...
  done.
  reallocating...
      done.
  reallocating...
  done.
  reallocating...
  ...
  stacksize: 8192 kb
  reallocating...
  done.
  reallocating...
  done.
  reallocating...
  done.
  ...
  generating a scoring matrix for nucleotide (dist=200) ... done
  Gap Penalty = -1.53, +0.00, +0.00
  
  
  
  Making a distance matrix ..
      1 / 2 (thread    1)
  done.
  
  Constructing a UPGMA tree (efffree=1) ... 
      0 / 2
  done.
  
  Progressive alignment 1/1... 
  
  len1=230480994, len2=223796828, Switching to the memsave mode
  STEP     1 / 1 (thread    0)m/usr/bin/mafft: line 2719:  3192 Killed                  &quot;<span class="math-container">$prefix/disttbfast" -q $</span>npickup -E <span class="math-container">$cycledisttbfast -V "-"$</span>gopdist -s <span class="math-container">$unalignlevel $</span>legacygapopt <span class="math-container">$mergearg -W $</span>tuplesize <span class="math-container">$termgapopt $</span>outnum <span class="math-container">$addarg $</span>add2ndhalfarg -C <span class="math-container">$numthreads-$</span>numthreadstb <span class="math-container">$memopt $</span>weightopt <span class="math-container">$treeinopt $</span>treeoutopt <span class="math-container">$distoutopt $</span>seqtype <span class="math-container">$model -g $</span>gexp -f &quot;-&quot;<span class="math-container">$gop -Q $</span>spfactor -h <span class="math-container">$aof $</span>param_fft <span class="math-container">$algopt $</span>treealg <span class="math-container">$scoreoutarg $</span>anchoropt &lt; infile &gt; pre 2&gt;&gt; &quot;$progressfile&quot;
  </code></pre>
  <p>Doesnt matter I do not enter <code>--thread 4</code> or not, <code>--maxiterate 2</code> or <code>--maxiterate 1000</code>, it still shows me same error. It takes around 2 minute. Ram usage when running at the first minute is 6656mb, after 1 min 11028mb and then it out of ram and error..</p>
  <p><em>PS</em></p>
  <p>discussion moved <a href="https://stackoverflow.com/questions/62488623/out-of-ram-while-work-with-very-large-matrix">here</a></p>
   3848: <p><code>--thread 4</code> is essential, <code>--maxiterate</code> just try and get it working, not getting it to work optimally. Just make sure there are no other processes running on the machine when --thread 4 is operating, e.g. you are using another machine as your desktop</p>
  <p>The gist is as you spread the calculation across your processors (4) and simplify the calculation (lowering the iterations), you are getting further into the calculation, for example now you are getting into gap penalties.</p>
  <p>If you are agreed then I would try performing the alignment using just one chromosome of the human genome against the chimp genome and I suspect that would work. Then take it from there.</p>
  <p>The objective is to get the calculation working, for example I wouldn't trust a result with max. iterations of 2, but once you have it working you can then start refining the parameters towards optimal results. The i3 processor isn't perfect in context.</p>
  <p>Using <code>--thread 4</code> is super important and you want to increase maxiterate towards the default if the machine supports it.</p>
  <p>I suspect the solution is to perform the calculation on an 8+-core machine and set <code>--thread 8</code> (or higher depending on the number of cores on the machine), at least it will get further into the calculation. With 4-cores and an i3 (diagnostics of RAM usage would be nice) in any case it just doesn't seem to like it. The alternative is to perform this across the genome via sampling the chromosomes on your current i3.</p>
  <hr />
  <p>Just to explain why this works. When you have a RAM intensive job and you spread it across cores the RAM overhead massively reduces for each core that is added. Don't ask me how this works in the hardware mecahanics of RAM distribution per core. The reason I know this is because of emperically parallelising Bayesian calculation using each chain on a different core.</p>
  