<p>I ran bash script from java code to sort a file of tens of millions of strings of &lt;25 characters. The java application is running in a kubernetes pod. The script failed and the error message is</p>
<pre><code>bash: line 1:   425 Killed                  
sort /tmp/keys &gt; /tmp/keys_sorted
</code></pre>
<p>What does 425 actually mean here? How can I fix it?</p>

<p>This means that <code>sort</code> was sent a SIGKILL signal.</p>
<p>On modern systems without something weird going on, this <em>almost</em> always means you're out of RAM.</p>
<p>Linux supports something called &quot;memory overcommit&quot;. Due to language runtimes (Java among them!) habitually allocating far more virtual memory than they'll ever need matching physical memory for, the kernel allows allocation requests to succeed <em>even if it doesn't have enough memory to back them</em>. When the application actually tries to <em>access</em> those virtual memory pages, the kernel's fault handler gets invoked to try to find physical memory to map to them.</p>
<p>Sometimes it can free up space by deleting unused pages -- discarding block cache contents, or memory pages that are mapped to file contents and thus can be reloaded from that file later. Sometimes, however, there's more memory outstanding than can be free'd, and the kernel invokes the &quot;OOM killer&quot; -- where OOM stands for &quot;out of memory&quot; -- to kill some processes and <em>make</em> more free RAM.</p>
<hr />
<p>In the case of <code>sort</code>ing a large file specifically:</p>
<ul>
<li>Make sure you're using GNU sort, not a clone like busybox sort.
This is because GNU sort, but not all its smaller clones, supports breaking a large stream into pieces, writing those pieces to disk, and then doing a merge sort to reassemble them later; so it can sort files larger than available RAM.</li>
<li>Make sure that you have temporary space that is <em>actually</em> disk.
If GNU sort tries to conserve RAM by shuffling contents off to disk <em>that is actually RAM itself</em>, that's obviously not going to go well.</li>
<li>Use the GNU <code>sort</code> argument <code>-S</code> to limit the amount of memory GNU sort will allocate before shunting data to temporary files on disk. (For example, one can use <code>sort -S 32M</code> to allow 32MB of RAM to be allocated for working space).</li>
</ul>
