title: Why using pipe for sort (linux command) is slow?
body: <p>I have a large text file of ~8GB which I need to do some simple filtering and then sort all the rows. I am on a 28-core machine with SSD and 128GB RAM. I have tried</p>
  
  <p>Method 1</p>
  
  <pre><code>awk '...' myBigFile | sort --parallel = 56 &gt; myBigFile.sorted
  </code></pre>
  
  <p>Method 2</p>
  
  <pre><code>awk '...' myBigFile &gt; myBigFile.tmp
  sort --parallel 56 myBigFile.tmp &gt; myBigFile.sorted
  </code></pre>
  
  <p>Surprisingly, method1 takes 11.5 min while method2 only takes (0.75 + 1 &lt; 2) min. Why is sorting so slow when piped? Is it not paralleled?</p>
  
  <p><strong>EDIT</strong></p>
  
  <p><code>awk</code> and <code>myBigFile</code> is not important, this experiment is repeatable by simply using <code>seq 1 10000000 | sort --parallel 56</code> (thanks to @Sergei Kurenkov), and I also observed a six-fold speed improvement using un-piped version on my machine.</p>