title: No garbage collection while PowerShell pipeline is executing
body: <p><strong>UPDATE:</strong> The following bug seems to be resolved with PowerShell 5. The bug remains in 3 and 4. So don't process any huge files with the pipeline unless you're running PowerShell 2 or 5.</p>
  
  <hr>
  
  <p>Consider the following code snippet:</p>
  
  <pre><code>function Get-DummyData() {
      for ($i = 0; $i -lt 10000000; $i++) {
          "This is freaking huge!! I'm a ninja! More words, yay!"
      }
  }
  
  Get-DummyData | Out-Null
  </code></pre>
  
  <p>This will cause PowerShell memory usage to grow uncontrollably. After executing <code>Get-DummyData | Out-Null</code> a few times, I have seen PowerShell memory usage get all the way up to 4 GB.</p>
  
  <p>According to <a href="https://www.red-gate.com/products/dotnet-development/ants-memory-profiler/" rel="noreferrer">ANTS Memory Profiler</a>, we have a whole lot of things sitting around in the garbage collector's finalization queue. When I call <code>[GC]::Collect()</code>, the memory goes from 4 GB to a mere 70 MB. So we don't have a memory leak, strictly speaking.</p>
  
  <p>Now, it's not good enough for me to be able to call <code>[GC]::Collect()</code> when I'm finished with a long-lived pipeline operation. I need garbage collection to happen <em>during</em> a pipeline operation. However if I try to invoke <code>[GC]::Collect()</code> while the pipeline is executing...</p>
  
  <pre><code>function Get-DummyData() {
      for ($i = 0; $i -lt 10000000; $i++) {
          "This is freaking huge!! I'm a ninja! More words, yay!"
  
          if ($i % 1000000 -eq 0) {
              Write-Host "Prompting a garbage collection..."
              [GC]::Collect()
          }
      }
  }
  
  Get-DummyData | Out-Null
  </code></pre>
  
  <p>... the problem remains. Memory usage grows uncontrollably again. I have tried several variations of this, such as adding <code>[GC]::WaitForPendingFinalizers()</code>, <code>Start-Sleep -Seconds 10</code>, etc. I have tried changing garbage collector <a href="https://msdn.microsoft.com/en-us/library/bb384202(v=vs.110).aspx" rel="noreferrer">latency modes</a> and forcing PowerShell to use <a href="https://msdn.microsoft.com/en-us/library/Cc165011(v=office.11).aspx" rel="noreferrer">server garbage collection</a> to no avail. I just can't get the garbage collector to do its thing while the pipeline is executing.</p>
  
  <p>This isn't a problem at all in PowerShell 2.0. It's also interesting to note that <code>$null = Get-DummyData</code> also seems to work without memory issues. So it seems tied to the pipeline, rather than the fact that we're generating tons of strings.</p>
  
  <p>How can I prevent my memory from growing uncontrollably during long pipelines?</p>
  
  <p>Side note:</p>
  
  <p>My Get-DummyData function is only for demonstration purposes. My real-world problem is that I'm unable to read through large files in PowerShell using <code>Get-Content</code> or <code>Import-Csv</code>. No, I'm <em>not</em> storing the contents of these files in variables. I'm <a href="http://tasteofpowershell.blogspot.com/2008/09/pipes-loops-and-exploding-memory-usage.html" rel="noreferrer">strictly using the pipeline</a> like I'm supposed to. <code>Get-Content .\super-huge-file.txt | Out-Null</code> produces the same problem.</p>