title: How to speed up script and do not clog RAM (dir size +2mil files)
body: <p>I have written this novice code which:</p>
  
  <p>Runs through master directory (+2mil files and many many sub-folders)
  filters .tmx files(around 12k files in m.dir) and extracts specific strings.
  and saves to .log file. After it is done there are sub procedures which clean up the log files and merges everything to 1 file. the problem is that i left script over-night and it just got stuck.</p>
  
  <p>I recon it's due to dir size. I have previously listed all directory files to .txt is it possible that script would read .txt and would process 1 file at a time, maybe this way it wont take up 99% RAM after a while.</p>
  
  <p>Also maybe you have other insights in speeding op or merging these procedures to one.</p>
  
  <pre><code>Get-ChildItem "MasterDirpath\*.tmx" -Recurse  | 
  Foreach-Object {
  $content = Get-Content $_.FullName
  
  #filter and save content to the original file
  #$content | Where-Object {$_ -match '&lt;tu '} | Set-Content $_.FullName
  
  #filter and save content to a new file 
  $content | Where-Object {!($_ -match '(?:creationid|changeid)="([^"]+)"' -or 
  $_ -match '(&lt;tuv.+?lang="[A-Za-z\-]+"&gt;)')} | %{$matches[1]} |Get-Unique  | 
  Set-Content ($_.BaseName + '_out.log') 
  
  }
  
  Get-ChildItem "dir\tologs" -Filter *.log |
  Foreach-Object {
  $content = Get-Content -Raw $_.FullName
  #make one line from extracted matches
  $content -Replace "`r`n&lt;" ,"`t&lt;"  |Set-Content $_.FullName
  
  }
  
  
  
  Get-ChildItem "dir\tologs"  -Filter *.log | 
  Foreach-Object {
  $content = Get-Content $_.FullName
  
  #filter and save content to the original file log file
  $content | Where-Object {$_ -match '^.+$ '} | Sort | Get-Unique | Set-
  Content $_.FullName
  }
  
  
  $path = "dir\tologs"
  $out  = "dir\tologs\output.txt"
  
  Get-ChildItem $path -Filter *.log | % {
  $file = $_.Name
  Get-Content $_.FullName | % {
      "${file}: $_" | Out-File -Append $out
  }
  }
  </code></pre>
  
  <p><strong>UPDATE:</strong>
  <strong>Sample input</strong></p>
  
  <p>These .tmx files vary in size from 1Mb to 2Gb directory size is around 1Tb. And all files there can be from few Mb upto few Gb. Script runs fine on small directory with 50 tmx files 1-100mb.</p>
  
  <pre><code>&lt;?xml version="1.0" encoding="utf-16"?&gt;
  &lt;!DOCTYPE tmx SYSTEM "tmx14.dtd"&gt;
  &lt;tmx version="1.4"&gt;
  &lt;header creationtool="MemoQ" creationtoolversion="7.0.68" segtype="sentence" 
  adminlang="en-us" creationid="lsmall" srclang="en-us" o-tmf="MemoQTM" 
  datatype="unknown"&gt;
  &lt;prop type="defclient"&gt; &lt;/prop&gt;
  &lt;prop type="defproject"&gt; &lt;/prop&gt;
  &lt;prop type="defdomain"&gt; &lt;/prop&gt;
  &lt;prop type="defsubject"&gt; &lt;/prop&gt;
  &lt;prop type="description"&gt; &lt;/prop&gt;
  &lt;prop type="targetlang"&gt;hu&lt;/prop&gt;
  &lt;prop type="name"&gt;28807_Project_HU&lt;/prop&gt;
  &lt;/header&gt;
  &lt;body&gt;
  &lt;tu changedate="20151104T174128Z" creationdate="20150929T180844Z" **creationid="pmccrory"** **changeid="lsmall"&gt;**
  &lt;prop type="client"&gt; &lt;/prop&gt;
  &lt;prop type="project"&gt; &lt;/prop&gt;
  &lt;prop type="domain"&gt; &lt;/prop&gt;
  &lt;prop type="subject"&gt; &lt;/prop&gt;
  &lt;prop type="corrected"&gt;no&lt;/prop&gt;
  &lt;prop type="aligned"&gt;no&lt;/prop&gt;
  &lt;prop type="x-document"&gt;node_data_en_final.xml&lt;/prop&gt;
  ***&lt;tuv xml:lang="en-us"&gt;***
    &lt;prop type="x-context-pre"&gt;&amp;lt;seg&amp;gt;Biomarkers and Integrated Solutions&amp;lt;/seg&amp;gt;&lt;/prop&gt;
    &lt;prop type="x-context-post"&gt;&amp;lt;seg&amp;gt;Novel therapeutic agents that have fast onset of action, good safety and tolerability profiles and that address common co-morbidities (for example, anxiety and substance abuse) &amp;lt;ph type='fmt'&amp;gt;{}&amp;lt;/ph&amp;gt;&amp;lt;it pos='begin'&amp;gt;&amp;amp;lt;ul&amp;amp;gt;&amp;lt;/it&amp;gt;&amp;lt;/seg&amp;gt;&lt;/prop&gt;
    &lt;seg&gt;&lt;it pos='begin'&gt;&amp;lt;ul class=&amp;quot;inline&amp;quot;&amp;gt;&lt;/it&gt;&lt;/seg&gt;
  &lt;/tuv&gt;
  ***&lt;tuv xml:lang="hu"&gt;***
    &lt;seg&gt;&lt;it pos='begin'&gt;&amp;lt;ul class=&amp;quot;inline&amp;quot;&amp;gt;&lt;/it&gt;&lt;/seg&gt;
  &lt;/tuv&gt;
  &lt;/tu&gt;
  &lt;/body&gt;
  &lt;/tmx&gt;
  </code></pre>
  
  <p><strong>Output after procedure:</strong></p>
  
  <pre>
  AABB-COR-09_Master_DE_out.log: 6293 SYB &lt;tuv xml:lang="en-us"&gt;    &lt;tuv xml:lang="de-de"&gt;
  ABB-COR-09_Master_DE_out.log: AD    &lt;tuv xml:lang="en-us"&gt;    &lt;tuv xml:lang="de-de"&gt;
  ABB-COR-09_Master_DE_out.log: AGENTILE  &lt;tuv xml:lang="en-us"&gt;    &lt;tuv xml:lang="de-de"&gt;
  ABB-COR-09_Master_DE_out.log: ALIGN!    &lt;tuv xml:lang="en-us"&gt;    &lt;tuv xml:lang="de-de"&gt;
  ABB-COR-09_Master_DE_out.log: ANGELIKA  &lt;tuv xml:lang="en-us"&gt;    &lt;tuv xml:lang="de-de"&gt;
  ABB-COR-09_Master_DE_out.log: ASEDR &lt;tuv xml:lang="en-us"&gt;    &lt;tuv xml:lang="de-de"&gt;
  </pre>