title: How to merge and remove duplicates of CSV files using Powershell
body: <p>I would like to remove duplicates in a CSV file using PowerShell. I know that there are posts about this already but I can't seem to find one that helps.</p>
  <p>I'm trying to merge 2 CSV Files that have the same header and then remove the duplicates of the resulting file based on the IDs listed in the first column and then put it to the same CSV file.</p>
  <p>The properties of the file are as follows:
  <a href="https://i.stack.imgur.com/n0O8m.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/n0O8m.png" alt="enter image description here" /></a></p>
  <p>And when I try to use the sort and unique method, I get the following (not a table:
  <a href="https://i.stack.imgur.com/o6I5g.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/o6I5g.png" alt="enter image description here" /></a></p>
  <p>Here is my code so far:
  <a href="https://i.stack.imgur.com/AxdLJ.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/AxdLJ.png" alt="enter image description here" /></a></p>
  <pre><code>####
  #MERGE
  $getFirstLine = $true
      get-childItem &quot;C:\IGHandover\Raw\IG_INC*.csv&quot;| foreach {
      $filePath = $_
      $lines =  $lines = Get-Content $filePath  
      $linesToWrite = switch($getFirstLine) {
             $true  {$lines}
             $false {$lines | Select -Skip 1}
      }
      $getFirstLine = $false
      Add-Content &quot;C:\IGHandover\new.csv&quot; $linesToWrite
      }
  
  ####
  #REMOVE DUPLICATES
  Import-Csv &quot;C:\IGHandover\new.csv&quot; | Sort inc_number -Unique |
      Set-Content &quot;C:\IGHandover\new.csv&quot;
  </code></pre>