title: How to remove duplicate rows from flat file using SSIS?
body: <p>Let me first say that being able to take 17 million records from a flat file, pushing to a DB on a remote box and having it take 7 minutes is amazing. SSIS truly is fantastic. But now that I have that data up there, how do I remove duplicates?</p>
  
  <p>Better yet, I want to take the flat file, remove the duplicates from the flat file and put them back into another flat file.</p>
  
  <p>I am thinking about a:</p>
  
  <p><strong><code>Data Flow Task</code></strong></p>
  
  <ul>
  <li>File source (with an associated file connection)</li>
  <li>A for loop container</li>
  <li>A script container that contains some logic to tell if another row exists</li>
  </ul>
  
  <p>Thak you, and everyone on this site is incredibly knowledgeable.</p>
  
  <p><strong><code>Update:</code></strong> <a href="http://rafael-salas.blogspot.com/2007/04/remove-duplicates-using-t-sql-rank.html" rel="noreferrer">I have found this link, might help in answering this question</a></p>