title: Redshift how to handle other files in S3 that are not overwrite
body: <p>I have a table in Amazon Redshift and my use case would be to unload this table on row number partition to multiple S3 file locations daily. The files from S3 locations will be loaded to DynamoDB by means of S3DataNode of Datapipeline with parameters of DirectoryPath that is pointing towards the S3 locations updated in above step.</p>
  
  <pre><code>unload('select * from test_table where (row_num%100) &gt;= 0 and (row_num%100)&lt;=30')
  to 's3://location1/file_'
  iam_role 'arn:aws:iam::xxxxxxxxxx'
  parallel off
  ALLOWOVERWRITE;
  
  unload('select * from test_table where (row_num%100) &gt;= 31 and (row_num%100)&lt;=100')
  to 's3://location2/file_'
  iam_role 'arn:aws:iam::xxxxxxxxxx'
  parallel off
  ALLOWOVERWRITE;
  </code></pre>
  
  <p>Day1: there are huge number of rows in Redshift and it created file_000, file_001, file_002 and file_003 in one of S3 location (say location2).</p>
  
  <p>Day2: there is not much of rows in Redshift and it created file_000. Because of <code>ALLOWOVERWRITE</code> option that I'm using while unload to S3, there is only overwrite of file_000. but still the other files exists(file_001, file_002, file_003 that were created on Day1) and this is resulting in updating already existing items from files_002 and file_003 to DynamoDB.</p>
  
  <p>How should I copy only overwrite files to DynamoDB? </p>
  
  <p>(Note: If I use MANIFEST on unload, Can the datapipeline S3 DataNode only copies the manifest generated files to DynamoDB ?)</p>