title: reorganising data into prefix inside Amazon S3
body: <p>I've a SQL tool for bigdata processing which dumps data into an Amazon S3 bucket in <code>.csv</code> files under the following prefix:</p>
  <pre><code>myS3Bucket
  |
  |__myTopPrefix
      |
      |_object1.csv
      |_object2.csv
      |_object3.csv
      |_object4.csv
      |_object5.csv
      |_object6.csv
  </code></pre>
  <p>My requirement is to further reorganise this data into sub-prefixes with following structure:</p>
  <pre><code>myS3Bucket
  |
  |__myTopPrefix
      |_subPrefix_1
      |   |_subSubPrefix_1
      |       |_object1.csv
      |       |_object2.csv
      |
      |_subPrefix_2
          |_subSubPrefix_2a
          |   |_object3.csv
          |   |_object4.csv
          |
          |_subSubPrefix_2b
              |_object5.csv
              |_object6.csv
  </code></pre>
  <p>The judgement of which object to put under which prefix is governed by looking at the header row in that object.csv file.</p>
  <p>I'm allowed to use any Tool or (preferably)AWS service to achieve this but I don't know how to do it. I thought of using an <code>aws lambda</code> function to do this and read the documentation for lambda and other resources but couldn't get through. I'm new to AWS services and don't have deep understanding of them yet.</p>
  <p><strike> Update 1: number of entries in object.csv might be in billions, like 10-12 Billion. </strike></p>
  <p><strong>Update 2:</strong> I've changed approach to my service and now &quot;updated 1&quot; is no more valid. The entries in any object.csv will now be around 10,000 but the object files that will be dumped in S3 in one execution of SQL script will now be increased to 12. Typical file size would be <code>150-200 MB</code> in worst case(which is never going to happen XD )</p>
  <p>Also, I need this reorganising of objects to happen after every new object pushed to S3. And the prefix name(or the hierarchical path) I've already created but they don't hold any data/object initially.</p>
  <p>As a workflow, I process data in my tool then dump it into <code>S3://myS3Bucket/myTopPrefix/</code> Now, that data should move to appropriate prefix based on information inside header of file. Job Done!</p>
  <p><strong>Update 3:</strong> For those who are thinking how 12 objects will be generated from a single SQL script: Basically in this tool, I can write several <code>select ... from ... where ...</code> and each query will output data as one object which is hardcoded(limitation of tool) to go under <code>S3://myS3Bucket/myTopPrefix/</code>. In current implementation, 12 objects will be generated in single execution of the SQL script.</p>
  <p>Any help is highly appreciated.
  Thanks!</p>