title: Uncompress tar file hierarchy on AWS S3 using CLI?
body: <p>I'm using the <a href="http://www.if4it.com/NOUNZ/" rel="nofollow noreferrer">NOUNZ data compiler</a> on OSX (or Linux), which automatically generates a massive directory structure of static HTML files (<em>hundreds of thousands and sometimes millions of files</em>).</p>
  
  <p>A simplified example of the generated directory tree looks something like the following...</p>
  
  <p><a href="https://i.stack.imgur.com/B1ltv.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/B1ltv.png" alt="enter image description here"></a></p>
  
  <p>Normally, if I want to move the entire tree to a remote web server, I simply tar and compress the tree, using the commands:</p>
  
  <pre><code>tar -cvf HTML.tar HTML
  gzip HTML.tar
  </code></pre>
  
  <p>This generates a tar-ed and compressed file called <code>HTML.tar.gz</code></p>
  
  <p>I can then FTP or SCP the above file to the remote web server and I can simply uncompress an untar the file using the following commands:</p>
  
  <pre><code>gzip -d HTML.tar.gz
  tar -xvf HTML.tar
  </code></pre>
  
  <p>This will result in the exact same file tree on the web server that was generated by the data compiler on the local machine.</p>
  
  <p><strong>THE PROBLEM:</strong> I'd like to mimic the same behavior as above using Amazon Web Services (AWS) Simple Storage Solution (S3).</p>
  
  <p><strong>MY QUESTION:</strong> What is the best way to mimic the same (or similar behavior), where I can move the entire tar-ed and compressed tree from a local server to AWS S3, and then uncompress and untar my file to recreate the entire directory structure?</p>
  
  <p>The <code>tar</code> and <code>gzip</code> commands are not a part of the S3 CLI API so I need to find a solid way of moving a directory structure that can contain millions of files (which will happen possibly once a day).  It would be VERY slow to move and recreate everything without first tar-ing and compressing.</p>
  
  <p>NOTE: Just an FYI that when the data compiler runs, it always deletes the entire old tree and regenerates an entire new tree, resulting in completely new inodes for all directories and files.  This means "incremental" copies and syncs are not viable.  I need to move the whole tree, each time.</p>