<p><strong>How would you go about organizing a process of zipping objects that reside an object storage?</strong></p>
<p>For context, our users sometimes request an extraction of their entire data from the app - think of &quot;<a href="https://help.twitter.com/en/managing-your-account/accessing-your-twitter-data" rel="nofollow noreferrer">Downloading Twitter archive</a>&quot; feature of Twitter.</p>
<p>Our users are able to upload files, so the extracted data must contain files stored in a object storage (Google Cloud Storage). The requested data must be packed into a single .zip archive.</p>
<p>A naive approach would look like this:</p>
<ol>
<li>download all files from object storage on a disk,</li>
<li>zip all files into an archive,</li>
<li>put it .zip back on an object storage,</li>
<li>send a link to download the .zip file back to user.</li>
</ol>
<p>However, there are multiple disadvantages here:</p>
<ol>
<li>sometimes files for even single user add up to gigabytes,</li>
<li>if the process of zipping is interrupted, it has to start over.</li>
</ol>
<p>What's a reasonable way to design a process of generating a .zip archive with user files, that originally reside on an object storage?</p>

oldhomemovie
<p>Unfortunately, your naive approach is the only way because Cloud Storage offers no compute abilities. Archiving files requires compute, memory, and temporary storage.</p>
<p>The key item is to choose a service, such as Compute Engine, that can meet your file processing requirements: multi-gig files, fast processing (compression), and high-speed networking.</p>
<p>Another issue will be the time that it takes to download, zip, and upload. That means using an asynchronous event-based design. Start file processing and notify the user (email, message, web inbox, etc) once the file processing is complete.</p>
<p>You could make the process synchronous and display a progress bar, but that will complicate the design.</p>
