title: Google Storage python api download in parallel
body: <p>It is trivial to download a large amount of files to your local machine in parallel using <a href="https://cloud.google.com/storage/docs/gsutil/commands/cp" rel="noreferrer">gsutil</a> by adding the <code>-m</code> flag:</p>
  
  <pre><code>gsutil -m cp gs://my-bucket/blob_prefix* .
  </code></pre>
  
  <p>In python, I can only download a single file at a time:</p>
  
  <pre><code>client = storage.Client()
  bucket = client.get_bucket(gs_bucket_name)
  blobs = [blob for blob in bucket.list_blobs(prefix=blob_prefix)]
  for blob in blobs:
      blob.download_to_filename(filename)
  </code></pre>
  
  <p>Preferably I'd like to download the data directly into memory (similar to <code>blob.download_as_string()</code>), and preferably into a generator.  Order doesn't really matter.</p>
  
  <p>Does this functionality exist somewhere within the python api?<br>
  If not, what would be the best way to do this?</p>
  
  <h3>Edit</h3>
  
  <p>I've implemented this hack:</p>
  
  <pre><code>def fetch_data_from_storage(fetch_pattern):
      """Download blobs to local first, then load them into Generator."""
      tmp_save_dir = os.path.join("/tmp", "tmp_gs_download")
      if os.path.isdir(tmp_save_dir):
          shutil.rmtree(tmp_save_dir)  # empty tmp dir first
      os.makedirs(tmp_save_dir)  # create tmp dir
  
      download_command = ["gsutil", "-m", "cp", "gs://{}/{}".format(bucket.name, fetch_pattern), tmp_save_dir]
      resp = subprocess.call(download_command)
  
      for file in os.listdir(tmp_save_dir):
          with open(os.path.join(tmp_save_dir, file), 'r') as f_data:
              content = json.load(f_data)
              yield content
  </code></pre>
  
  <p>Please advise if this was implemented in a better way somewhere.</p>