title: Getting spider on Scrapy Cloud to store files on Google Cloud Storage using GCSFilesStore and getting ImportError
body: <p>Deploying a spider to Scraping Cloud. It gathers download links for files and should save those files in a Google Cloud bucket. It works when running locally. But when deploying to Scraping Hub it return the following errors: </p>
  
  <pre><code>Traceback (most recent call last):
    File "/usr/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1299, in _inlineCallbacks
      result = g.send(result)
    File "/usr/local/lib/python2.7/site-packages/scrapy/crawler.py", line 95, in crawl
      six.reraise(*exc_info)
    File "/usr/local/lib/python2.7/site-packages/scrapy/crawler.py", line 77, in crawl
      self.engine = self._create_engine()
    File "/usr/local/lib/python2.7/site-packages/scrapy/crawler.py", line 102, in _create_engine
      return ExecutionEngine(self, lambda _: self.stop())
    File "/usr/local/lib/python2.7/site-packages/scrapy/core/engine.py", line 70, in __init__
      self.scraper = Scraper(crawler)
    File "/usr/local/lib/python2.7/site-packages/scrapy/core/scraper.py", line 71, in __init__
      self.itemproc = itemproc_cls.from_crawler(crawler)
    File "/usr/local/lib/python2.7/site-packages/scrapy/middleware.py", line 58, in from_crawler
      return cls.from_settings(crawler.settings, crawler)
    File "/usr/local/lib/python2.7/site-packages/scrapy/middleware.py", line 34, in from_settings
      mwcls = load_object(clspath)
    File "/usr/local/lib/python2.7/site-packages/scrapy/utils/misc.py", line 44, in load_object
      mod = import_module(module)
    File "/usr/local/lib/python2.7/importlib/__init__.py", line 37, in import_module
      __import__(name)
    File "/app/__main__.egg/tceq_crawler/pipelines.py", line 4, in &lt;module&gt;
  ImportError: cannot import name GCSFilesStore
  </code></pre>
  
  <p>It runs locally and is able to upload files to the GCloud bucket. There's a service account created just for the spider. I exported the JSON credentials file and pasted it directly into the credentials object. </p>
  
  <p>In the YML file I specified the requirements file. In the requirements.txt file I specified all the packages that are required.</p>
  
  <p>At this point I'm not sure why it can't import GCSFilesStore on Scrapy Cloud when it imports FilesPipeline fine.</p>
  
  <p>Here is my code: </p>
  
  <p><code>pipelines.py</code></p>
  
  <pre><code># -*- coding: utf-8 -*-
  
  from scrapy.pipelines.files import FilesPipeline
  from scrapy.pipelines.files import GCSFilesStore
  from scrapy.http.request import Request
  
  # Define your item pipelines here
  #
  # Don't forget to add your pipeline to the ITEM_PIPELINES setting
  # See: https://doc.scrapy.org/en/latest/topics/item-pipeline.html
  
  class TceqCrawlerPipeline(object):
      def process_item(self, item, spider):
          return item
  
  class CustomFileNamePipeline(FilesPipeline):
  
      def get_media_requests(self, item, info):
          return [Request(x, meta={'file_name': item["file_name"]})
                  for x in item.get('file_urls', [])]
  
      def file_path(self, request, response=None, info=None):
          return request.meta['file_name']
  
  class GCSFilesStoreJSON(GCSFilesStore):
      CREDENTIALS = {
          "type": "service_account",
          "project_id": "project_id",
          "private_key_id": "FROM JSON FILE",
          "private_key": "-----BEGIN PRIVATE KEY-----FROM JSON FILE-----END PRIVATE KEY-----\n",
          "client_email": "FROM JSON FILE",
          "client_id": "FROM JSON FILE",
          "auth_uri": "https://accounts.google.com/o/oauth2/auth",
          "token_uri": "https://oauth2.googleapis.com/token",
          "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
          "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/FROM JSON FILE"
      }
      def __init__(self, uri):
          from google.cloud import storage
          client = storage.Client.from_service_account_info(self.CREDENTIALS)
          bucket, prefix = uri[5:].split('/', 1)
          self.bucket = client.bucket(bucket)
          self.prefix = prefix
  
  class GCSFilePipeline(FilesPipeline):
      def __init__(self, store_uri, download_func=None, settings=None):
          super(GCSFilePipeline, self).__init__(store_uri,download_func,settings)
  
  </code></pre>
  
  <p><code>settings.py</code></p>
  
  <pre><code>DOWNLOADER_MIDDLEWARES = {'scrapy_crawlera.CrawleraMiddleware': 300}
  CRAWLERA_ENABLED = True
  CRAWLERA_APIKEY = 'API_KEY'
  
  
  ITEM_PIPELINES = {
      'tceq_crawler.pipelines.GCSFilePipeline': 1,
      'tceq_crawler.pipelines.CustomFileNamePipeline': 200
  }
  ##GOOGLE CLOUD STORAGE
  FILES_STORE = 'gs://bucket-name/'
  GCS_PROJECT_ID = 'project_id'
  
  </code></pre>
  
  <p><code>scrapinghub.yml</code></p>
  
  <pre><code>project: 12345
  
  requirements:
    file: requirements.txt
  </code></pre>
  
  <p><code>requirements.txt</code></p>
  
  <pre><code>pyasn1==0.4.1
  google-api-core==1.8.1                     
  google-auth==1.6.3                     
  google-cloud-core==0.29.1                    
  google-cloud-storage==1.14.0                  
  google-resumable-media==0.3.2                    
  googleapis-common-protos==1.5.8
  </code></pre>