title: Request had insufficient authentication scopes (403) when trying to write crawling data to Bigquery from pipeline of Scrapy
body: <p>I'm trying to build Scrapy crawler: spider will crawl data then in pipeline.py, the data will save to Bigquery. I built it by docker, setup crontab job and push to Google Cloud Server to daily running.</p>
  
  <p>The problem is when crontab executes scrapy crawler, it got "google.api_core.exceptions.Forbidden: 403 GET <a href="https://www.googleapis.com/bigquery/v2/projects/project_name/datasets/dataset_name/tables/table_name" rel="nofollow noreferrer">https://www.googleapis.com/bigquery/v2/projects/project_name/datasets/dataset_name/tables/table_name</a>: Request had insufficient authentication scopes.".</p>
  
  <p><strong>For more detail, when accessing to its container(docker exec -it ... /bin/bash) and execute it manually(scrapy crawl spider_name), it works like charm. The data appears in Bigquery.</strong></p>
  
  <p>I use service account (json file) having bigquery.admin role to setup GOOGLE_APPLICATION_CREDENTIALS.</p>
  
  <pre class="lang-py prettyprint-override"><code># spider file is fine
  
  # pipeline.py
  from google.cloud import bigquery
  import logging
  from scrapy.exceptions import DropItem
  ...
  
  class SpiderPipeline(object):
      def __init__(self):
  
          # BIGQUERY
          # Setup GOOGLE_APPLICATION_CREDENTIALS in docker file
          self.client = bigquery.Client()
          table_ref = self.client.dataset('dataset').table('data')
          self.table = self.client.get_table(table_ref)
  
      def process_item(self, item, spider):
          if item['key']:
  
              # BIGQUERY
              '''Order: key, source, lang, created, previous_price, lastest_price, rating, review_no, booking_no'''
              rows_to_insert = [( item['key'], item['source'], item['lang'])]
              error = self.client.insert_rows(self.table, rows_to_insert)
              if error == []:
                  logging.debug('...Save data to bigquery {}...'.format(item['key']))
                  # raise DropItem("Missing %s!" % item)
              else:
                  logging.debug('[Error upload to Bigquery]: {}'.format(error))
  
              return item
          raise DropItem("Missing %s!" % item)
  </code></pre>
  
  <p>In docker file:</p>
  
  <pre><code>FROM python:3.5-stretch
  
  WORKDIR /app
  
  COPY requirements.txt ./
  
  RUN pip install --trusted-host pypi.python.org -r requirements.txt
  
  COPY . /app
  
  # For Bigquery
  # key.json is already in right location
  ENV GOOGLE_APPLICATION_CREDENTIALS='/app/key.json'
  
  # Sheduler cron
  
  RUN apt-get update &amp;&amp; apt-get -y install cron
  
  # Add crontab file in the cron directory
  ADD crontab /etc/cron.d/s-cron
  
  # Give execution rights on the cron job
  RUN chmod 0644 /etc/cron.d/s-cron
  
  # Apply cron job
  RUN crontab /etc/cron.d/s-cron
  
  # Create the log file to be able to run tail
  RUN touch /var/log/cron.log
  
  # Run the command on container startup
  CMD cron &amp;&amp; tail -f /var/log/cron.log
  </code></pre>
  
  <p>In crontab:</p>
  
  <pre><code># Run once every day at midnight. Need empty line at the end to run.
  0 0 * * * cd /app &amp;&amp; /usr/local/bin/scrapy crawl spider &gt;&gt; /var/log/cron.log 2&gt;&amp;1
  
  </code></pre>
  
  <p>In conclusion, how to get crontab run crawler without 403 error. Thank anyone so much for support.</p>