title: How to add Headers to Scrapy CrawlSpider Requests?
body: <p>I'm working with the CrawlSpider class to crawl a website and I would like to modify the headers that are sent in each request.  Specifically, I would like to add the referer to the request.</p>
  
  <p>As per <a href="https://stackoverflow.com/questions/12054958/scrapyhow-to-print-request-referrer/12055059#12055059">this question</a>, I checked </p>
  
  <pre><code>response.request.headers.get('Referer', None)
  </code></pre>
  
  <p>in my response parsing function and the <code>Referer</code> header is not present.  I assume that means the Referer is not being submitted in the request (unless the website doesn't return it, I'm not sure on that).</p>
  
  <p>I haven't been able to figure out how to modify the headers of a request.  Again, my spider is derived from CrawlSpider.  Overriding CrawlSpider's <code>_requests_to_follow</code> or specifying a <code>process_request</code> callback for a rule will not work because the referer is not in scope at those times.</p>
  
  <p>Does anyone know how to modify request headers dynamically?</p>