title: Scrapy callback after redirect
body: <p>I have a very basic scrapy spider, which grabs urls from the file and then downloads them. The only problem is that some of them got redirected to a slightly modified url within same domain. I want to get them in my callback function using <em>response.meta</em>, and it works on a normal urls, but then url is redirected callback doesn't seem to get called. How can I fix it?
  Here's my code.</p>
  
  <pre><code>from scrapy.contrib.spiders import CrawlSpider
  from scrapy import log
  from scrapy import Request
  class DmozSpider(CrawlSpider):
      name = "dmoz"
      handle_httpstatus_list = [302]
      allowed_domains = ["http://www.exmaple.net/"]) 
      f = open("C:\\python27\\1a.csv",'r')
      url = 'http://www.exmaple.net/Query?indx='
      start_urls = [url+row for row in f.readlines()]
      def parse(self, response):
              print response.meta.get('redirect_urls', [response.url])
              print response.status 
              print (response.headers.get('Location'))
  </code></pre>
  
  <p>I've also tried something like that:</p>
  
  <pre><code>def parse(self, response):
           return Request(response.url, meta={'dont_redirect': True, 'handle_httpstatus_list':     [302]}, callback=self.parse_my_url)
  def parse_my_url(self, response):
          print response.status 
          print (response.headers.get('Location'))
  </code></pre>
  
  <p>And it doesn't work either.</p>