title: scrapy - Get final redirected URL
body: <p>I am trying to get the final redirected URL in scrapy. For example, if an anchor tag has a specific format:</p>
  
  <pre><code>&lt;a href="http://www.example.com/index.php" class="FOO_X_Y_Z" /&gt;
  </code></pre>
  
  <p>Then I need to get the URL that URL redirects to (if it does, if its 200 then OK). For example, I get the appropriate anchor tags like this:</p>
  
  <pre><code>def parse (self, response)  
      hxs     = HtmlXPathSelector (response);
      anchors = hxs.select("//a[@class='FOO_X_Y_Z']/@href");
  
      // Lets assume anchor contains the actual link (http://...)
      for anchor in anchors:
          final_url = get_final_url (anchor);   // &lt;&lt; I would need something like this
  
          // Save final_url
  </code></pre>
  
  <p>So if I visited <code>http://www.example.com/index.php</code> and that would send me through 10 redirects and finally it would stop at <code>http://www.example.com/final.php</code> - this is what I would need <code>get_final_url()</code> to return. </p>
  
  <p>I thought of hacking my way to a solution but am asking here to see if scrapy has one already provided?</p>