title: Crawling redirected url in scrapy
body: <p>I am working in scrapy.</p>
  
  <p>I am fetching a site which consists of a list of urls.</p>
  
  <p>So I requested the main url in <code>start_url</code> and I got all the href tags(links to fetch data) in  a list, I again requested each and every url in the list further for fetching data, but some of the urls are redirecting like below:</p>
  
  <pre><code>Redirecting (301) to &lt;GET example.com/sch/mobile-68745.php&gt; from Redirecting (301) to &lt;GET example.com/sch/mobile-8974.php&gt;
  </code></pre>
  
  <p>I came to know that scrapy ignores the redirected links, but I want to catch the redirected url and want to scrape the same like the url with 200 status</p>
  
  <p>Is there anyway to catch that redirect url and scrape the data from them, I mean do we need disable redirect middleware? Or do we need to use any meta tag in Request command, can youu provide me an example of that?</p>