title: Scrapy - Visiting nested links and grabbing meta data from each level
body: <p>I am relatively new to scrapy and have been getting a lot of exceptions...
  Here is what I am trying to do:</p>
  
  <p>There 4 nested links that I want to grab data from:
  Let's say I have 5 items that I want to crawl in total. These items are </p>
  
  <pre><code>Industry=scrapy.Field()
  Company=scrapy.Field()
  Contact_First_name=scrapy.Field()
  Contact_Last_name=scrapy.Field()
  Website=scrapy.Field()
  </code></pre>
  
  <ol>
  <li>Now to begin crawling I would first have to get the Industry.
  The Industry xpath also contains the link to individual listings of companies that belong to their Industry segments.</li>
  <li>Next I want to use the Industry xpath and go into the link. This page does not contain any data that I want to crawl. But this page contains href links to individual companies that have their own basic info page. </li>
  <li>Using the href link from the listings page, I now arrive at one page that contains the information for one company. Now I want to scrape the Company, Address, and Website.
  There is another href link that I need to click in order to lead to Contact_First_Name, Contact_Last_Name. </li>
  <li><p>Using the href link, I now arrive at another page that contains the Contact_First_Name, and Contact_Last_Name </p>
  
  <p><code>After crawling all of these pages, I should have items that look somewhat like this: 
  Industry    Company    Website    Contact_First_Name     Contact_Last_Name
  Finance      JPMC       JP.com       Jamie                 Dimon
  Finance      BOA        BOA.com      Bryan                 Moynihan
  Technology   ADSK       ADSK.com     Carl                  Bass</code></p></li>
  </ol>
  
  <p>EDITED</p>
  
  <p>Here is the code that is working. Anzel's recommendations really helped out but i realized the subclass allowed_domains was wrong which stopped the nested links from following through. Once I changed it, it works.  </p>
  
  <pre><code>class PschamberSpider(scrapy.Spider):
      name="pschamber"
      allowed_domains = ["cm.pschamber.com"]
      start_urls = ["http://cm.pschamber.com/list/"]
  
  
      def parse(self, response):
          item = PschamberItem()
          for sel in response.xpath('//*[@id="mn-ql"]/ul/li/a'):
              # xpath and xpath().extract() will return a list
              # extract()[0] will return the first element in the list
              item['Industry'] = sel.xpath('text()').extract()
              # another mistake you made here
              # you're trying to call scrapy.Request(LIST of hrefs) which will fail
              # scrapy.Request only takes a url string, not list
              # another big mistake is you're trying to yield the item,
              # whereas you should yield the Request object
              yield scrapy.Request(sel.xpath('@href').extract()[0], callback=self.parse_2, meta={'item': item})
  
      # another mistake, your callback function DOESNT take item as argument
      def parse_2(self, response):
          for sel in response.xpath('.//*[@id="mn-members"]/div/div/div/div/div/a').extract():
              # you can access your response meta like this
              item=response.meta['item']
              item['Company'] = sel.xpath('text()').extract()
              yield scrapy.Request(sel.xpath('@href').extract()[0], callback=self.parse_3, meta={'item': item})
  
              # again, yield the Request object
  
  
      def parse_3(self, response):
          item=response.meta['item']
          item['Website'] = response.xpath('.//[@id="mn-memberinfo-block-website"]/a/@href').extract()
          # OK, finally assume you're done, just return the item object
          return item   
  </code></pre>