title: How to properly call another function while parsing current page?
body: <p>Currently, I am trying to scrape the IMDB movie pages and so far, I am able to scrape the movie details. However, to scrape user reviews, I have to move to a separate page from the movie page itself, which I am currently unsure if my approach to deal with it is correct.</p>
  <p>I am using a global variable (data[]) to store the different fields (which will ultimately be dumped into a json file for elasticsearch), however, one of my functions does not seem to be called, and I am not sure what am I missing</p>
  <p>The general logic that I am thinking of is as follows:</p>
  <ol>
  <li>Grab all the links in my list (parse_movie_list)</li>
  <li>For each Link, go inside and scrape details of that individual movie (parse_individual_movie)</li>
  <li>Once done, go to scrape the reviews (parse_individual_movie_reviews)</li>
  <li>Yield data</li>
  </ol>
  <hr />
  <pre><code>def parse_movie_list(self, response):
      links = response.css('span.lister-item-header a::attr(href)').extract()
      for link in links:
          yield response.follow(link, callback=self.parse_individual_movie)
  def parse_individual_movie(self, response):
  
      reviewlink = response.xpath(&quot;//div[contains(@id, 'quicklinksMainSection')]/a[3]/@href&quot;).extract()[0] or None
      data['reviewlink'] = reviewlink
      newlink = &quot;https://www.imdb.com&quot;+reviewlink
      request = scrapy.Request(newlink,callback=self.parse_individual_movie_reviews)        
  
      yield data
  
  def parse_individual_movie_reviews(self,response):
  
      reviews = response.xpath('//div[contains(@id, &quot;main&quot;)/section/div[2]/div[2]/div[1]/div[1]/div[1]/div[3]/div[1]/text()').extract() or None
  
      data['reviewLA'] =  [reviews.strip() for review in reviews] or None
  
      yield reviews
  </code></pre>
  <p>Right now, my problem appears to be that parse_individual_movie_reviews is not called (as the final json does not reflect the existence of reviewLA), ideally the goal is to have reviewLA containing the scraped user reviews from the document.</p>