title: Yield Request call produce weird result in recursive method with scrapy
body: <p>I'm trying to scrap all departures and arrivals in one day from all airports in all country using Python and Scrapy.</p>
  
  <p>The JSON database used by this famous site (flight radar) need to query page by page when departure or arrival is > 100 in one airport. I also compute a timestamp based on an actual day UTC for the query.</p>
  
  <p>I try to create a database with this hierarchy: </p>
  
  <pre><code>country 1
   - airport 1
      - departures
        - page 1
        - page ...
      - arrivals
        - page 1
        - page ...
  - airport 2
      - departures
        - page 1
        - page ...
      - arrivals
        - page 
        - page ...
  ...
  </code></pre>
  
  <p>I use two methods to compute timestamp and url query by page : </p>
  
  <pre><code>def compute_timestamp(self):
      from datetime import datetime, date
      import calendar
      # +/- 24 heures
      d = date(2017, 4, 27)
      timestamp = calendar.timegm(d.timetuple())
      return timestamp
  
  def build_api_call(self,code,page,timestamp):
      return 'https://api.flightradar24.com/common/v1/airport.json?code={code}&amp;plugin\[\]=&amp;plugin-setting\[schedule\]\[mode\]=&amp;plugin-setting\[schedule\]\[timestamp\]={timestamp}&amp;page={page}&amp;limit=100&amp;token='.format(
          code=code, page=page, timestamp=timestamp)
  </code></pre>
  
  <p>I store result into <code>CountryItem</code>, which contain lots of <code>AirportItem</code> into airports. My <code>item.py</code> is :</p>
  
  <pre><code>class CountryItem(scrapy.Item):
      name = scrapy.Field()
      link = scrapy.Field()
      num_airports = scrapy.Field()
      airports = scrapy.Field()
      other_url= scrapy.Field()
      last_updated = scrapy.Field(serializer=str)
  
  class AirportItem(scrapy.Item):
      name = scrapy.Field()
      code_little = scrapy.Field()
      code_total = scrapy.Field()
      lat = scrapy.Field()
      lon = scrapy.Field()
      link = scrapy.Field()
      departures = scrapy.Field()
      arrivals = scrapy.Field()
  </code></pre>
  
  <p>My main parse builds a Country item for all countries (i limit here to Israel for example). Next, I yield for each country a <code>scrapy.Request</code> to scrape airports.</p>
  
  <pre><code>###################################
  # MAIN PARSE
  ####################################
  def parse(self, response):
      count_country = 0
      countries = []
      for country in response.xpath('//a[@data-country]'):
          item = CountryItem()
          url =  country.xpath('./@href').extract()
          name = country.xpath('./@title').extract()
          item['link'] = url[0]
          item['name'] = name[0]
          item['airports'] = []
          count_country += 1
          if name[0] == "Israel":
              countries.append(item)
              self.logger.info("Country name : %s with link %s" , item['name'] , item['link'])
              yield scrapy.Request(url[0],meta={'my_country_item':item}, callback=self.parse_airports)
  </code></pre>
  
  <p>This method scrape information for each airport, and also call for each airport a <code>scrapy.request</code> with airport url to scrape departures and arrivals : </p>
  
  <pre><code>  ###################################
  # PARSE EACH AIRPORT
  ####################################
  def parse_airports(self, response):
      item = response.meta['my_country_item']
      item['airports'] = []
  
      for airport in response.xpath('//a[@data-iata]'):
          url = airport.xpath('./@href').extract()
          iata = airport.xpath('./@data-iata').extract()
          iatabis = airport.xpath('./small/text()').extract()
          name = ''.join(airport.xpath('./text()').extract()).strip()
          lat = airport.xpath("./@data-lat").extract()
          lon = airport.xpath("./@data-lon").extract()
          iAirport = AirportItem()
          iAirport['name'] = self.clean_html(name)
          iAirport['link'] = url[0]
          iAirport['lat'] = lat[0]
          iAirport['lon'] = lon[0]
          iAirport['code_little'] = iata[0]
          iAirport['code_total'] = iatabis[0]
  
          item['airports'].append(iAirport)
  
      urls = []
      for airport in item['airports']:
          json_url = self.build_api_call(airport['code_little'], 1, self.compute_timestamp())
          urls.append(json_url)
      if not urls:
          return item
  
      # start with first url
      next_url = urls.pop()
      return scrapy.Request(next_url, self.parse_schedule, meta={'airport_item': item, 'airport_urls': urls, 'i': 0})
  </code></pre>
  
  <p>With the recursive method <code>parse_schedule</code> i add each airport to country item. SO members already <a href="https://stackoverflow.com/questions/41634126/multiple-nested-request-with-scrapy">help me</a> on this point.</p>
  
  <pre><code>###################################
  # PARSE EACH AIRPORT OF COUNTRY
  ###################################
  def parse_schedule(self, response):
          """we want to loop this continuously to build every departure and arrivals requests"""
          item = response.meta['airport_item']
          i = response.meta['i']
          urls = response.meta['airport_urls']
  
          urls_departures, urls_arrivals = self.compute_urls_by_page(response, item['airports'][i]['name'], item['airports'][i]['code_little'])
  
          print("urls_departures = ", len(urls_departures))
          print("urls_arrivals = ", len(urls_arrivals))
  
          ## YIELD NOT CALLED
          yield scrapy.Request(response.url, self.parse_departures_page, meta={'airport_item': item, 'page_urls': urls_departures, 'i':0 , 'p': 0}, dont_filter=True)
  
          # now do next schedule items
          if not urls:
              yield item
              return
          url = urls.pop()
  
          yield scrapy.Request(url, self.parse_schedule, meta={'airport_item': item, 'airport_urls': urls, 'i': i + 1})
  </code></pre>
  
  <p>the <code>self.compute_urls_by_page</code> method compute correct URLs to retrieve all departure and arrivals for one airport. </p>
  
  <pre><code>###################################
  # PARSE EACH DEPARTURES / ARRIVALS
  ###################################
  def parse_departures_page(self, response):
      item = response.meta['airport_item']
      p = response.meta['p']
      i = response.meta['i']
      page_urls = response.meta['page_urls']
  
      print("PAGE URL = ", page_urls)
  
      if not page_urls:
          yield item
          return
      page_url = page_urls.pop()
  
      print("GET PAGE FOR  ", item['airports'][i]['name'], "&gt;&gt; ", p)
  
      jsonload = json.loads(response.body_as_unicode())
      json_expression = jmespath.compile("result.response.airport.pluginData.schedule.departures.data")
      item['airports'][i]['departures'] = json_expression.search(jsonload)
  
      yield scrapy.Request(page_url, self.parse_departures_page, meta={'airport_item': item, 'page_urls': page_urls, 'i': i, 'p': p + 1})
  </code></pre>
  
  <p>Next, the first yield in <code>parse_schedule</code> which normally call <code>self.parse_departure_page</code> recursive method produces weird results. <strong>Scrapy call this method, but I collect the departures page for only one airport i don't understand why...</strong> I have probably an ordering error in my request or yield source code, so perhaps you could help me to find out.</p>
  
  <p>The complete code is on GitHub <a href="https://github.com/IDEES-Rouen/Flight-Scrapping/tree/master/flight/flight_project" rel="nofollow noreferrer">https://github.com/IDEES-Rouen/Flight-Scrapping/tree/master/flight/flight_project</a></p>
  
  <p>You could run it using <code>scrapy cawl airports</code> commands.</p>
  
  <p><strong>Update 1 :</strong> </p>
  
  <p>I try to answer the question alone using <code>yield from</code>, without success as you can see answer bottom ... so if you have an idea?</p>