title: Scraping to a csv using Scrapy & Python, not appending correct values
body: <p>I'm trying to save some data to a CSV file using Scrapy, I run 3 Scraping proceses inside one of my spiders. But the CSV file that the crawl generates only saves the first data I extracted with the first function <strong>"parse"</strong>, what I need is to save the data of the third one <strong>"parse_following_urls_2"</strong>.</p>
  
  <p>I even tried using Pandas to generate the CSV file but Scrapy seems to be having trouble with Pandas.</p>
  
  <p><strong>To run the code</strong> <code>scrapy crawl goodmans -o goodmans.csv -t csv</code></p>
  
  <p>What can I do? Here is my code:</p>
  
  <pre><code>import os
  import scrapy
  from ..items import TutorialItem
  import pandas as pd
  from scrapy.selector import Selector
  from scrapy.http import Request
  BASE_DIR = os.path.dirname(os.path.abspath(__file__))
  
  class GoodmanSpider(scrapy.Spider):
  name = "goodmans"
  start_urls = ['http://www.goodmans.net/d/1706/brands.htm']
  supplier_urls = []
  
  def parse(self, response):
      items = TutorialItem()
      all_data = response.css('.SubDepartments')
      for data in all_data:
          category = data.css('.SubDepartments a::text').extract()
          category_url = data.css('.SubDepartments a::attr(href)').extract()
          items['category'] = category
          items['category_url'] = category_url
          for cat, url in zip(category, category_url):
              item = dict(category=cat, category_url=url)
              yield item
              yield Request(url, callback=self.parse_following_urls, dont_filter=True)
              # yield {'Category': category, 'url': category_url}
  
  def parse_following_urls(self, response):  
      items = TutorialItem()
      all_data = response.css('.SubDepartments')
      for data in all_data:         
          article_url = data.css('.SubDepartments a::attr(href)').extract() 
          items['article_url'] = article_url
          for url in article_url:
              item = dict(article_url=url)        
              # yield item    
              yield Request(url, callback=self.parse_following_urls_2, dont_filter=True)
  
  def parse_following_urls_2(self, response):  
      items = TutorialItem()
      all_data = response.css('.recin')
      for data in all_data:
          article_url_2 = data.css('.recit a').xpath('@href').extract() 
          article_name = data.css('.recit span::text').extract()
          article_price = data.css('.price span::text').extract()
          article_item_num = data.css('.itemno span::text').extract() 
          article_image = data.css('.linkit img').xpath('@src').extract()
          items['article_url_2'] = article_url_2
          items['article_name'] = article_name
          items['article_price'] = article_price
          items['article_item_num'] = article_item_num
          items['article_image'] = article_image
          for au, an, ap, ain, ai in zip(article_url_2, article_name, article_price, article_item_num, article_image):
              item = dict(article_url_2=au, article_name=an, article_price=ap, article_item_num=ain, article_image=ai)
              df_result = pd.DataFrame.from_dict(item)
              path_file = os.path.join(BASE_DIR, 'csv/supplier_product_database.csv')
              my_csv = df_result.to_csv(path_file, index=None, header=True)
              yield item          
  </code></pre>
  
  <p><strong>items.py file</strong></p>
  
  <p><a href="https://i.stack.imgur.com/ABTN0.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/ABTN0.png" alt="itemps.py"></a></p>