title: saving scraped items to json/csv/xml file using scrapy
body: <p>I am learning <a href="http://scrapy.org/" rel="nofollow noreferrer">Scrapy</a>(A web crawling framework) from their <a href="http://doc.scrapy.org/en/latest/" rel="nofollow noreferrer">official doc</a>.<br />
  By following examples and documentation I created my spider to scrape data using sitemap</p>
  <pre><code>from scrapy.contrib.spiders import SitemapSpider
  from scrapy.selector import Selector
  from MyProject1.items import MyProject1Item
  
  class MySpider(SitemapSpider):
      name=&quot;myspider&quot;
      sitemap_urls = ['http://www.somesite.com/sitemap.xml']
      sitemap_follow = ['/sitemapbrowsesection\d+']
      count=0
      def parse(self, response):
          self.count=self.count+1
          item=MyProject1Item()
          sel=Selector(response)
          item[&quot;exp&quot;]=sel.xpath('/html/head/title/text()').extract()
          print str(self.count)+&quot;:\t&quot;+response.url
          print sel.xpath('/html/head/title/text()').extract()
          print &quot;\n\n&quot;
  </code></pre>
  <p>I can see the scraped results on screen with log by running command<br />
  <code>scrapy crawl myspider</code><br />
  I can save scraped results to json/csv/xml file by adding option in command<br />
  <code>scrapy crawl myspider -o item.json -t json</code> for getting results in <a href="http://en.wikipedia.org/wiki/JSON" rel="nofollow noreferrer">json</a> file.</p>
  <p>My problem is that scrapy dumps scraped results to item.json only when scrapy is done with crawling. means  have to wait till crawling get over. so for big project I will have to wait for very very long time as I guess scrapy will write scraped results to json file after all the crawling to be done.</p>
  <p>I want scrapy to write to json file promptly or almost promptly or in the mid of crawling so I can see the results of sites which have crawled while scrapy is running.</p>
  <p>I know there must be something inbuilt in scrapy that I am not able to catch. I tried to get help from <a href="http://doc.scrapy.org/en/latest/topics/feed-exports.html" rel="nofollow noreferrer">http://doc.scrapy.org/en/latest/topics/feed-exports.html</a><br />
  and<br />
  <a href="http://doc.scrapy.org/en/latest/topics/exporters.html" rel="nofollow noreferrer">http://doc.scrapy.org/en/latest/topics/exporters.html</a><br />
  but not able to solve my problem. so I am looking for some help or example code otherwise I have to add few lines to <code>parse(self, response)</code> function to create json/csv file and write scrapped result into it.</p>