title: How can i export scraped data to csv file in the right format?
body: <p>I made an improvement to my code according to <a href="https://stackoverflow.com/a/45958106/8476752">this</a> suggestion from @paultrmbrth. what i need is to scrape data from pages that are similar to <a href="http://www.imdb.com/title/tt0092455/trivia?tab=mc&amp;ref_=tt_trv_cnn" rel="nofollow noreferrer">this</a> and <a href="http://www.imdb.com/title/tt0094118/trivia?tab=mc&amp;ref_=tt_trv_cnn" rel="nofollow noreferrer">this</a> one and i want the csv output to be like the picture below.<a href="https://i.stack.imgur.com/S50q4.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/S50q4.png" alt="enter image description here"></a> </p>
  
  <p>But my code's csv output is little messy, like this:
  <a href="https://i.stack.imgur.com/EEcRE.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/EEcRE.png" alt="enter image description here"></a></p>
  
  <p>I have two questions, Is there anyway that the csv output can be like the first picture? and my second question is, i want the movie tittle to be scrapped too, Please give me a hint or provide to me a code that i can use to scrape the movie title and the contents. </p>
  
  <p><strong>UPDATE</strong><br>
  The problem has been solved by Tarun Lalwani perfectly. But Now, the csv File's Header only contains the first scraped url categories. for example when i try to scrape <a href="http://www.imdb.com/title/tt0093777/trivia?tab=mc&amp;ref_=tt_trv_cnn" rel="nofollow noreferrer">this webpage</a> which has <code>References, Referenced in, Features, Featured in and Spoofed in</code> categories and <a href="http://www.imdb.com/title/tt0096874/trivia?tab=mc&amp;ref_=tt_trv_cnn" rel="nofollow noreferrer">this webpage</a> which has  <code>Follows, Followed by, Edited from, Edited into, Spin-off, References, Referenced in, Features, Featured in, Spoofs and Spoofed in</code>  categories then the csv output file header will only contain the first webpage's categories i.e <code>References, Referenced in, Features, Featured in and Spoofed in</code>  so some categories from the 2nd webpage like <code>Follows, Followed by, Edited from, Edited into and Spoofs</code>will not be on the output csv file header so is its contents.<br>
  Here is the code i used:</p>
  
  <pre><code>import scrapy
  
  
  class ExampleSpider(scrapy.Spider):
      name = "example"
      allowed_domains = ["imdb.com"]
      start_urls = (
          'http://www.imdb.com/title/tt0093777/trivia?tab=mc&amp;ref_=tt_trv_cnn',
          'http://www.imdb.com/title/tt0096874/trivia?tab=mc&amp;ref_=tt_trv_cnn',
      )
  
      def parse(self, response):
          item = {}
          for cnt, h4 in enumerate(response.css('div.list &gt; h4.li_group'), start=1):
              item['Title'] = response.css("h3[itemprop='name'] a::text").extract_first()
              key = h4.xpath('normalize-space()').get().strip()
              if key in ['Follows', 'Followed by', 'Edited into', 'Spun-off from', 'Spin-off', 'Referenced in',
                         'Featured in', 'Spoofed in', 'References', 'Spoofs', 'Version of', 'Remade as', 'Edited from',
                         'Features']:
                  values = h4.xpath('following-sibling::div[count(preceding-sibling::h4)=$cnt]', cnt=cnt).xpath(
                      'string(.//a)').getall(),
                  item[key] = values
          yield item
  </code></pre>
  
  <p>and here is <code>exporters.py</code> file:</p>
  
  <pre><code>try:
      from itertools import zip_longest as zip_longest
  except:
      from itertools import izip_longest as zip_longest
  from scrapy.exporters import CsvItemExporter
  from scrapy.conf import settings
  
  
  class NewLineRowCsvItemExporter(CsvItemExporter):
  
      def __init__(self, file, include_headers_line=True, join_multivalued=',', **kwargs):
          super(NewLineRowCsvItemExporter, self).__init__(file, include_headers_line, join_multivalued, **kwargs)
  
      def export_item(self, item):
          if self._headers_not_written:
              self._headers_not_written = False
              self._write_headers_and_set_fields_to_export(item)
  
          fields = self._get_serialized_fields(item, default_value='',
                                               include_empty=True)
          values = list(self._build_row(x for _, x in fields))
  
          values = [
              (val[0] if len(val) == 1 and type(val[0]) in (list, tuple) else val)
              if type(val) in (list, tuple)
              else (val, )
              for val in values]
  
          multi_row = zip_longest(*values, fillvalue='')
  
          for row in multi_row:
              self.csv_writer.writerow([unicode(s).encode("utf-8") for s in row])
  </code></pre>
  
  <p>What I'm trying to achieve is i want all these categories to be on the csv output header.</p>
  
  <pre><code>'Follows', 'Followed by', 'Edited into', 'Spun-off from', 'Spin-off', 'Referenced in',
  'Featured in', 'Spoofed in', 'References', 'Spoofs', 'Version of', 'Remade as', 'Edited from', 'Features'   
  </code></pre>
  
  <p><strong>Any help would be appreciated.</strong></p>