title: How to parse huge XML with Python and XSLT file iteratively and write to CSV
body: <p>I'm unable to flatten and transform an XML to CSV file using XSLT, when dealing with large XML files.</p>
  
  <p>Currently, I'm parsing a nested XML file with <code>lxml</code> using a XSL file to flatten the output and then I write the output to a CSV file.</p>
  
  <p>My XML looks something like this:</p>
  
  <pre><code>&lt;root&gt;
      &lt;level1&gt;
          &lt;level2&gt;
              &lt;topid&gt;1&lt;/topid&gt;
              &lt;level3&gt;
                  &lt;subtopid&gt;1&lt;/topid&gt;
                  &lt;level4&gt;
                      &lt;subid&gt;1&lt;/id&gt;
                      &lt;descr&gt;test&lt;/descr&gt;
                  &lt;/level4&gt;
                  &lt;level4&gt;
                      &lt;subid&gt;2&lt;/id&gt;
                      &lt;descr&gt;test2&lt;/descr&gt;
                  &lt;/level4&gt;
                  ...
              &lt;/level3&gt;
              ...
          &lt;/level2&gt;
      &lt;/level1&gt;
  &lt;/root&gt;
  </code></pre>
  
  <p>I want to end up with the following CSV file:</p>
  
  <pre><code>topid,subtopid,subid,descr
  1,1,1,test
  1,1,2,test2
  ....
  </code></pre>
  
  <p>My XSLT:</p>
  
  <pre><code>&lt;?xml version="1.0" encoding="UTF-8" ?&gt;
  &lt;xsl:stylesheet version="2.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"&gt;
  &lt;xsl:output method="text" encoding="utf-8" use-character-maps="map"/&gt;
  &lt;xsl:character-map name="map"&gt;
      &lt;xsl:output-character character="," string=" "/&gt;
  &lt;/xsl:character-map&gt;
  
  &lt;xsl:strip-space elements="*"/&gt;
  &lt;xsl:variable name="delimiter" select="','"/&gt;
  &lt;xsl:variable name="newline" select="'&amp;#xd;'" /&gt;
  
  &lt;xsl:template match="/root"&gt;
      &lt;xsl:text&gt;topid,subtopid,subid,descr&lt;/xsl:text&gt;
      &lt;xsl:value-of select="$newline" /&gt;
  
      &lt;xsl:for-each select="level1/level2/level3/level4"&gt;
          &lt;xsl:value-of select="ancestor::root/level1/level2/topid" /&gt;
          &lt;xsl:value-of select="$delimiter" /&gt;
          &lt;xsl:value-of select="ancestor::root/level1/level2/level3/subtopid" /&gt;
          &lt;xsl:value-of select="$delimiter" /&gt;
          &lt;xsl:value-of select="subid" /&gt;
          &lt;xsl:value-of select="$delimiter" /&gt;
          &lt;xsl:value-of select="descr" /&gt;
          &lt;xsl:value-of select="$newline" /&gt;
      &lt;/xsl:for-each&gt;
  &lt;/xsl:template&gt;
  </code></pre>
  
  <p></p>
  
  <p>My Python code:</p>
  
  <pre><code>import lxml.etree as ET
  
  xsltfile = ET.XSLT(ET.parse('transactions.xsl'))
  xmlfile  = ET.parse('myxmlfile.xml')
  output   = xsltfile(xmlfile).write_output('output.csv')
  </code></pre>
  
  <p>This works great for small files, but now I want to do the same with an XML file of +- 2.5gb. Using etree.parse will load it into memory, which won't work with larger files obviously.</p>
  
  <p>I want to iterate somewhere, so I'm not loading the XML file into memory and  write to CSV line for line, while still making use of the XSLT for transforming. I'm using the XSLT file because it's the only way I know (now)  how to flatten a nested XML file.</p>