title: lxml iterparse using up memory when setting tag argument and setting events to "start"
body: <p>I noticed there are a lot of questions &amp; answers regarding the memory consumption of <code>iterparse</code> (e.g. <a href="https://stackoverflow.com/questions/12160418/why-is-lxml-etree-iterparse-eating-up-all-my-memory">this one</a>), but even after using the cache-clearing code, my memory usage still blows up when I set <code>tag</code> to some level (not too high, though not too low either) and assigning &quot;start&quot; to <code>event</code>.</p>
  <h2>Problem statement</h2>
  <p>I am parsing an xml file ~30 GBs with <code>lxml.etree.iterparse</code>.  Since a BFS is the most appropriate solution for my task, I tried to set <code>tag</code> in <code>iterparse</code> and also only consider <code>start</code>.  Then, when I use <code>iter</code> and <code>next</code> to test, the memory was used up (Jupyter crashed) when I run the block for the third time.  The cause is most probably not that the iterator is not able to find the tag; when I added a typo into the tag name, it ran for way longer time than the crash.</p>
  <p>However, this issue does not happen when I either (1) change <code>start</code> to <code>end</code>; or (2) change the tag name to something at a lower level (e.g. <code>SimpleAllele</code>, etc.); or (3) just remove the tag and use if-else.</p>
  <p>I am able to proceed with my task right now, but I just wanted to learn why this happens.  Through a quick search I don't recognize any significant cost of memory in my procedure, but it is highly possible that I am not aware of some intrinsic mechanisms of <code>iterparse</code>.</p>
  <h2>Code</h2>
  <p>My current code is as simple as below:</p>
  <pre class="lang-py prettyprint-override"><code>from lxml import etree
  
  file = CLINVAR_DIR+'/xml/ClinVarVariationRelease_00-latest.xml'
  file_handler = etree.iterparse(file, events=(&quot;start&quot;, ), tag=&quot;VariationArchive&quot;)
  
  # ------------ Split into two cells and run the following multiple times -------------
  
  event, elem = next(file_handler)
  
  print(event, elem.tag, elem.attrib, list(elem))
  for ancestor in elem.xpath('ancestor-or-self::*'):
      while ancestor.getprevious() is not None:
          del ancestor.getparent()[0]
  
  elem.clear()
  del elem
  </code></pre>
  <h2>Data</h2>
  <p>The data I'm using can be downloaded <a href="https://ftp.ncbi.nlm.nih.gov/pub/clinvar/xml/clinvar_variation/ClinVarVariationRelease_00-latest.xml.gz" rel="nofollow noreferrer">here</a> (note this is huge), and a demo would look like <a href="https://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?db=clinvar&amp;rettype=vcv&amp;id=VCV000638577" rel="nofollow noreferrer">this</a>.</p>