title: Parse multiple large XML files and write to CSV
body: <p>I have around 1000 XML files each of 250 MB size. I need to extract some data from them and write to CSV. <strong>There cannot not be any duplicate entries.</strong></p>
  
  <p>I have a system with 4GB RAM and an AMD A8 processor.</p>
  
  <p>I have already gone through some previous posts here but they don't seem to answer my problem.</p>
  
  <p>I have already written the code in Python and tested it on a sample XML and it worked well. </p>
  
  <p>However it was very slow (almost 15 mins for each file) when I used it on all files and had to terminate the process midway.</p>
  
  <p>What can be an optimal solution to speed up the process?</p>
  
  <p>Here's the code</p>
  
  <pre><code>path='data/*.xml'
  t=[]
  for fname in glob.glob(path):
      print('Parsing ',fname)
      tree=ET.parse(fname)
      root=tree.getroot()
      x=root.findall('//Article/AuthorList//Author')
      for child in x:
          try:
              lastName=child.find('LastName').text
          except AttributeError:
              lastName=''
          try:
              foreName=child.find('ForeName').text
          except AttributeError:
              foreName=''
          t.append((lastName,foreName))
      print('Parsed ',fname)
  
  t=set(t)
  </code></pre>
  
  <p>I want the fastest method to get the entries <strong>without any duplicate values</strong>.
  (Maybe storing in some DB instead of variable t, Will storing each entry in DB speed up due to more free RAM ?- whatever be the method I need direction towards it)</p>