title: Improve speed parsing XML with elements and namespace, into Pandas
body: <p>So I have a <code>52M</code> xml file, which consists of <code>115139</code> elements.</p>
  
  <pre><code>from lxml import etree
  tree  = etree.parse(file)
  root  = tree.getroot()
  In [76]: len(root)
  Out[76]: 115139
  </code></pre>
  
  <p>I have this function that iterates over the elements within <code>root</code> and inserts each parsed element inside a Pandas DataFrame.</p>
  
  <pre><code>def fnc_parse_xml(file, columns):
      start = datetime.datetime.now()
      df    = pd.DataFrame(columns=columns)
      tree  = etree.parse(file)
      root  = tree.getroot()
      xmlns = './/{' + root.nsmap[None] + '}'
  
      for loc,e in enumerate(root):
          tot = []
          for column in columns:
              tot.append(e.find(xmlns + column).text)
          df.at[loc,columns] = tot
  
      end = datetime.datetime.now()
      diff = end-start
      return df,diff
  </code></pre>
  
  <p>This process works but it takes a lot of time. I have an i7 with 16GB of RAM.</p>
  
  <pre><code>In [75]: diff.total_seconds()/60                                                                                                      
  Out[75]: 36.41769186666667
  In [77]: len(df)                                                                                                                      
  Out[77]: 115139
  </code></pre>
  
  <p>I'm pretty sure there is a better way of parsing a <code>52M</code> xml file into a Pandas DataFrame.</p>
  
  <p>This is an extract of the xml file ...</p>
  
  <pre><code>&lt;findToFileResponse xmlns="xmlapi_1.0"&gt;
      &lt;equipment.MediaIndependentStats&gt;
          &lt;rxOctets&gt;0&lt;/rxOctets&gt;
          &lt;txOctets&gt;0&lt;/txOctets&gt;
          &lt;inSpeed&gt;10000000&lt;/inSpeed&gt;
          &lt;outSpeed&gt;10000000&lt;/outSpeed&gt;
          &lt;time&gt;1587080746395&lt;/time&gt;
          &lt;seconds&gt;931265&lt;/seconds&gt;
          &lt;port&gt;Port 3/1/6&lt;/port&gt;
          &lt;ip&gt;192.168.157.204&lt;/ip&gt;
          &lt;name&gt;RouterA&lt;/name&gt;
      &lt;/equipment.MediaIndependentStats&gt;
      &lt;equipment.MediaIndependentStats&gt;
          &lt;rxOctets&gt;0&lt;/rxOctets&gt;
          &lt;txOctets&gt;0&lt;/txOctets&gt;
          &lt;inSpeed&gt;100000&lt;/inSpeed&gt;
          &lt;outSpeed&gt;100000&lt;/outSpeed&gt;
          &lt;time&gt;1587080739924&lt;/time&gt;
          &lt;seconds&gt;928831&lt;/seconds&gt;
          &lt;port&gt;Port 1/1/1&lt;/port&gt;
          &lt;ip&gt;192.168.154.63&lt;/ip&gt;
          &lt;name&gt;RouterB&lt;/name&gt;
      &lt;/equipment.MediaIndependentStats&gt;
  &lt;/findToFileResponse&gt;
  </code></pre>
  
  <p>any ideas on how to improve speed? </p>
  
  <p>For the extract of the above xml, the function <code>fnc_parse_xml(file, columns)</code> returns this DF....</p>
  
  <pre><code>In [83]: df                                                                                                                           
  Out[83]: 
    rxOctets txOctets   inSpeed  outSpeed           time seconds        port               ip     name
  0        0        0  10000000  10000000  1587080746395  931265  Port 3/1/6  192.168.157.204  RouterA
  1        0        0    100000    100000  1587080739924  928831  Port 1/1/1   192.168.154.63  RouterB
  </code></pre>