title: Python Parsing Multiple XML Nodes with Dynamic Data
body: <p>I have an log file from an application in XML-like format that I'm trying to parse. As you can see from the file, one &quot;group&quot; starts with a [trace] line, and contains 4 nodes - RequestMeta, Request, ReplyMeta, and Reply.</p>
  <p>Once the file is parsed, I want to create an object for each &quot;group&quot; and use the objects for further processing. There could be from 1:n groups depending on the complexity of the log file.</p>
  <p>I have been able to parse the XML, but I have some questions on how best to proceed based on it's structure.</p>
  <p>The first problem is how to structure/re-structure the file for parsing. Since I'm adding a single root node to more than one &quot;group&quot;, there will be no easy way for me to know which children of the root node belong together in that group. In the original file, the group is denoted as everything between the [trace] line and the next [trace] line.</p>
  <p>I think I could potentially solve this by taking each string &quot;group&quot; and create a tree for each group instead of a tree for the entire file.</p>
  <p>The second problem is how to store the data once it's parsed. Each and every request/reply will contain different data elements under the <code>srvdata</code> node. I'm not sure how to dynamically store a variable number of values that have a variable number of names.</p>
  <p>After parsing all of the data, I want to output it in a simple webpage that looks something like <a href="https://imgur.com/a/2l6ZSJK" rel="nofollow noreferrer">https://imgur.com/a/2l6ZSJK</a></p>
  <p>py script</p>
  <pre class="lang-py prettyprint-override"><code>import xml.etree.ElementTree as ET
  
  with open('C:/code/mra/requestreply.txt') as f:
      txt = f.read()
  
  pos = 0
  # replace all [trace] lines
  while pos &gt;= 0:
      pos = txt.find('[trace-')
      pos2 = txt.find('\n', pos + 1) + 1
      if pos &gt;= 0:
          txt = txt.replace(txt[pos:pos2], '')
  
  # replace all xml instances because they are out of order
  txt = txt.replace('&lt;?xml version=&quot;1.0&quot; encoding=&quot;utf-8&quot;?&gt;\n', '')
  # add a master root node
  xml = '&lt;root&gt;\n' + txt + '&lt;/root&gt;'
  
  tree = ET.fromstring(xml)
  </code></pre>
  <p>xml file - this is considered a single group (there could be hundreds)</p>
  <pre><code>[trace-592] TransactionID=6010 TransactionName=CPM.ExecuteDiscernScript User=MEPPS
  &lt;RequestMeta&gt;
    &lt;?xml version=&quot;1.0&quot; encoding=&quot;utf-8&quot;?&gt;
    &lt;srvxml&gt;
      &lt;xs:schema xmlns:xs=&quot;http://www.w3.org/2001/XMLSchema&quot;&gt;
        ....
      &lt;/xs:schema&gt;
      &lt;srvdata lang=&quot;C&quot;&gt;
        ....
       &lt;/srvdata&gt;
    &lt;/srvxml&gt;
  &lt;/RequestMeta&gt;
  &lt;Request&gt;
    &lt;?xml version=&quot;1.0&quot; encoding=&quot;utf-8&quot;?&gt;
    &lt;srvxml&gt;
      &lt;xs:schema xmlns:xs=&quot;http://www.w3.org/2001/XMLSchema&quot;&gt;
        ....
      &lt;/xs:schema&gt;
      &lt;srvdata lang=&quot;C&quot;&gt;
        ....
      &lt;/srvdata&gt;
   &lt;/srvxml&gt;
  &lt;/Request&gt;
  &lt;ReplyMeta&gt;
    &lt;?xml version=&quot;1.0&quot; encoding=&quot;utf-8&quot;?&gt;
    &lt;srvxml&gt;
      &lt;xs:schema xmlns:xs=&quot;http://www.w3.org/2001/XMLSchema&quot;&gt;
        ....
      &lt;/xs:schema&gt;
      &lt;srvdata lang=&quot;C&quot;&gt;
        ....
       &lt;/srvdata&gt;
    &lt;/srvxml&gt;
  &lt;/ReplyMeta&gt;
  &lt;Reply&gt;
    &lt;?xml version=&quot;1.0&quot; encoding=&quot;utf-8&quot;?&gt;
    &lt;srvxml&gt;
      &lt;xs:schema xmlns:xs=&quot;http://www.w3.org/2001/XMLSchema&quot;&gt;
        ....
      &lt;/xs:schema&gt;
      &lt;srvdata lang=&quot;C&quot;&gt;
        ....
      &lt;/srvdata&gt;
   &lt;/srvxml&gt;
  &lt;/Reply&gt;
  </code></pre>