title: Extract XML content from a log file using Sed and dump each result to a different file
body: <p>I have the following 10 GB log file that I need to analyze directly on a Unix server. </p>
  
  <pre><code>2017-12-12 13:04:28,716 [ABC] [DEF] DEBUG some message1
  2017-12-12 13:04:28,716 [ABC] [DEF] DEBUG some message2
  2017-12-12 13:04:28,716 [ABC] [DEF] DEBUG some message3
  2017-12-12 13:04:28,716 [ABC] [DEF] DEBUG some message4
  2017-12-12 13:04:28,716 [ABC] [DEF] DEBUG some message5
  2017-12-12 13:04:28,732 [ABC] [DEF] DEBUG some message6
  2017-12-12 13:04:28,732 [ABC] [DEF] DEBUG &lt;xml&gt;
  &lt;id&gt;1&lt;/id&gt; 
  &lt;!—- id is not unique since the XML data provides all the
  information of an object X defined by its id at a specific point in time --&gt;
  some XML content on more than 500 lines
  &lt;/xml&gt;
  2017-12-12 13:04:30,330 [ABC] [DEF] DEBUG some message8
  2017-12-12 13:04:30,333 [ABC] [DEF] DEBUG some message9
  2017-12-12 13:04:30,334 [ABC] [DEF] INFO some message10
  2017-12-12 13:04:30,334 [ABC] [DEF] INFO some message11
  2017-12-12 13:04:31,431 [ABC] [DEF] INFO some message12
  2017-12-12 13:04:28,732 [ABC] [DEF] DEBUG &lt;xml&gt;
  &lt;id&gt;2&lt;/id&gt;
  some XML content on more than 500 lines 
  &lt;/xml&gt;
  2017-12-12 13:04:31,432 [ABC] [DEF] DEBUG some message13
  2017-12-12 13:04:31,476 [ABC] [DEF] INFO some message14
  2017-12-12 13:04:31,476 [ABC] [DEF] DEBUG some message14
  2017-12-12 13:04:31,490 [ABC] [DEF] DEBUG some message15
  2017-12-12 13:04:28,732 [ABC] [DEF] DEBUG &lt;xml&gt;
  &lt;id&gt;1&lt;/id&gt;
  some XML content on more than 500 lines 
  &lt;/xml&gt;
  2017-12-12 13:04:31,491 [ABC] [DEF] DEBUG some message16
  2017-12-12 13:04:31,491 [ABC] [DEF] DEBUG some message17
  2017-12-12 13:04:31,496 [ABC] [DEF] DEBUG some message18
  2017-12-12 13:04:31,996 [ABC] [DEF] INFO some message19
  </code></pre>
  
  <p>In order to do so, I would like to <strong>extract each XML message and dump it in a separate file</strong>.</p>
  
  <p><strong>For example:</strong> the first XML message would be stored in <code>file1.xml</code>, the second one in <code>file2.xml</code>, and so on.</p>
  
  <p>If all the patterns had to be extracted to one single file, it would be quite direct with something like:</p>
  
  <pre><code>sed -n 's~&lt;xml&gt;(\s*\.*\s*)\s*&lt;/xml&gt;~p' file.in &gt; file.out #just a prototype
  </code></pre>
  
  <p>I thought about going to a solution in which I could use a back reference with the <code>&lt;id&gt;</code> tag of the XML and use it to name the file in which I would dump it, but it is not working since same values of <code>&lt;id&gt;</code> tag do appear at different places in the log file, which would overwrite the previous extractions. </p>
  
  <pre><code>sed -r 's~(&lt;xml&gt;…&lt;id&gt;(.*)&lt;/id&gt;…&lt;/xml&gt;)~echo "\1" &gt;&gt; \2.out~e' file.in #just a prototype
  </code></pre>
  
  <p>With <code>awk</code>, if the XML content was on one single line, it would be also quite straightforward. However, it is not the case and I don’t know which line separator I should define for <code>RS</code> to treat the XML content as if it was on a single line and dump it in separate files. </p>
  
  <p>With <code>awk</code>, what I thought feasable was: </p>
  
  <ul>
  <li>first identifying the <code>&lt;xml&gt;</code> starting tag in the log and change a test variable to <code>yes</code> </li>
  <li>store each line of the XML in a buffer variable before dumping it to a <code>file$i.out</code> as soon as I get the <code>&lt;/xml&gt;</code> (and of course resetting the test variable to <code>no</code>). </li>
  </ul>
  
  <p>If you have a better solution with <code>awk</code> or a solution with <code>sed</code> in which I could access a variable containing the number of the pattern being currently treated and reuse it to generate the output files, it would be great. (something like that: <code>current_pattern_position</code> used to generate <code>file_$current_pattern_position.out</code>)</p>
  
  <blockquote>
    <p><strong>I got already pretty interesting solutions using <code>awk</code> and <code>perl</code>. I would like to have a <code>sed</code> working solution for this case</strong></p>
  </blockquote>