title: Combining Two Text Files Removing Duplicates
body: <p>I have <strong>2 text files</strong> that are as follows (large numbers like <code>1466786391</code> being unique timestamps):</p>
  
  <pre><code>--- 10.0.0.6 ping statistics ---
  50 packets transmitted, 49 packets received, 2% packet loss
  round-trip min/avg/max = 20.917/70.216/147.258 ms
  1466786342
  PING 10.0.0.6 (10.0.0.6): 56 data bytes
  
  ....
  
  --- 10.0.0.6 ping statistics ---
  50 packets transmitted, 50 packets received, 0% packet loss
  round-trip min/avg/max = 29.535/65.768/126.983 ms
  1466786391
  </code></pre>
  
  <p>and this:</p>
  
  <pre><code>--- 10.0.0.6 ping statistics ---
  50 packets transmitted, 49 packets received, 2% packet loss
  round-trip min/avg/max = 20.917/70.216/147.258 ms
  1466786342
  PING 10.0.0.6 (10.0.0.6): 56 data bytes
  
  --- 10.0.0.6 ping statistics ---
  50 packets transmitted, 50 packets received, 0% packet loss
  round-trip min/avg/max = 29.535/65.768/126.983 ms
  1466786391
  PING 10.0.0.6 (10.0.0.6): 56 data byte
  
  --- 10.0.0.6 ping statistics ---
  50 packets transmitted, 44 packets received, 12% packet loss
  round-trip min/avg/max = 30.238/62.772/102.959 ms
  1466786442
  PING 10.0.0.6 (10.0.0.6): 56 data bytes
  ....
  </code></pre>
  
  <p>So the first file ends with the <code>timestamp</code> <strong>1466786391</strong> and the second file has the same data block somewhere in the middle and more data afterwards, the data before the specific timestamp is exactly same as the first file.</p>
  
  <p>So the output I want is this:</p>
  
  <pre><code>--- 10.0.0.6 ping statistics ---
      50 packets transmitted, 49 packets received, 2% packet loss
      round-trip min/avg/max = 20.917/70.216/147.258 ms
      1466786342
      PING 10.0.0.6 (10.0.0.6): 56 data bytes
  
      ....
  
      --- 10.0.0.6 ping statistics ---
      50 packets transmitted, 50 packets received, 0% packet loss
      round-trip min/avg/max = 29.535/65.768/126.983 ms
      1466786391
  
   --- 10.0.0.6 ping statistics ---
      50 packets transmitted, 44 packets received, 12% packet loss
      round-trip min/avg/max = 30.238/62.772/102.959 ms
      1466786442
      PING 10.0.0.6 (10.0.0.6): 56 data bytes
  ....
  </code></pre>
  
  <p>That is, concatenate the two files, and create a third one removing the duplicates of the second file (the text blocks that's already present in the first file. Here's my code:</p>
  
  <pre><code>public static void UnionFiles()
  { 
  
      string folderPath = Path.Combine(Path.GetDirectoryName(Assembly.GetEntryAssembly().Location), "http");
      string outputFilePath = Path.Combine(Path.GetDirectoryName(Assembly.GetEntryAssembly().Location), "http\\union.dat");
      var union = Enumerable.Empty&lt;string&gt;();
  
      foreach (string filePath in Directory
                  .EnumerateFiles(folderPath, "*.txt")
                  .OrderBy(x =&gt; Path.GetFileNameWithoutExtension(x)))
      {
          union = union.Union(File.ReadAllLines(filePath));
      }
      File.WriteAllLines(outputFilePath, union);
  }
  </code></pre>
  
  <p>This is the wrong output I am getting (the file structure is destroyed):</p>
  
  <pre><code>--- 10.0.0.6 ping statistics ---
  50 packets transmitted, 49 packets received, 2% packet loss
  round-trip min/avg/max = 20.917/70.216/147.258 ms
  1466786342
  PING 10.0.0.6 (10.0.0.6): 56 data bytes
  
  --- 10.0.0.6 ping statistics ---
  50 packets transmitted, 50 packets received, 0% packet loss
  round-trip min/avg/max = 29.535/65.768/126.983 ms
  1466786391
  round-trip min/avg/max = 30.238/62.772/102.959 ms
  1466786442
  round-trip min/avg/max = 5.475/40.986/96.964 ms
  1466786492
  round-trip min/avg/max = 5.276/61.309/112.530 ms
  </code></pre>
  
  <p>EDIT: This code was written to handle multiple files, however I am happy even if just 2 can be done correctly.</p>
  
  <p>However, this doesn't remove the <code>textblocks</code> as it should, it removes several useful lines and makes the the output utter useless. I am stuck.</p>
  
  <p>How to achieve this?
  Thanks.</p>