title: Comparing two files in python, with each file having duplicate data
body: <p>I am trying to compare two files in python, its actually trying to find new warnings after comparing it with the old warning file.</p>
  
  <p>The content of the old file is something like this:</p>
  
  <pre><code>warning1~file1
  warning1~file1
  warning2~file2
  warning2~file2
  warning2~file2
  </code></pre>
  
  <p>and content of the new file is something like this</p>
  
  <pre><code>warning1~file1
  warning1~file1
  warning1~file1
  warning3~file3
  warning2~file2
  warning2~file2
  warning2~file2
  </code></pre>
  
  <p>As u can see, in the new file i have 2 new line of text warning1~file1 warning3~file3, i searched for comparing two files over the internet but everywhere they assume that every line of text is different.</p>
  
  <pre><code>small_file = open('file1','r')
  long_file = open('file2','r')
  output_file = open('newfile','w')
  
  try:
      small_lines = small_file.readlines()
      small_lines_cleaned = [line.rstrip().lower() for line in small_lines]
      long_lines = long_file.readlines()
      long_lines_cleaned = [line.rstrip().lower() for line in long_lines]
  
      #for line in long_lines_cleaned:
      for line in long_lines_cleaned:
          if line not in small_lines_cleaned:
              output_file.writelines(line)
  </code></pre>
  
  <p>I tried this code which i found here, but after running it, realized that it also just checks whether a line from file1 is available in file2 or not. If not, then write to a new file.
  This method only fetches warning3, not the new warning1.</p>
  
  <p>I need something where every line is compared only once... and the lines which are left are written to the new file.</p>
  
  <p>I hope i have explained the problem properly.</p>