title: How to completely erase the duplicated lines by linux tools?
body: <p>This question is not equal to <a href="https://stackoverflow.com/questions/23740545/how-to-print-only-the-unique-lines-in-bash">How to print only the unique lines in BASH?</a> because that ones suggests to remove all copies of the duplicated lines, while this one is about eliminating their duplicates only, i..e, change <code>1, 2, 3, 3</code> into <code>1, 2, 3</code> instead of just <code>1, 2</code>.</p>
  <p>This question is really hard to write because I cannot see anything to give meaning to it. But the example is clearly straight. If I have a file like this:</p>
  <pre><code>1
  2
  2
  3
  4
  </code></pre>
  <p>After to parse the file erasing the duplicated lines, becoming it like this:</p>
  <pre><code>1
  3
  4
  </code></pre>
  <p>I know python or some of it, this is a python script I wrote to perform it. Create a file called <code>clean_duplicates.py</code> and run it as:</p>
  <pre><code>import sys
  
  #
  # To run it use:
  # python clean_duplicates.py &lt; input.txt &gt; clean.txt
  #
  def main():
  
      lines = sys.stdin.readlines()
  
      # print( lines )
      clean_duplicates( lines )
  
  #
  # It does only removes adjacent duplicated lines, so your need to sort them
  # with sensitive case before run it.
  # 
  def clean_duplicates( lines ):
  
      lastLine    = lines[ 0 ]
      nextLine    = None
      currentLine = None
      linesCount  = len( lines )
  
      # If it is a one lined file, to print it and stop the algorithm
      if linesCount == 1:
  
          sys.stdout.write( lines[ linesCount - 1 ] )
          sys.exit()
  
      # To print the first line
      if linesCount &gt; 1 and lines[ 0 ] != lines[ 1 ]:
  
          sys.stdout.write( lines[ 0 ] )
  
      # To print the middle lines, range( 0, 2 ) create the list [0, 1]
      for index in range( 1, linesCount - 1 ):
  
          currentLine = lines[ index ]
          nextLine    = lines[ index + 1 ]
  
          if currentLine == lastLine:
  
              continue
  
          lastLine = lines[ index ]
  
          if currentLine == nextLine:
  
              continue
  
          sys.stdout.write( currentLine )
  
      # To print the last line
      if linesCount &gt; 2 and lines[ linesCount - 2 ] != lines[ linesCount - 1 ]:
  
          sys.stdout.write( lines[ linesCount - 1 ] )
  
  if __name__ == &quot;__main__&quot;:
  
      main()
  </code></pre>
  <p>Although, while searching for duplicates lines remove seems to be easier to use tools as grep, sort, sed, uniq:</p>
  <ol>
  <li><a href="https://unix.stackexchange.com/questions/30173/how-to-remove-duplicate-lines-inside-a-text-file">How to remove duplicate lines inside a text file?</a></li>
  <li><a href="https://stackoverflow.com/questions/21582610/removing-line-from-list-using-sort-grep-linux">removing line from list using sort, grep LINUX</a></li>
  <li><a href="https://stackoverflow.com/questions/6712437/find-duplicate-lines-in-a-file-and-count-how-many-time-each-line-was-duplicated">Find duplicate lines in a file and count how many time each line was duplicated?</a></li>
  <li><a href="https://stackoverflow.com/questions/9377040/remove-duplicate-entries-using-a-bash-script">Remove duplicate entries in a Bash script</a></li>
  <li><a href="https://stackoverflow.com/questions/1444406/how-can-i-delete-duplicate-lines-in-a-file-in-unix">How to delete duplicate lines in a file without sorting it in Unix?</a></li>
  <li><a href="https://stackoverflow.com/questions/19039036/how-to-delete-duplicate-lines-in-a-file-awk-sed-uniq-not-working-on-my-file">How to delete duplicate lines in a file...AWK, SED, UNIQ not working on my file</a></li>
  </ol>