title: Remove duplicates after altering items
body: <p>I have a script to clean urls to get base domains from example.com/example1 and example.com/example2 down to example.com My issue is when it goes to through the file of urls it will have duplicate base domains. I want to remove the duplicates while printing the urls to a file. below is the code I currently have.</p>
  
  <pre><code>enter from Tkinter import *
  import tkFileDialog
  import re
  
  def main():
  
          fileOpen = Tk()
          fileOpen.withdraw() #hiding tkinter window
  
          file_path = tkFileDialog.askopenfilename(
          title="Open file", filetypes=[("txt file",".txt")])
  
          if file_path != "":
              print "you chose file with path:", file_path
  
          else:
              print "you didn't open anything!"
  
          fin = open(file_path)
          fout = open("URL Cleaned.txt", "wt")
          for line in fin.readlines():
                  editor = (line.replace('[.]', '.')
                  .replace('[dot]', '.')
                  .replace('hxxp://www.', '')
                  .replace('hxxps://www.', '')
                  .replace('hxxps://', '')
                  .replace('hxxp://', '')
                  .replace('www.', '')
                  .replace('http://www.', '')
                  .replace('https://www.', '')
                  .replace('https://', '')
                  .replace('http://', ''))
                  editor = re.sub(r'/.*', '', editor)
  
  if __name__ == '__main__':
          main()
  </code></pre>
  
  <p>Any help is appreciated. I have scoured the posts and tried all of the suggestions for my issue and have not found one that works.</p>