title: How to store (and access) a very large list in python
body: <p>I have a file with over 4 million lines that I want to read in, line by line, and perform actions on the first "column" of data. There may be duplicated data, and I want to make sure that I am only performing these actions once, and be able to restart the program and start where it left off.  </p>
  
  <p>My solution was to parse the line, store the value into a list, and then "If x not in list" perform actions, and then store that list as a pickle if/when the program dies/stops/is killed.</p>
  
  <p>However the script segfaults after running for several hours, and I assume it is because I have burned through all my ram due to storing a lot of MD5 sums in the checkedFile list. </p>
  
  <p>Because the file is so large (4million+ entries) I need something that is fast for lookups, hoping there is a simple solution.  Thanks!</p>
  
  <p>Data looks like this:  6e5f6c90e6bf31c31283afb8dca93da8|mouse.gif|10102017</p>
  
  <p>My current code looks like this:</p>
  
  <pre><code>checkedFile = []
  
  def readline(inFile):
  print("Opening file: %s\n" % inFile)
  try:
      with open(inFile, "r") as inputFile:
          for line in inputFile: # read in file line by line
              fileHash, garbage = line.split("|",1) 
              if fileHash not in checkedFile:
                  checkedFile.append(fileHash)
                  checkit(fileHash) # 
  </code></pre>