title: How to optimize code of comparison between a list and strings through stdin - performance is important
body: <p>Right now i have a script that receives strings from stdin and i have also a list which is about 70mb on disk (in many partial files) which i load into memory into one list.</p>
  
  <p>I then search each string as it comes in from stdin and search if it exists in the list. I understand this is slow because of the huge list and the number of strings that can also be a great amount that come in.</p>
  
  <p>It goes like this:</p>
  
  <pre><code>def buildindex():
      # j = 0
      # while j &lt; len(parts_list):
          # f = urllib2.urlopen("https://s3.amazonaws.com/source123/output/" + parts_list[j])
      j = 0
      while j &lt;= 200:
          if j &lt; 10:
              f = urllib2.urlopen("https://s3.amazonaws.com/source123/output/part-0000" + str(j))
          if j &lt; 100 and j &gt;= 10:
              f = urllib2.urlopen("https://s3.amazonaws.com/source123/output/part-000" + str(j))
          if j &gt;= 100:
              f = urllib2.urlopen("https://s3.amazonaws.com/source123/output/part-00" + str(j))
  
          for line in f.readlines():
              line = line.rstrip()
              yield line
              print line
          j += 1
          f.close()
  
  linelist = list(buildindex())
  
  for suspicious_line in sys.stdin:
      if "," in suspicious_line:
          suspicious_key, suspicious_source, suspicious_start, suspicious_end = suspicious_line.strip().split(",")
  
          x=re.compile(suspicious_key)
          sub_list = filter(x.match, linelist)
          # do something
  </code></pre>
  
  <p>I tried to run this locally and its been over 20 minutes and its still going. Also i will use these scripts on Amazon EMR (hadoop) and it also fails for some reason. If i try a subset of the list, it works.</p>
  
  <p>What performance wise changes can i make to keep things neat and relatively fast?</p>