title: Python Threading stdin/stdout
body: <p>I have a file which contains a lot of data. Each row is a record. And I am trying to do some ETL work against the whole file. Right now I am using standard input to read the data line by line. The cool thing about this is your script could be very flexible to integrate with other script and shell commands.  I write the result to standard output. For example. </p>
  
  <pre><code>$ cat input_file
  line1 
  line2
  line3
  line4
  ...
  </code></pre>
  
  <p>My current python code looks like this - parse.py</p>
  
  <pre><code>import sys
  for line in sys.stdin:
      result = ETL(line)    # ETL is some self defined function which takes a while to execute.
      print result
  </code></pre>
  
  <p>The code below is how it is working right now:</p>
  
  <pre><code>cat input_file | python parse.py &gt; output_file
  </code></pre>
  
  <p>I have looked at the Threading module of Python and I am wondering if the performance would be dramatically improved if I use that module. </p>
  
  <p>Question1: <strong><em>How should I plan the quotas for each thread, why?</em></strong> </p>
  
  <pre><code>...
  counter = 0
  buffer = []
  for line in sys.stdin:
      buffer.append(line)
      if counter % 5 == 0:   # maybe assign 5 rows to each thread? if not, is there a rule of thumb to determine
          counter = 0
          thread = parser(buffer)
          buffer = []
          thread.start() 
  </code></pre>
  
  <p>Question2: <strong><em>Multiple Threads might print the result back to stdout at the same time, how to organize them and avoid the situation below?</em></strong></p>
  
  <pre><code>import threading
  import time
  
  class parser(threading.Thread):
      def __init__ (self, data_input):
          threading.Thread.__init__(self)
          self.data_input = data_input
  
      def run(self):
          for elem in self.data_input:
              time.sleep(3)
              print elem + 'Finished'
  
  work = ['a', 'b', 'c', 'd', 'e', 'f']
  
  thread1 = parser(['a', 'b'])  
  thread2 = parser(['c', 'd'])
  thread3 = parser(['e', 'f'])
  
  thread1.start()
  thread2.start()
  thread3.start()   
  </code></pre>
  
  <p>The output is really ugly, where one row contains the outputs from two threads. </p>
  
  <pre><code>aFinished
  cFinishedeFinished
  
  bFinished
  fFinished
  dFinished
  </code></pre>