title: Receiving socket data in one thread, writing the data in another -- python
body: <p>I'm currently writing a Python program to receive data from either a TCP/UDP socket, and then write the data to a file. Right now, my program is I/O bound by writing each datagram to the file as it comes in (I'm doing this for very large files, so the slowdown is considerable). With that in mind, I've decided that I'd like to trying receiving the data from the socket in one thread, and then write that data in a different thread. So far, I've come up with the following rough draft. At the moment, it only writes a single data chunk (512 bytes) to a file. </p>
  
  <pre><code>f = open("t1.txt","wb")
  def write_to_file(data):
      f.write(data)
  
  def recv_data():
      dataChunk, addr = sock.recvfrom(buf) #THIS IS THE DATA THAT GETS WRITTEN
      try:
          w = threading.Thread(target = write_to_file, args = (dataChunk,))
          threads.append(w)
          w.start()
          while(dataChunk):
              sock.settimeout(4)
              dataChunk,addr = sock.recvfrom(buf)
      except socket.timeout:
          print "Timeout"
          sock.close()
          f.close()
  
  threads = []
  r = threading.Thread(target=recv_data)
  threads.append(r)
  r.start()
  </code></pre>
  
  <p>I imagine I'm doing something wrong, I'm just not sure what the best way to use threading is. Right now, my issue is that I have to supply an argument when I create my thread, but the value of that argument doesn't properly change to reflect the new data chunks that come in. However, if I put the line <code>w=threading.Thread(target=write_to_file, arg=(dataChunk,))</code> inside the <code>while(dataChunk)</code> loop, wouldn't I be creating a new thread each iteration?</p>
  
  <p>Also, for what it's worth, this is just my small proof-of-concept for using separate receive and write threads. This is not the larger program that should ultimately make use of this concept. </p>