title: PySerial loses data
body: <p>My problem is, that PySerial seems to lose some data packages and I don't know why. </p>
  
  <p>I have two python scripts, the first one reads data from a text file and writes it to a microcontroller, where the data is manipulated. Then, the microcontroller sends the modified data over a different serial port back to the PC. (for clarification: I need two serial ports, because in the final application, the scripts will be running on different PCs. However, for testing purpose, it is easier to run both scripts on one PC)</p>
  
  <p>So basically, my hardware setup looks like: </p>
  
  <pre><code>PC ----(serial port 1)----&gt; microcontroller
     &lt;---(serial port 2)----- 
  </code></pre>
  
  <p>After writing the data to the microcontroller I am expecting to get a certain amount of data bytes back. If I use a terminal program (like HyperTerminal) to monitor the data which is received, everything looks fine. However, if I try to read the data with a Python script, I only get a fraction of the expected data bytes. </p>
  
  <p>For example: </p>
  
  <pre><code>+--------------------+--------------------+
  | with HyperTerminal | with Python script |
  +--------------------+--------------------+
  | 1:W:00522          | 1:W:00522          |
  | 1:W:00532          | 1:W:00532          |
  | 1:W:00518          | 1:W:00522          |
  | 1:W:00522          | 1:W:00526          |
  | 1:W:00522          | 1:W:00514          |
  | 1:W:00526          | 1:W:00520          |
  | 1:W:00514          | 1:W:00514          |
  | 1:W:00520          | 1:W:00522          |
  | 1:W:00520          | 1:W:00526          |
  | 1:W:00514          | 1:W:00520          |
  | 1:W:00516          | 1:W:00526          |
  | 1:W:00522          | 1:W:00520          |
  | 1:W:00526          | 1:W:00524          |
  | 1:W:00520          | 1:W:00526          |
  | 1:W:00520          | 1:W:00532          |
  | 1:W:00526          | 1:W:00506          |
  | 1:W:00522          | 1:W:00520          |
  | 1:W:00520          | 1:W:00526          |
  | 1:W:00524          | 1:W:00524          |
  | 1:W:00522          | 1:W:00526          |
  | 1:W:00526          | 1:W:00514          |
  | 1:W:00514          | 1:W:00522          |
  | 1:W:00532          | 1:W:00520          |
  | 1:W:00506          | 1:W:00510          |
  | 1:W:00522          | 1:W:00506          |
  | 1:W:00520          |                    |
  | 1:W:00526          |                    |
  | 1:W:00530          |                    |
  | 1:W:00524          |                    |
  | 1:W:00526          |                    |
  | 1:W:00514          |                    |
  | 1:W:00514          |                    |
  | 1:W:00522          |                    |
  | 1:W:00524          |                    |
  | 1:W:00520          |                    |
  | 1:W:00510          |                    |
  | 1:W:00506          |                    |
  +--------------------+--------------------+
  </code></pre>
  
  <p>As you can see, if I try to read from the serial port with my Python script, I am missing some data. Due to the fact, that I get the expected data if I use a terminal program, I assume, that my Python script has an error. </p>
  
  <p>My Python script for sending data to the microcontroller looks like: </p>
  
  <pre><code>import serial
  import re
  import time
  
  class digiRealTest():
      def __init__(self):
          #configure serial port 
          self.ser = serial.Serial(7, 9600, parity=serial.PARITY_NONE) 
  
  
      def main(self):
          filepath = 'C:\\Users\\Bernhard\\Desktop\\TomatoView\\Qt\\test_output.txt'
          with open(filepath, 'r') as content_file:
              content = content_file.read().decode("hex")
              for match in re.finditer('[\02](.*?)[\03]', content, re.S):
                  res = match.group(1)
                  complete =  '\x02' + res + '\x03'
                  # time.sleep(0.3) &lt;-- if i uncomment this line, it work's!!!
                  self.ser.write(complete)
  
  if __name__ == "__main__": #wenn Modul direkt ausgefuehrt wird
      d = digiRealTest()
      d.main()
  </code></pre>
  
  <p>My Python script for receiving the data sent from the microcontroller: </p>
  
  <pre><code>import Queue
  import threading
  import serial
  
  class mySerial(threading.Thread):
      def __init__(self, queue):
          super(mySerial, self).__init__()
  
          self.queue = queue #the received data is put in a queue
  
          self.buffer = ''
  
          #configure serial connection
          self.ser = serial.Serial(timeout = 0, port = 3, baudrate=9600)
  
      def run(self):              
          while True:
              self.buffer += self.ser.read(self.ser.inWaiting()) #read all char in buffer
              if '\n' in self.buffer: #split data line by line and store it in var
                  var, self.buffer = self.buffer.split('\n')[-2:]
                  self.queue.put(var) #put received line in the queue
          time.sleep(0.01)   #do not monopolize CPU
  
  
  
  class Base():
      def __init__(self):
          self.queue = Queue.Queue(0) #create a new queue
          self.ser = mySerial(self.queue) 
          self.ser.start() #run thread
  
  
      def main(self   ):
          while(True):
              try:
                  var = self.queue.get(False) #try to fetch a value from queue
              except Queue.Empty: 
                  pass #if it is empty, do nothing
              else:
                  print(var) 
  
  
  if __name__ == '__main__':
      b = Base() 
      b.main()
  </code></pre>
  
  <p>I don't know why, but if I uncomment the line <code>#time.sleep(0.3)</code> in the sending script, everything works fine and I get the expected data back. So to me, it seems, that somehow my script for reading the data from the serial port is too slow....but why? </p>