title: How to efficiently read, encode and store data from the serial port in Python?
body: <p>The purpose of the program: Read serial data from the serial port, convert them to int and store them into a list. This data will be later processed and printed in real-time using matplotlib.</p>
  
  <p>The problems: </p>
  
  <ol>
  <li>The list is taking too long to be created (several seconds, maybe
  minutes).<br>
  ps: the size o the list is n = 1024.</li>
  <li><p>After all that minutes to create the list I get this:<br>
  "(...)<br>
  File "C:\Users\ianmc\AppData\Local\Programs\Python\Python35-32\lib\site-packages\matplotlib\lines.py", line 645, in recache<br>
  raise RuntimeError('xdata and ydata must be the same length')<br>
  RuntimeError: xdata and ydata must be the same length  "</p>
  
  <p>Probably the plot arguments got different lengths but why? </p></li>
  </ol>
  
  <p>The code:</p>
  
  <pre><code>import serial
  import math
  import numpy as np
  import matplotlib.pyplot as plt
  import matplotlib.animation as animation
  
  
  dt = 0.01
  Fs = 44000.0                     # sample rate
  timestep = 1.0/Fs                # sample spacing (1/sample rate)
  n = 1024                         # size of the array data (BUFFER SIZE)
  t = np.arange(0, (n/100), dt)    # t range
  w = 10000                        # frequency of the input
  
  #initialize data list
  data = [0] * n
  
  #initialize magnitude list
  magnitude = [0] * (n//2)
  
  # open and prepare serial port
  ser = serial.Serial('COM4', 9600, timeout=8,
                      parity=serial.PARITY_EVEN, rtscts=1)
  
  
  # calculates the fft dB
  def createMagnitudeDB(magnitude):
      magnitudeDB = 20 * (math.log(magnitude,10))
      return magnitudeDB
  
  def update(data):
      # update the curves with the incoming data
      timeCurve.set_ydata(data[0])
      freqCurve.set_ydata(data[1])
      return timeCurve, freqCurve,
  
  def generateData():
      # simulate new data coming in
      while True:
  
          # THE EFFICIENCY PROBLEM IS IN THIS FOR
          # Read data from arduino
          for i in range(n):     #while you are taking data
              PreData = ser.readline()
              data.append(int(PreData))
  
  
          # fft computing and normalization
          magnitude = np.fft.fft(data)/n
          magnitude = np.abs(magnitude[range(n//2)])
  
          # calculates the fft dB
          magnitude_dB = [createMagnitudeDB(x) for x in magnitude]
  
          yield (data, magnitude)
  
  
  fig = plt.figure()
  
  # create time graph axis
  timeGraph = plt.subplot(2, 1, 1)
  timeGraph.set_ylim(-200, 200)
  timeGraph.set_xlim([0, n])
  timeGraph.set_xlabel('Time')
  timeGraph.set_ylabel('Amplitude')
  
  # create frequency graph axis
  freqGraph = plt.subplot(2, 1, 2)
  freqGraph.set_ylim([0, 70])
  freqGraph.set_xlim([0, (Fs/2)])
  freqGraph.set_xlabel('Freq (Hz)')
  freqGraph.set_ylabel('Magnitude')
  
  
  #get frequency range
  k = np.arange(n)
  T = n/Fs
  freq = k/T               # two sides frequency range
  freq = freq[range(n//2)] # one side frequency range
  
  # plot the curves
  timeCurve, = timeGraph.plot(np.linspace(0, 1, n),'b')
  freqCurve, = freqGraph.plot(freq, magnitude,'g')
  
  # animate the curves
  ani = animation.FuncAnimation(fig, update, generateData,
                                interval = 10, blit=True)
  
  # open window
  plt.show()
  
  # close serial connection
  ser.close()
  </code></pre>
  
  <p>Another piece of information: If instead of using the serial port I generate my own data the program works fine, here is it:  </p>
  
  <pre><code>def generateData():
      # simulate new data coming in
      while True:
  
          nse = np.random.randn(len(t))
          r = np.exp(-t/0.05)
          cnse = np.convolve(nse, r)*dt
          cnse = cnse[:len(t)]
          data =  100*np.sin(2*np.pi*t) + 500*cnse
  
  
          # fft computing and normalization
          magnitude = np.fft.fft(data)/n
          magnitude = np.abs(magnitude[range(n//2)])
  
          # calculates the fft dB
          magnitude_dB = [createMagnitudeDB(x) for x in magnitude]
  
          yield (data, magnitude)
  </code></pre>
  
  <p>The only difference is the generateData function and not using the open/close serial commands, thats why I don't understand the "xdata and ydata must be the same length" error on the other code (data has the same length on both codes)</p>
  
  <p><strong>EDIT :</strong></p>
  
  <p>The program is receiving data via bluetooth from an arduino that is reading an analog pin. I measured the reading time of the arduino and it's taking "0.015ms" to read each value. To read and print each value it's taking "3.1ms". The reading time of the computer for each value is "100ms".  </p>
  
  <p>Which means it will take n*100(ms) to create the list, being n the number of elements of the list. The minimum 'n' I could use is 256, this would take 25.6s to create the list. Thats just too long for a real-time graph.</p>