title: Filtering 1D numpy arrays in Python
body: <p><strong>Explanation:</strong></p>
  
  <p>I have two numpy arrays: dataX and dataY, and I am trying to filter each array to reduce the noise. The image shown below shows the actual input data (blue dots) and an example of what I want it to be like(red dots). I do not need the filtered data to be as perfect as in the example but I do want it to be as straight as possible. I have provided sample data in the code. 
  <a href="https://i.stack.imgur.com/7Murx.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/7Murx.png" alt="enter image description here"></a></p>
  
  <p><strong>What I have tried:</strong></p>
  
  <p>Firstly, you can see that the data isn't 'continuous', so I first divided them into individual 'segments' ( 4 of them in this example), and then applied a filter to each 'segment'. Someone suggested that I use a <a href="https://docs.scipy.org/doc/scipy-1.0.0/reference/generated/scipy.signal.savgol_filter.html#scipy.signal.savgol_filter" rel="nofollow noreferrer">Savitzky-Golay filter</a>. The full, run-able code is below:</p>
  
  <pre><code>import scipy as sc
  import scipy.signal
  import numpy as np
  import matplotlib.pyplot as plt
  
  # Sample Data
  ydata = np.array([1,0,1,2,1,2,1,0,1,1,2,2,0,0,1,0,1,0,1,2,7,6,8,6,8,6,6,8,6,6,8,6,6,7,6,5,5,6,6, 10,11,12,13,12,11,10,10,11,10,12,11,10,10,10,10,12,12,10,10,17,16,15,17,16, 17,16,18,19,18,17,16,16,16,16,16,15,16])
  xdata = np.array([1,2,3,1,5,4,7,8,6,10,11,12,13,10,12,13,17,16,19,18,21,19,23,21,25,20,26,27,28,26,26,26,29,30,30,29,30,32,33, 1,2,3,1,5,4,7,8,6,10,11,12,13,10,12,13,17,16,19,18,21,19,23,21,25,20,26,27,28,26,26,26,29,30,30,29,30,32])
  
  # Used a diff array to find where there is a big change in Y. 
  # If there's a big change in Y, then there must be a change of 'segment'.
  diffy = np.diff(ydata)
  
  # Create empty numpy arrays to append values into
  filteredX = np.array([])
  filteredY = np.array([])
  
  # Chose 3 to be the value indicating the change in Y
  index = np.where(diffy &gt;3)
  
  # Loop through the array
  start = 0
  for i in range (0, (index[0].size +1) ):
  # Check if last segment is reached
      if i == index[0].size:
          print xdata[start:]
          partSize = xdata[start:].size
  # Window length must be an odd integer
          if partSize % 2 == 0:
              partSize = partSize - 1
  
          filteredDataX = sc.signal.savgol_filter(xdata[start:], partSize, 3)
          filteredDataY = sc.signal.savgol_filter(ydata[start:], partSize, 3)
          filteredX = np.append(filteredX, filteredDataX)
          filteredY = np.append(filteredY, filteredDataY)
  
      else:
          print xdata[start:index[0][i]]
          partSize = xdata[start:index[0][i]].size
          if partSize % 2 == 0:
              partSize = partSize - 1
          filteredDataX = sc.signal.savgol_filter(xdata[start:index[0][i]], partSize, 3)
          filteredDataY = sc.signal.savgol_filter(ydata[start:index[0][i]], partSize, 3)
          start = index[0][i] 
          filteredX = np.append(filteredX, filteredDataX)
          filteredY = np.append(filteredY, filteredDataY)
  
  # Plots
  plt.plot(xdata,ydata, 'bo', label = 'Input Data')
  plt.plot(filteredX, filteredY, 'ro', label = 'Filtered Data')
  
  plt.xlabel('X')
  plt.ylabel('Y')
  plt.title('Result')
  plt.legend()
  plt.show()
  </code></pre>
  
  <p>This is my result:
  <a href="https://i.stack.imgur.com/HI1m5.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/HI1m5.png" alt="enter image description here"></a>
  When each point is connected, the result looks as follows.
  <a href="https://i.stack.imgur.com/sfdEo.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/sfdEo.png" alt="enter image description here"></a>
  I have played around with the order, but it seems like a third order gave the best result.</p>
  
  <p>I have also tried these filters, among a few others:</p>
  
  <ul>
  <li><a href="https://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.signal.medfilt.html" rel="nofollow noreferrer">scipy.signal.medfilt</a> </li>
  <li><a href="https://docs.scipy.org/doc/scipy-0.15.1/reference/generated/scipy.ndimage.filters.uniform_filter1d.html" rel="nofollow noreferrer">scipy.ndimage.filters.uniform_filter1d</a></li>
  </ul>
  
  <p>But so far none of the filters I have tried were close to what I really wanted. What is the best way to filter data such as this? Looking forward to your help. </p>