title: Modify outliers caused by sensor-failures in timeseries data
body: <p>I am working with timeseries data collected from a sensor at 5min intervals. Unfortunately, there are cases when the measured value (PV yield in watts) is <strong>suddenly 0 or very high</strong>. The <strong>values before and after are correct</strong>:</p>
  <p><a href="https://i.stack.imgur.com/bkql4.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/bkql4.png" alt="enter image description here" /></a></p>
  <p>My goal is to identify these '<em>outliers</em>' and (in a second step) calculate the mean of the previous and next value to fix the measured value. I've experimented with two approaches so far, but am receiving many 'outliers' which are not measurement-errors. Hence, I am looking for better approaches.</p>
  <p><strong>Try 1: Classic outlier detection with IQR <a href="https://stackoverflow.com/questions/50461349/how-to-remove-outlier-from-dataframe-using-iqr">Source</a></strong></p>
  <pre><code>def updateOutliersIQR(group):
    Q1 = group.yield.quantile(0.25)
    Q3 = group.yield.quantile(0.75)
    IQR = Q3 - Q1
    outliers = (group.yield &lt; (Q1 - 1.5 * IQR)) | (group.yield &gt; (Q3 + 1.5 * IQR))
    print(outliers[outliers == True]) 
  
  # calling the function on a per-day level
  df.groupby(df.index.date).apply(updateOutliers)
  </code></pre>
  <p><strong>Try 2: kernel density estimation <a href="https://stackoverflow.com/questions/54048291/how-to-detect-a-sudden-change-in-a-time-series-plot-in-pandas">Source</a></strong></p>
  <pre><code>def updateOutliersKDE(group):
    a = 0.9
    r = group.yield.rolling(3, min_periods=1, win_type='parzen').sum()
    n = r.max()
    outliers = (r &gt; n*a)
    print(outliers[outliers == True]) 
  
  # calling the function on a per-day level
  df.groupby(df.index.date).apply(updateOutliers)
  </code></pre>
  <p><strong>Try 3: Median Filter <a href="https://stackoverflow.com/questions/46964363/filtering-out-outliers-in-pandas-dataframe-with-rolling-median">Source</a></strong>
  (As suggested by Jonnor)</p>
  <pre><code>def median_filter(num_std=3):
    def _median_filter(x):
      _median = np.median(x)
      _std = np.std(x)
      s = x[-3]
      if (s &gt;= _median - num_std * _std and s &lt;= _median + num_std * _std):
        return s
      else:
        return _median
    return _median_filter
  
  # calling the function
  df.yield.rolling(5, center=True).apply(median_filter(2), raw=True)
  </code></pre>
  <p>Edit: with try 3 and a window of 5 and std of 3, it finally catches the massive outlier, but will also loose accuracy of the other (non-faulty) sensor-measurements:
  <a href="https://i.stack.imgur.com/YLzha.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/YLzha.png" alt="enter image description here" /></a></p>
  <p>Are there any better approaches to detect the described 'outliers' or perform smoothing in timeseries data with the <strong>occasional</strong> sensor measurement issue?</p>