<p>But ML might be a good solution. It does everything you want and solves anything easily...at least you might get the impression listening to the buuuuuzzword people :)</p>

<p>Using only the distribution of the value will probably not work. But you could try to use a combination of distribution of values and distribution of gradients to make an educated guess.</p>

<p>This <em>might</em> be literature that might describe the approach</p>

<p><a href="https://rstudio-pubs-static.s3.amazonaws.com/161075_05ce98dc51c844e0833c06835c9ce4c3.html" rel="nofollow noreferrer">https://rstudio-pubs-static.s3.amazonaws.com/161075_05ce98dc51c844e0833c06835c9ce4c3.html</a></p>

<p><a href="https://arxiv.org/pdf/1905.08850" rel="nofollow noreferrer">https://arxiv.org/pdf/1905.08850</a></p>
smoothing data and forecast next data point
<p>I'm working with time series data which roughly oscillates and has around 3000 data points. There are 2 things I would like to accomplish:
1) smooth data to remove jagged edges
2) predict the next data point with weight based off location in the distribution diagram</p>

<p>I have included the data distribution diagram and a sample plot of the data in blue with the data points in black. The yellow line represents Lowess smoothed data points with a frac of 4/len(df) so the window for local regression stays consistent with additional data.The problem is that it is horrific for predicting the next data point. Simple /Exponential Moving average is not an option because of the lag. I have used several scipy modules under signal and optimize such as curve_fit but have not found anything close to match statsmodel lowess accuracy besides the forecasting of the next data point. I'm trying to stay away from going to ML if possible.</p>

<p>My searching has been pointing to using a gaussian process with bayesian optimization but this is a bit over my abilities to implement as my own custom function.</p>

<p>If I am stuck building my own custom function, any links or feedback how to proceed would be greatly appreciated. </p>

<pre><code>c_list = [2.8, 2.1, 4.0, 4.7, 4.7, 3.0, 0.2, -0.4, -3.2, 1.0, 4.0, -3.7, -3.7, -4.3, -2.7, 0.2, 3.4, 4.3, 4.2, 3.8, -0.3, 2.4, -0.2, -0.2, -2.6, -3.3, -4.3, -3.6, 0.5, 0.3, 0.9, 3.3, 3.3, 3.6, 3.9, 4.1, -0.3, -0.9, -2.9, -0.9, 1.9, 2.8, 4.4, 3.9, 3.3, -2.6, -3.1, -3.2, -0.2, 3.2]

c_series = pd.Series(c_list)
x = c_series.index.values
y = c_series.values
window = 4/len(c_series)

l = lowess(y, x, window)

c_series.plot()
plt.scatter(x, y, s=9)
plt.plot(l[:,1])
</code></pre>

<p><a href="https://i.stack.imgur.com/3mqtw.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/3mqtw.png" alt="Ploted data points in blue with lowess in yellow"></a></p>

<p><a href="https://i.stack.imgur.com/txnc1.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/txnc1.png" alt="Distribution graph of data points"></a></p>