<p>A problem you may encountering is that you don't have enough training data for the model to be able to fit well. In your example, <strong>you only have 21 training instances, each with only 1 feature</strong>. Broadly speaking with neural network models, you need on the order of 10K or more training instances to produce a decent model.</p>

<p>Consider the following code that generates a noisy sine wave and tries to train a densely-connected feed-forward neural network to fit the data. My model has two linear layers, each with 50 hidden units and a ReLU activation function. The experiments are parameterized with the variable <code>num_points</code> which I will increase.</p>

<pre class="lang-py prettyprint-override"><code>import tensorflow as tf
from tensorflow import keras

from tensorflow.keras import layers
import numpy as np
import matplotlib.pyplot as plt


np.random.seed(7)

def generate_data(num_points=100):
    X = np.linspace(0.0 , 2.0 * np.pi, num_points).reshape(-1, 1)
    noise = np.random.normal(0, 1, num_points).reshape(-1, 1)
    y = 3 * np.sin(X) + noise
    return X, y

def run_experiment(X_train, y_train, X_test, batch_size=64):
    num_points = X_train.shape[0]

    model = keras.Sequential()
    model.add(layers.Dense(50, input_shape=(1, ), activation='relu'))
    model.add(layers.Dense(50, activation='relu'))
    model.add(layers.Dense(1, activation='linear'))
    model.compile(loss = "mse", optimizer = "adam", metrics=["mse"] )
    history = model.fit(X_train, y_train, epochs=10,
                        batch_size=batch_size, verbose=0)

    yhat = model.predict(X_test, batch_size=batch_size)
    plt.figure(figsize=(5, 5))
    plt.plot(X_train, y_train, "ro", markersize=2, label='True')
    plt.plot(X_train, yhat, "bo", markersize=1, label='Predicted')
    plt.ylim(-5, 5)
    plt.title('N=%d points' % (num_points))
    plt.legend()
    plt.grid()
    plt.show()
</code></pre>

<p>Here is how I invoke the code:</p>

<pre class="lang-py prettyprint-override"><code>num_points = 100
X, y = generate_data(num_points)
run_experiment(X, y, X)
</code></pre>

<p>Now, if I run the experiment with <code>num_points = 100</code>, the model predictions (in blue) do a terrible job at fitting the true noisy sine wave (in red).</p>

<p><a href="https://i.stack.imgur.com/pIlnM.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/pIlnM.png" alt="enter image description here"></a></p>

<p>Now, here is <code>num_points = 1000</code>:</p>

<p><a href="https://i.stack.imgur.com/GSUS7.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/GSUS7.png" alt="enter image description here"></a></p>

<p>Here is <code>num_points = 10000</code>:</p>

<p><a href="https://i.stack.imgur.com/SBgCK.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/SBgCK.png" alt="enter image description here"></a></p>

<p>And here is <code>num_points = 100000</code>:</p>

<p><a href="https://i.stack.imgur.com/sQDSG.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/sQDSG.png" alt="enter image description here"></a></p>

<p>As you can see, <em>for my chosen NN architecture</em>, adding more training instances allows the neural network to better (over)fit the data.</p>

<p>If you do have a lot of training instances, then if you want to purposefully overfit your data, you can either increase the neural network capacity or reduce regularization. Specifically, you can control the following knobs:</p>

<ul>
<li>increase the number of layers</li>
<li>increase the number of hidden units</li>
<li>increase the number of features per data instance</li>
<li>reduce regularization (e.g. by removing dropout layers)</li>
<li>use a more complex neural network architecture (e.g. transformer blocks instead of RNN)</li>
</ul>

<p>You may be wondering if neural networks can fit arbitrary data rather than just a noisy sine wave as in my example. Previous research says that, yes, a big enough neural network can fit any data. See:</p>

<ul>
<li>Universal approximation theorem. <a href="https://en.wikipedia.org/wiki/Universal_approximation_theorem" rel="nofollow noreferrer">https://en.wikipedia.org/wiki/Universal_approximation_theorem</a></li>
<li>Zhang 2016, "Understanding deep learning requires rethinking generalization". <a href="https://arxiv.org/abs/1611.03530" rel="nofollow noreferrer">https://arxiv.org/abs/1611.03530</a></li>
</ul>
How to overfit data with Keras?
<p>I'm trying to build a simple regression model using keras and tensorflow. In my problem I have data in the form <code>(x, y)</code>, where <code>x</code> and <code>y</code> are simply numbers. I'd like to build a keras model in order to predict <code>y</code> using <code>x</code> as an input.</p>

<p>Since I think images better explains thing, these are my data:</p>

<p><a href="https://i.stack.imgur.com/npCjz.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/npCjz.jpg" alt="enter image description here"></a></p>

<p>We may discuss if they are good or not, but in my problem I cannot really cheat them.</p>

<p>My keras model is the following (data are splitted 30% test <code>(X_test, y_test)</code> and 70% training <code>(X_train, y_train)</code>):</p>

<pre><code>model = tf.keras.Sequential()

model.add(tf.keras.layers.Dense(32, input_shape=() activation="relu", name="first_layer"))
model.add(tf.keras.layers.Dense(16, activation="relu", name="second_layer"))
model.add(tf.keras.layers.Dense(1, name="output_layer"))

model.compile(loss = "mean_squared_error", optimizer = "adam", metrics=["mse"] )

history = model.fit(X_train, y_train, epochs=500, batch_size=1, verbose=0, shuffle=False) 
eval_result = model.evaluate(X_test, y_test)
print("\n\nTest loss:", eval_result, "\n")

predict_Y = model.predict(X)
</code></pre>

<p>note: <code>X</code> contains both <code>X_test</code> and <code>X_train</code>.</p>

<p>Plotting the prediction I get (blue squares are the prediction <code>predict_Y</code>)</p>

<p><a href="https://i.stack.imgur.com/KKiRd.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/KKiRd.png" alt="enter image description here"></a></p>

<p>I'm playing a lot with layers, activation funztions and other parameters. My goal is to find the best parameters to train the model, but the actual question, here, is slightly different: in fact I have hard times to force the model to overfit the data (as you can see from the above results).</p>

<p>Does anyone have some sort of idea about how to reproduce overfitting?</p>

<p>This is the outcome I would like to get:
<a href="https://i.stack.imgur.com/4IvqX.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/4IvqX.png" alt="enter image description here"></a></p>

<p>(red dots are under blue squares!)</p>

<p>EDIT:</p>

<p>Here I provide you the data used in the example above: you can copy paste directly to a python interpreter:</p>

<pre><code>X_train = [0.704619794270697, 0.6779457393024553, 0.8207082120250023, 0.8588819357831449, 0.8692320257603844, 0.6878750931810429, 0.9556331888763945, 0.77677964510883, 0.7211381534179618, 0.6438319113259414, 0.6478339581502052, 0.9710222750072649, 0.8952188423349681, 0.6303124926673513, 0.9640316662124185, 0.869691568491902, 0.8320164648420931, 0.8236399177660375, 0.8877334038470911, 0.8084042532069621, 0.8045680821762038]
y_train = [0.7766424210611557, 0.8210846773655833, 0.9996114311913593, 0.8041331063189883, 0.9980525368790883, 0.8164056182686034, 0.8925487603333683, 0.7758207470960685, 0.37345286573743475, 0.9325789202459493, 0.6060269037514895, 0.9319771743389491, 0.9990691225991941, 0.9320002808310418, 0.9992560731072977, 0.9980241561997089, 0.8882905258641204, 0.4678339275898943, 0.9312152374846061, 0.9542371205095945, 0.8885893668675711]
X_test = [0.9749191829308574, 0.8735366740730178, 0.8882783211709133, 0.8022891400991644, 0.8650601322313454, 0.8697902997857514, 1.0, 0.8165876695985228, 0.8923841531760973]
y_test = [0.975653685270635, 0.9096752789481569, 0.6653736469114154, 0.46367666660348744, 0.9991817903431941, 1.0, 0.9111205717076893, 0.5264993912088891, 0.9989199241685126]
X = [0.704619794270697, 0.77677964510883, 0.7211381534179618, 0.6478339581502052, 0.6779457393024553, 0.8588819357831449, 0.8045680821762038, 0.8320164648420931, 0.8650601322313454, 0.8697902997857514, 0.8236399177660375, 0.6878750931810429, 0.8923841531760973, 0.8692320257603844, 0.8877334038470911, 0.8735366740730178, 0.8207082120250023, 0.8022891400991644, 0.6303124926673513, 0.8084042532069621, 0.869691568491902, 0.9710222750072649, 0.9556331888763945, 0.8882783211709133, 0.8165876695985228, 0.6438319113259414, 0.8952188423349681, 0.9749191829308574, 1.0, 0.9640316662124185]
Y = [0.7766424210611557, 0.7758207470960685, 0.37345286573743475, 0.6060269037514895, 0.8210846773655833, 0.8041331063189883, 0.8885893668675711, 0.8882905258641204, 0.9991817903431941, 1.0, 0.4678339275898943, 0.8164056182686034, 0.9989199241685126, 0.9980525368790883, 0.9312152374846061, 0.9096752789481569, 0.9996114311913593, 0.46367666660348744, 0.9320002808310418, 0.9542371205095945, 0.9980241561997089, 0.9319771743389491, 0.8925487603333683, 0.6653736469114154, 0.5264993912088891, 0.9325789202459493, 0.9990691225991941, 0.975653685270635, 0.9111205717076893, 0.9992560731072977]
</code></pre>

<p>Where <code>X</code> contains the list of the x values and <code>Y</code> the corresponding y value. (X_test, y_test) and (X_train, y_train) are two (non overlapping) subset of (X, Y).</p>

<p>To predict and show the model results I simply use matplotlib (imported as plt):</p>

<pre><code>predict_Y = model.predict(X)
plt.plot(X, Y, "ro", X, predict_Y, "bs")
plt.show()
</code></pre>