## Simple Linear Regression

In this example, we illustrate how to solve a linear regression problem.
Suppose we have training data, can we fit a neural network on it? The trained neural network (an MLP) is then validated on the test split.

Let's load the python packages first.


```python
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

# numpy package
import numpy as np
# for plotting
import matplotlib.pyplot as plt

# keras modules
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
from tensorflow.keras.utils import plot_model
```

### Generate Artificial Data

Assuming we have 1D input and 1D output, let us generate the train data assuminng the underlying function is a polynomial:
\begin{align}
y= f(x) = x^2 - 2x + 1   
\end{align}


```python
# generate train x data
x_train = np.random.uniform(low=-1.0, high=2.0, size=[200,1])
# generate train y data
y_train = x_train*x_train - 2*x_train + 1

plt.figure()
plt.plot(x_train, y_train, 'bo')
```

Then, let us generate the test data and plot the train and test data together. Note that the test data is beyond the [-1,2] range of `x_train`. This will enable us to verify if our model can generalize (ie make prediction from never seen before data)


```python
# generate test x data
x_test = np.random.uniform(low=-2.0, high=3.0, size=[20,1])
# generate test y data
y_test = x_test*x_test - 2*x_test + 1

plt.plot(x_train, y_train, 'bo', x_test, y_test, 'r*')
plt.show()
```

### 3-layer MLP Model

Then, let us build a 3-layer 64-128-1 MLP.


```python
# build 2-layer MLP network 
model = Sequential(name='linear_regressor')
# 1st layer (input layer) has 64 units (perceptron), input is 1-dim
model.add(Dense(units=64, input_dim=1, activation='relu', name='input_layer'))
# 2nd layer has 128 unit, output is 1-dim
model.add(Dense(units=128, activation='relu', name='hidden_layer'))
# 3rd layer (output layer) has 1 unit, output is 1-dim
model.add(Dense(units=1, name='output_layer'))
# print summary to double check the network
model.summary()
# indicate the loss function and use stochastic gradient descent
# (sgd) as optimizer
model.compile(loss='mse', optimizer='sgd')
# feed the network with complete dataset (1 epoch) 100 times
# batch size of sgd is 4
history = model.fit(x_train, y_train, epochs=100, batch_size=4, validation_data=(x_test, y_test))
# simple validation by predicting the output based on x
y_pred = model.predict(x_test, verbose=0)
```

    Model: "linear_regressor"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #   
    =================================================================
    input_layer (Dense)          (None, 64)                128       
    _________________________________________________________________
    hidden_layer (Dense)         (None, 128)               8320      
    _________________________________________________________________
    output_layer (Dense)         (None, 1)                 129       
    =================================================================
    Total params: 8,577
    Trainable params: 8,577
    Non-trainable params: 0
    _________________________________________________________________
    Epoch 1/100
    50/50 [==============================] - 0s 3ms/step - loss: 1.0276 - val_loss: 4.9706
    Epoch 2/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.1931 - val_loss: 2.7254
    Epoch 3/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0804 - val_loss: 2.2440
    Epoch 4/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0575 - val_loss: 1.8591
    Epoch 5/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0475 - val_loss: 1.6330
    Epoch 6/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0409 - val_loss: 1.5586
    Epoch 7/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0375 - val_loss: 1.4400
    Epoch 8/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0333 - val_loss: 1.4230
    Epoch 9/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0307 - val_loss: 1.3709
    Epoch 10/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0289 - val_loss: 1.2765
    Epoch 11/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0266 - val_loss: 1.3010
    Epoch 12/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0244 - val_loss: 1.2236
    Epoch 13/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0235 - val_loss: 1.2113
    Epoch 14/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0214 - val_loss: 1.2156
    Epoch 15/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0196 - val_loss: 1.1431
    Epoch 16/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0186 - val_loss: 1.1463
    Epoch 17/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0171 - val_loss: 1.1123
    Epoch 18/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0156 - val_loss: 1.0624
    Epoch 19/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0146 - val_loss: 1.0403
    Epoch 20/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0138 - val_loss: 1.0194
    Epoch 21/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0130 - val_loss: 0.9956
    Epoch 22/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0110 - val_loss: 0.9711
    Epoch 23/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0113 - val_loss: 0.9489
    Epoch 24/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0103 - val_loss: 0.9598
    Epoch 25/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0097 - val_loss: 0.9324
    Epoch 26/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0089 - val_loss: 0.9162
    Epoch 27/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0084 - val_loss: 0.9133
    Epoch 28/100
    50/50 [==============================] - 0s 2ms/step - loss: 0.0079 - val_loss: 0.9003
    Epoch 29/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0072 - val_loss: 0.9243
    Epoch 30/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0070 - val_loss: 0.8808
    Epoch 31/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0064 - val_loss: 0.8721
    Epoch 32/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0061 - val_loss: 0.8552
    Epoch 33/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0057 - val_loss: 0.8433
    Epoch 34/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0053 - val_loss: 0.8256
    Epoch 35/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0051 - val_loss: 0.8175
    Epoch 36/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0047 - val_loss: 0.7790
    Epoch 37/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0046 - val_loss: 0.7914
    Epoch 38/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0044 - val_loss: 0.7798
    Epoch 39/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0041 - val_loss: 0.7754
    Epoch 40/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0040 - val_loss: 0.7534
    Epoch 41/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0038 - val_loss: 0.7344
    Epoch 42/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0035 - val_loss: 0.7535
    Epoch 43/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0033 - val_loss: 0.7402
    Epoch 44/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0032 - val_loss: 0.7357
    Epoch 45/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0032 - val_loss: 0.7198
    Epoch 46/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0031 - val_loss: 0.7145
    Epoch 47/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0029 - val_loss: 0.7006
    Epoch 48/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0028 - val_loss: 0.6839
    Epoch 49/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0026 - val_loss: 0.7008
    Epoch 50/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0026 - val_loss: 0.6864
    Epoch 51/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0024 - val_loss: 0.6754
    Epoch 52/100
    50/50 [==============================] - 0s 946us/step - loss: 0.0024 - val_loss: 0.6899
    Epoch 53/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0024 - val_loss: 0.6637
    Epoch 54/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0022 - val_loss: 0.6664
    Epoch 55/100
    50/50 [==============================] - 0s 862us/step - loss: 0.0023 - val_loss: 0.6661
    Epoch 56/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0021 - val_loss: 0.6515
    Epoch 57/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0020 - val_loss: 0.6346
    Epoch 58/100
    50/50 [==============================] - 0s 913us/step - loss: 0.0020 - val_loss: 0.6373
    Epoch 59/100
    50/50 [==============================] - 0s 937us/step - loss: 0.0019 - val_loss: 0.6442
    Epoch 60/100
    50/50 [==============================] - 0s 931us/step - loss: 0.0018 - val_loss: 0.6325
    Epoch 61/100
    50/50 [==============================] - 0s 957us/step - loss: 0.0018 - val_loss: 0.6241
    Epoch 62/100
    50/50 [==============================] - 0s 968us/step - loss: 0.0018 - val_loss: 0.6105
    Epoch 63/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0017 - val_loss: 0.6168
    Epoch 64/100
    50/50 [==============================] - 0s 944us/step - loss: 0.0017 - val_loss: 0.6222
    Epoch 65/100
    50/50 [==============================] - 0s 934us/step - loss: 0.0016 - val_loss: 0.6176
    Epoch 66/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0016 - val_loss: 0.6129
    Epoch 67/100
    50/50 [==============================] - 0s 930us/step - loss: 0.0015 - val_loss: 0.5915
    Epoch 68/100
    50/50 [==============================] - 0s 917us/step - loss: 0.0015 - val_loss: 0.6006
    Epoch 69/100
    50/50 [==============================] - 0s 908us/step - loss: 0.0014 - val_loss: 0.5912
    Epoch 70/100
    50/50 [==============================] - 0s 990us/step - loss: 0.0014 - val_loss: 0.5981
    Epoch 71/100
    50/50 [==============================] - 0s 980us/step - loss: 0.0014 - val_loss: 0.5944
    Epoch 72/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0014 - val_loss: 0.5867
    Epoch 73/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0013 - val_loss: 0.5740
    Epoch 74/100
    50/50 [==============================] - 0s 962us/step - loss: 0.0013 - val_loss: 0.5755
    Epoch 75/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0013 - val_loss: 0.5755
    Epoch 76/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0012 - val_loss: 0.5674
    Epoch 77/100
    50/50 [==============================] - 0s 846us/step - loss: 0.0012 - val_loss: 0.5701
    Epoch 78/100
    50/50 [==============================] - 0s 900us/step - loss: 0.0012 - val_loss: 0.5657
    Epoch 79/100
    50/50 [==============================] - 0s 930us/step - loss: 0.0011 - val_loss: 0.5661
    Epoch 80/100
    50/50 [==============================] - 0s 906us/step - loss: 0.0011 - val_loss: 0.5600
    Epoch 81/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0011 - val_loss: 0.5582
    Epoch 82/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0011 - val_loss: 0.5476
    Epoch 83/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0011 - val_loss: 0.5522
    Epoch 84/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0010 - val_loss: 0.5569
    Epoch 85/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0010 - val_loss: 0.5534
    Epoch 86/100
    50/50 [==============================] - 0s 1ms/step - loss: 9.9959e-04 - val_loss: 0.5383
    Epoch 87/100
    50/50 [==============================] - 0s 1ms/step - loss: 9.7837e-04 - val_loss: 0.5421
    Epoch 88/100
    50/50 [==============================] - 0s 1ms/step - loss: 9.6033e-04 - val_loss: 0.5373
    Epoch 89/100
    50/50 [==============================] - 0s 1ms/step - loss: 9.5100e-04 - val_loss: 0.5317
    Epoch 90/100
    50/50 [==============================] - 0s 1ms/step - loss: 8.7060e-04 - val_loss: 0.5299
    Epoch 91/100
    50/50 [==============================] - 0s 1ms/step - loss: 9.2609e-04 - val_loss: 0.5346
    Epoch 92/100
    50/50 [==============================] - 0s 1ms/step - loss: 8.8382e-04 - val_loss: 0.5254
    Epoch 93/100
    50/50 [==============================] - 0s 1ms/step - loss: 8.7679e-04 - val_loss: 0.5178
    Epoch 94/100
    50/50 [==============================] - 0s 1ms/step - loss: 8.3923e-04 - val_loss: 0.5225
    Epoch 95/100
    50/50 [==============================] - 0s 1ms/step - loss: 8.1455e-04 - val_loss: 0.5237
    Epoch 96/100
    50/50 [==============================] - 0s 1ms/step - loss: 8.0873e-04 - val_loss: 0.5184
    Epoch 97/100
    50/50 [==============================] - 0s 1ms/step - loss: 8.0517e-04 - val_loss: 0.5233
    Epoch 98/100
    50/50 [==============================] - 0s 993us/step - loss: 7.8329e-04 - val_loss: 0.5172
    Epoch 99/100
    50/50 [==============================] - 0s 1ms/step - loss: 7.5604e-04 - val_loss: 0.5214
    Epoch 100/100
    50/50 [==============================] - 0s 974us/step - loss: 7.6083e-04 - val_loss: 0.5108
    WARNING:tensorflow:11 out of the last 11 calls to <function Model.make_predict_function.<locals>.predict_function at 0x7f9590811440> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for  more details.


### Qualitative Evaluation
Let us plot the prediction (green dots) vs test dataset (red stars).
We can see that the test data deviates largely when the input is beyond the range of the train data.


```python
plt.plot(x_test, y_test, 'r*', x_test, y_pred, 'go', )
plt.show()
```

### Examine the Train History

The `history` variable stores value informationn during training such as value of loss function.


```python
print(history.history.keys())
```

    dict_keys(['loss', 'val_loss'])



```python
# Plot history: MSE
plt.plot(history.history['loss'], label='Train loss')
plt.plot(history.history['val_loss'], label='Test loss')
plt.title('MSE')
plt.ylabel('MSE value')
plt.xlabel('Epochs')
plt.legend(loc="upper right")
plt.show()
```


```python

```

56e287de6756f54018aaa2af297e7ef3744b7cfb
ipynb
Jupyter Notebook
versions/2020/MLP/code/tf.keras/linear.ipynb
Cahlil-Togonon/Deep-Learning-Experiments
501ae610b0a8fb7fb75a53dcfdab71be49274b58
MIT
2017-01-17T11:56:51.000Z
2022-03-22T11:51:40.000Z
versions/2020/MLP/code/tf.keras/linear.ipynb
Cahlil-Togonon/Deep-Learning-Experiments
501ae610b0a8fb7fb75a53dcfdab71be49274b58
MIT
2017-06-01T01:30:16.000Z
2021-06-11T17:27:51.000Z
versions/2020/MLP/code/tf.keras/linear.ipynb
Cahlil-Togonon/Deep-Learning-Experiments
501ae610b0a8fb7fb75a53dcfdab71be49274b58
MIT
2017-02-16T08:53:14.000Z
2022-03-27T14:33:39.000Z
Qwen/Qwen-72B
1. YES
2. YES