<p>I once trained a siamese network where I realised that if I use higher learning rates the training loss was going down smooth (as expected since that is what the neural network is learning) ,but saw huge ups and downs with the val loss.</p>
<p>This had never happened before when I was using lower learning rate (in the order of 1e-05). I believe that the train loss is actually false since recent papers have proved that large neural networks (I mean neural networks with more complexity) can learn random data flawlessly in the training set, though they performed extremely worse while validating them, I have attached the paper for your reference below which clearly explains this phenomena related to overfitting. So one can't conclude the overall model's performance by just observing the training data.</p>
<p>Though other parameters mentioned above also matter, but I guess one should start tweaking the learning rates initially in such a case before tweaking the model itself.</p>
<p>Link for the paper : <a href="https://arxiv.org/pdf/1611.03530" rel="nofollow noreferrer">https://arxiv.org/pdf/1611.03530</a></p>
<p>Please correct me if I am wrong...</p>
How to correct unstable loss and accuracy during training? (binary classification)
<p>I am currently working on a small binary classification project using the new keras API in tensorflow. The problem is a simplified version of the Higgs Boson challenge posted on Kaggle.com a few years back. The dataset shape is 2000x14, where the first 13 elements of each row form the input vector, and the 14th element is the corresponding label. Here is a sample of said dataset:</p>

<pre><code>86.043,52.881,61.231,95.475,0.273,77.169,-0.015,1.856,32.636,202.068, 2.432,-0.419,0.0,0
138.149,69.197,58.607,129.848,0.941,120.276,3.811,1.886,71.435,384.916,2.447,1.408,0.0,1
137.457,3.018,74.670,81.705,5.954,775.772,-8.854,2.625,1.942,157.231,1.193,0.873,0.824,1
</code></pre>

<p>I am relatively new to machine learning and tensorflow, but I am familiar with the higher level concepts such as loss functions, optimizers and activation functions. I have tried building various models inspired by examples of binary classification problems found online, but I am having difficulties with training the model. During training, the loss somethimes increases within the same epoch, leading to unstable learning. The accuracy hits a plateau around 70%. I have tried changing the learning rate and other hyperparameters but to no avail. In comparison, I have hardcoded a fully-connected feed forward neural net that reaches around 80-85% accuracy on the same problem.</p>

<p>Here is my current model:</p>

<pre><code>import tensorflow as tf
from tensorflow.python.keras.layers.core import Dense
import numpy as np
import pandas as pd

def normalize(array):
    return array/np.linalg.norm(array, ord=2, axis=1, keepdims=True)

x_train = pd.read_csv('data/labeled.csv', sep='\s+').iloc[:1800, :-1].values
y_train = pd.read_csv('data/labeled.csv', sep='\s+').iloc[:1800, -1:].values

x_test = pd.read_csv('data/labeled.csv', sep='\s+').iloc[1800:, :-1].values
y_test = pd.read_csv('data/labeled.csv', sep='\s+').iloc[1800:, -1:].values

x_train = normalize(x_train)
x_test = normalize(x_test)

model = tf.keras.Sequential()
model.add(Dense(9, input_dim=13, activation=tf.nn.sigmoid)
model.add(Dense(6, activation=tf.nn.sigmoid))
model.add(Dense(1, activation=tf.nn.sigmoid))

model.compile(optimizer='adam',
              loss='binary_crossentropy',
              metrics=['accuracy'])

model.fit(x_train, y_train, epochs=50)
model.evaluate(x_test, y_test)
</code></pre>

<p>As mentionned, some of the epochs start with a higher accuracy than they finish with, leading to unstable learning. </p>

<pre><code>  32/1800 [..............................] - ETA: 0s - loss: 0.6830 - acc: 0.5938
1152/1800 [==================&gt;...........] - ETA: 0s - loss: 0.6175 - acc: 0.6727
1800/1800 [==============================] - 0s 52us/step - loss: 0.6098 - acc: 0.6861
Epoch 54/250

  32/1800 [..............................] - ETA: 0s - loss: 0.5195 - acc: 0.8125
1376/1800 [=====================&gt;........] - ETA: 0s - loss: 0.6224 - acc: 0.6672
1800/1800 [==============================] - 0s 43us/step - loss: 0.6091 - acc: 0.6850
Epoch 55/250
</code></pre>

<p>What could be the cause of these oscillations in learning in such a simple model? 
Thanks</p>

<hr>

<p>EDIT:</p>

<p>I have followed some suggestions from the comments and have modified the model accordingly. It now looks more like this:</p>

<pre><code>model = tf.keras.Sequential()
model.add(Dense(250, input_dim=13, activation=tf.nn.relu))
model.add(Dropout(0.4))
model.add(Dense(200, activation=tf.nn.relu))
model.add(Dropout(0.4))
model.add(Dense(100, activation=tf.nn.relu))
model.add(Dropout(0.3))
model.add(Dense(50, activation=tf.nn.relu))
model.add(Dense(1, activation=tf.nn.sigmoid))

model.compile(optimizer='adadelta',
              loss='binary_crossentropy',
              metrics=['accuracy'])
</code></pre>