title: How to correct unstable loss and accuracy during training? (binary classification)
body: <p>I am currently working on a small binary classification project using the new keras API in tensorflow. The problem is a simplified version of the Higgs Boson challenge posted on Kaggle.com a few years back. The dataset shape is 2000x14, where the first 13 elements of each row form the input vector, and the 14th element is the corresponding label. Here is a sample of said dataset:</p>
  
  <pre><code>86.043,52.881,61.231,95.475,0.273,77.169,-0.015,1.856,32.636,202.068, 2.432,-0.419,0.0,0
  138.149,69.197,58.607,129.848,0.941,120.276,3.811,1.886,71.435,384.916,2.447,1.408,0.0,1
  137.457,3.018,74.670,81.705,5.954,775.772,-8.854,2.625,1.942,157.231,1.193,0.873,0.824,1
  </code></pre>
  
  <p>I am relatively new to machine learning and tensorflow, but I am familiar with the higher level concepts such as loss functions, optimizers and activation functions. I have tried building various models inspired by examples of binary classification problems found online, but I am having difficulties with training the model. During training, the loss somethimes increases within the same epoch, leading to unstable learning. The accuracy hits a plateau around 70%. I have tried changing the learning rate and other hyperparameters but to no avail. In comparison, I have hardcoded a fully-connected feed forward neural net that reaches around 80-85% accuracy on the same problem.</p>
  
  <p>Here is my current model:</p>
  
  <pre><code>import tensorflow as tf
  from tensorflow.python.keras.layers.core import Dense
  import numpy as np
  import pandas as pd
  
  def normalize(array):
      return array/np.linalg.norm(array, ord=2, axis=1, keepdims=True)
  
  x_train = pd.read_csv('data/labeled.csv', sep='\s+').iloc[:1800, :-1].values
  y_train = pd.read_csv('data/labeled.csv', sep='\s+').iloc[:1800, -1:].values
  
  x_test = pd.read_csv('data/labeled.csv', sep='\s+').iloc[1800:, :-1].values
  y_test = pd.read_csv('data/labeled.csv', sep='\s+').iloc[1800:, -1:].values
  
  x_train = normalize(x_train)
  x_test = normalize(x_test)
  
  model = tf.keras.Sequential()
  model.add(Dense(9, input_dim=13, activation=tf.nn.sigmoid)
  model.add(Dense(6, activation=tf.nn.sigmoid))
  model.add(Dense(1, activation=tf.nn.sigmoid))
  
  model.compile(optimizer='adam',
                loss='binary_crossentropy',
                metrics=['accuracy'])
  
  model.fit(x_train, y_train, epochs=50)
  model.evaluate(x_test, y_test)
  </code></pre>
  
  <p>As mentionned, some of the epochs start with a higher accuracy than they finish with, leading to unstable learning. </p>
  
  <pre><code>  32/1800 [..............................] - ETA: 0s - loss: 0.6830 - acc: 0.5938
  1152/1800 [==================&gt;...........] - ETA: 0s - loss: 0.6175 - acc: 0.6727
  1800/1800 [==============================] - 0s 52us/step - loss: 0.6098 - acc: 0.6861
  Epoch 54/250
  
    32/1800 [..............................] - ETA: 0s - loss: 0.5195 - acc: 0.8125
  1376/1800 [=====================&gt;........] - ETA: 0s - loss: 0.6224 - acc: 0.6672
  1800/1800 [==============================] - 0s 43us/step - loss: 0.6091 - acc: 0.6850
  Epoch 55/250
  </code></pre>
  
  <p>What could be the cause of these oscillations in learning in such a simple model? 
  Thanks</p>
  
  <hr>
  
  <p>EDIT:</p>
  
  <p>I have followed some suggestions from the comments and have modified the model accordingly. It now looks more like this:</p>
  
  <pre><code>model = tf.keras.Sequential()
  model.add(Dense(250, input_dim=13, activation=tf.nn.relu))
  model.add(Dropout(0.4))
  model.add(Dense(200, activation=tf.nn.relu))
  model.add(Dropout(0.4))
  model.add(Dense(100, activation=tf.nn.relu))
  model.add(Dropout(0.3))
  model.add(Dense(50, activation=tf.nn.relu))
  model.add(Dense(1, activation=tf.nn.sigmoid))
  
  model.compile(optimizer='adadelta',
                loss='binary_crossentropy',
                metrics=['accuracy'])
  </code></pre>