title: Keras CNN overfitting for more than four classes
body: <p>I'm trying to train a classifier on <a href="https://quickdraw.withgoogle.com/" rel="nofollow noreferrer">Google QuickDraw</a> drawings using <a href="https://keras.io" rel="nofollow noreferrer">Keras</a>:</p>
  
  <pre><code>import numpy as np
  from tensorflow.keras.layers import Conv2D, Dense, Flatten, MaxPooling2D
  from tensorflow.keras.models import Sequential
  
  model = Sequential()
  
  model.add(Conv2D(filters=32, kernel_size=5, data_format="channels_last", activation="relu", input_shape=(28, 28, 1)))
  model.add(MaxPooling2D(data_format="channels_last"))
  model.add(Conv2D(filters=16, kernel_size=3, data_format="channels_last", activation="relu"))
  model.add(MaxPooling2D(data_format="channels_last"))
  
  model.add(Flatten(data_format="channels_last"))
  
  model.add(Dense(units=128, activation="relu"))
  model.add(Dense(units=64, activation="relu"))
  model.add(Dense(units=4, activation="softmax"))
  
  model.compile(optimizer="adam", loss="categorical_crossentropy", metrics=["accuracy"])
  
  x = np.load("./x.npy")
  y = np.load("./y.npy")
  
  model.fit(x=x, y=y, batch_size=100, epochs=40, validation_split=0.2)
  </code></pre>
  
  <p>The input data is a 4d array with 12000 normalized images (28 x 28 x 1) per class. The output data is an array of one hot encoded vectors.</p>
  
  <p>If I train this model on four classes, it produces convincing results:</p>
  
  <p><a href="https://i.stack.imgur.com/l6jeA.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/l6jeA.png" alt="loss 4 classes"></a>
  <a href="https://i.stack.imgur.com/wRcbA.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/wRcbA.png" alt="accuracy 4 classes"></a></p>
  
  <p>(red is training data, blue is validation data)</p>
  
  <p>I know the model is slightly overfitted. However, I want to keep the architecture as simple as possible, so I accepted that.</p>
  
  <p>My problem is that as soon as I add just one arbitrary class, the model starts to overfit extremely:</p>
  
  <p><a href="https://i.stack.imgur.com/6skTf.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/6skTf.png" alt="loss 5 classes"></a>
  <a href="https://i.stack.imgur.com/vcmAM.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/vcmAM.png" alt="accuracy 5 classes"></a></p>
  
  <p>I tried many different things to prevent it from overfitting such as Batch Normalization, Dropout, Kernel Regularizers, much more training data and different batch sizes, none of which caused any significant improvement.</p>
  
  <p>What could be the reason why my CNN overfits so much?</p>
  
  <hr>
  
  <p><strong>EDIT:</strong> This is the code I used to create <code>x.npy</code> and <code>y.npy</code>:</p>
  
  <pre><code>import numpy as np
  
  from tensorflow.keras.utils import to_categorical
  
  files = ['cat.npy', 'dog.npy', 'apple.npy', 'banana.npy', 'flower.npy']
  
  SAMPLES = 12000
  
  x = np.concatenate([np.load(f'./data/{f}')[:SAMPLES] for f in files]) / 255.0
  y = np.concatenate([np.full(SAMPLES, i) for i in range(len(files))])
  
  # (samples, rows, cols, channels)
  x = x.reshape(x.shape[0], 28, 28, 1).astype('float32')
  y = to_categorical(y)
  
  np.save('./x.npy', x)
  np.save('./y.npy', y)
  </code></pre>
  
  <p>The <code>.npy</code> files come from <a href="https://github.com/googlecreativelab/quickdraw-dataset#numpy-bitmaps-npy" rel="nofollow noreferrer">here</a>.</p>