title: Why can't I get reproducible results in Keras even though I set the random seeds?
body: <p>I'm training a MobileNet architecture on dummy data with Keras, on Mac OSX. I set both <code>nump.random</code> and <code>tensorflow.set_random_seed</code>, but for some reasons I can't get reproducible results: each time I rerun the code, I get different results. Why? This is not due to the GPU, because I'm running on a MacBook Pro 2017 which has a Radeon graphics card, thus Tensorflow doesn't take advantage of it. The code is run with</p>
  
  <pre><code>python Keras_test.py
  </code></pre>
  
  <p>so it's not a problem of state (I'm not using Jupyter or IPython: the environment should be reset each time I run the code). </p>
  
  <p><strong>EDIT</strong>: I changed my code by moving the setting of all seeds <em>before</em> importing Keras. The results are still not deterministic, however the variance in results is much smaller than it was before. This is very bizarre.</p>
  
  <p>The current model is very small (as far as deep neural network go) without being trivial, it doesn't need a GPU to run and it trains in a few minutes on a modern laptop, so repeating my experiments is within anyone's reach. I invite you to do it: I'd be very interested in learning about the level of variation from a system to another.</p>
  
  <pre><code>import numpy as np
  # random seeds must be set before importing keras &amp; tensorflow
  my_seed = 512
  np.random.seed(my_seed)
  import random 
  random.seed(my_seed)
  import tensorflow as tf
  tf.set_random_seed(my_seed)
  
  # now we can import keras
  import keras.utils
  from keras.applications import MobileNet
  from keras.callbacks import ModelCheckpoint
  from keras.optimizers import Adam
  import os
  
  height = 224
  width = 224
  channels = 3
  epochs = 10
  num_classes = 10
  
  
  
  # Generate dummy data
  batch_size = 32  
  n_train = 256
  n_test = 64
  x_train = np.random.random((n_train, height, width, channels))
  y_train = keras.utils.to_categorical(np.random.randint(num_classes, size=(n_train, 1)), num_classes=num_classes)
  x_test = np.random.random((n_test, height, width, channels))
  y_test = keras.utils.to_categorical(np.random.randint(num_classes, size=(n_test, 1)), num_classes=num_classes)
  # Get input shape
  input_shape = x_train.shape[1:]
  
  # Instantiate model 
  model = MobileNet(weights=None,
                    input_shape=input_shape,
                    classes=num_classes)
  
  model.compile(loss='categorical_crossentropy',
                optimizer='adam',
                metrics=['accuracy'])
  # Viewing Model Configuration
  model.summary()
  
  # Model file name
  filepath = 'model_epoch_{epoch:02d}_loss_{loss:0.2f}_val_{val_loss:.2f}.hdf5'
  
  # Define save_best_only checkpointer
  checkpointer = ModelCheckpoint(filepath=filepath,
                               monitor='val_acc',
                               verbose=1,
                               save_best_only=True)
  
  # Let's fit!
  model.fit(x_train, y_train,
            batch_size=batch_size,
            epochs=epochs,
            validation_data=(x_test, y_test),
            callbacks=[checkpointer])
  </code></pre>
  
  <p>As always, here are my Python, Keras &amp; Tensorflow versions:</p>
  
  <pre><code>python -c 'import keras; import tensorflow; import sys; print(sys.version, 'keras.__version__', 'tensorflow.__version__')'
  /anaconda2/lib/python2.7/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
    from ._conv import register_converters as _register_converters
  Using TensorFlow backend.
  ('2.7.15 |Anaconda, Inc.| (default, May  1 2018, 18:37:05) \n[GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)]', '2.1.6', '1.8.0')
  </code></pre>
  
  <p>Here are some results obtained running this code multiple times: as you can see, the code saves the best model (best validation accuracy) out of 10 epochs with a descriptive filename, so comparing filenames across different runs allows to judge the variability in results.</p>
  
  <pre><code>model_epoch_01_loss_2.39_val_3.28.hdf5
  model_epoch_01_loss_2.39_val_3.54.hdf5
  model_epoch_01_loss_2.40_val_3.47.hdf5
  model_epoch_01_loss_2.41_val_3.08.hdf5
  </code></pre>