title: Use keras in multiprocessing
body: <p>This is basically a duplicate of: <a href="https://stackoverflow.com/questions/42504669/keras-tensorflow-and-multiprocessing-in-python">Keras + Tensorflow and Multiprocessing in Python</a>
  But my setup is a bit different, and their solution doesn't work for me.</p>
  
  <p>I need to train a keras model against predictions made from another model.
  The predictions are connected to some CPU heavy code, so I would like to parallelize them and have the code run in worker processes.
  Here is the code I would like to execute:</p>
  
  <pre><code>import numpy as np
  
  from keras.layers import Input, Dense
  from keras.models import Model
  from keras.optimizers import Adam
  
  def create_model():
      input_layer = Input((10,))
      dense = Dense(10)(input_layer)
  
      return Model(inputs=input_layer, outputs=dense)
  
  model_outside = create_model()
  model_outside.compile(Adam(1e-3), "mse")
  
  def subprocess_routine(weights):
      model_inside = create_model()
      model_inside.set_weights(weights)
  
      while True:
          # lots of CPU
          batch = np.random.rand(10, 10)
          prediction = model_inside.predict(batch)
  
          yield batch, prediction
  
  weights = model_outside.get_weights()
  
  model_outside.fit_generator(subprocess_routine(weights),
                              epochs=10,
                              steps_per_epoch=100,
                              use_multiprocessing=True,
                              workers=1)
  </code></pre>
  
  <p>This produces an error</p>
  
  <blockquote>
    <p>E tensorflow/core/grappler/clusters/utils.cc:81] Failed to get device
    properties, error code: 3</p>
  </blockquote>
  
  <p>I found the above question, the answer is to move keras imports into the subprocess. I have added all imports into the <code>subprocess_routine</code>. But that doesn't change the error. It would probably be necessary to eliminate keras imports altogether from the main process, but in my setup, that would mean huge refactorings.</p>
  
  <p>Keras + multithreading seems to work. In this issue, scroll down to the very last comment:
  <a href="https://github.com/keras-team/keras/issues/5640" rel="nofollow noreferrer">https://github.com/keras-team/keras/issues/5640</a>
  In my code, it looks like this:</p>
  
  <pre><code>model_inside = create_model()
  model_inside._make_predict_function()
  
  graph = tf.get_default_graph()
  
  def subprocess_routine(model_inside, graph):
  
      while True:
          batch = np.random.rand(10, 10)
  
          with graph.as_default():
              prediction = model_inside.predict(batch)
  
          yield batch, prediction
  
  model_outside.fit_generator(subprocess_routine(model_inside, graph),
                              epochs=10,
                              steps_per_epoch=100,
                              use_multiprocessing=True,
                              workers=1)
  </code></pre>
  
  <p>But the error message is identical.</p>
  
  <p>Since the problem is apparently related to initialization of the subprocesses, I tried to create a new session in each subprocess:</p>
  
  <pre><code>def subprocess_routine(weights):
  
      import keras.backend as K
      import tensorflow as tf
      sess = tf.Session()
      K.set_session(sess)
  
      model_inside = create_model()
      model_inside.set_weights(weights)
  
      while True:
          batch = np.random.rand(10, 10)
          prediction = model_inside.predict(batch)
  
          yield batch, prediction
  </code></pre>
  
  <p>It produces a variation on the same error message: </p>
  
  <blockquote>
    <p>E tensorflow/stream_executor/cuda/cuda_driver.cc:1300] could not
    retrieve CUDA device count: CUDA_ERROR_NOT_INITIALIZED</p>
  </blockquote>
  
  <p>So again, the initialization seems broken.</p>
  
  <p>How can I run keras both in my main process and subprocesses spawned by multiprocessing ?</p>