title: How to do inference in parallel with tensorflow saved model predictors?
body: <p>Tensorflow version: 1.14</p>
  
  <p>Our current setup is using tensorflow estimators to do live NER i.e. perform inference one document at a time. We have 30 different fields to extract, and we run one model per field, so got total of 30 models. </p>
  
  <p>Our current setup uses python multiprocessing to do the inferences in parallel. (The inference is done on CPUs.) This approach reloads the model weights each time a prediction is made.</p>
  
  <p>Using the approach mentioned <a href="https://guillaumegenthial.github.io/serving-tensorflow-estimator.html" rel="nofollow noreferrer">here</a>, we exported the estimator models as <code>tf.saved_model</code>. This works as expected in that it does not reload the weights for each request. It also works fine for a single field inference in one process, but doesn't work with multiprocessing. All the processes hang when we make the predict function (<code>predict_fn</code> in the linked post) call.</p>
  
  <p><a href="https://stackoverflow.com/questions/36610290/tensorflow-and-multiprocessing-passing-sessions">This post</a> is related, but not sure how to adapt it for saved model.</p>
  
  <p>Importing tensorflow individually for each of the predictors did not work either:</p>
  
  <pre><code>class SavedModelPredictor():
  
      def __init__(self, model_path):
          import tensorflow as tf
          self.predictor_fn = tf.contrib.predictor.from_saved_model(model_path)
  
      def predictor_fn(self, input_dict):
          return self.predictor_fn(input_dict)
  </code></pre>
  
  <p>How to make <code>tf.saved_model</code> work with multiprocessing?</p>