title: Use multiple GPUs in TensorFlow to inference with pb model
body: <p>I use a server with 8 Titan X, trying to predict image much faster than use a single one GPU.
  I load the PB model like this:</p>
  
  <pre><code>model_dir = "./model"
      model = "nasnet_large_v1.pb"
      model_path = os.path.join(model_dir, model)
      model_graph = tf.Graph()
      with model_graph.as_default():
          with tf.gfile.GFile(model_path, 'rb') as f:
              graph_def = tf.GraphDef()
              graph_def.ParseFromString(f.read())
              _ = tf.import_graph_def(graph_def, name='')
              input_layer = model_graph.get_tensor_by_name("input:0")
              output_layer = model_graph.get_tensor_by_name('final_layer/predictions:0')
  </code></pre>
  
  <p>then I start iterating files in the <code>./data_input</code> directory like this:</p>
  
  <pre><code>with tf.Session(graph = model_graph, config=config) as inference_session:
          # Initialize session
          initializer = np.zeros([1, 331, 331, 3])
          print("Initialing session...")
          inference_session.run(output_layer, feed_dict={input_layer: initializer})
          print("Done initialing.")
  
          # Prediction
          file_list = []
          processed_files = []
  
          for path, dir, files in os.walk('./model_output/processed_files'):
              for file in files:
                  processed_files.append(file.split('_')[0]+'.tfrecord')
  
          print("Processed files: ")
          for f in processed_files:
              print('\t', f)
  
          while True:
              for path, dir, files in os.walk("./data_input"):
                  for file in files:
                      if file == '.DS_Store': continue
                      if file in processed_files: continue
                      print("Reading file {}".format(file))
                      file_path = os.path.join('./data_input', file)
                      file_list.append(file_path)
                      res = predict(file_path)
                      processed_files.append(file)
  
                      with open('./model_output/processed_files/{}_{}_processed_files.json'.format(file.split('.')[0], model.split('.')[0]), 'w') as f:
                          f.write(json.dumps(processed_files))
  
                      with open('./model_output/classify_result/{}_{}_classify_result.json'.format(file.split('.')[0], model.split('.')[0]), 'w') as f:
                          f.write(json.dumps(res, indent=4, separators=(',',':')))
  
              time.sleep(1)
  </code></pre>
  
  <p>in the <code>predict()</code> function, I wrote code like this:</p>
  
  <pre><code>label_map = get_label()
      # read tfrecord file by tf.data
      dataset = get_dataset(filename)
      # dataset.apply(tf.contrib.data.prefetch_to_device("/gpu:0"))
      # load data
      iterator = dataset.make_one_shot_iterator()
      features = iterator.get_next()
  
      result = []
      content = {}
      count = 0
      # session
      with tf.Session() as sess:
          tf.global_variables_initializer()
          t1 = time.time()
          try:
              while True:
                  [_image, _label, _filepath] = sess.run(fetches=features)
                  _image = np.asarray([_image])
                  _image = _image.reshape(-1, 331, 331, 3)
  
                  predictions = inference_session.run(output_layer, feed_dict={input_layer: _image})
                  predictions = np.squeeze(predictions)
  
                  # res = []
                  for i, pred in enumerate(predictions):
                      count += 1
                      overall_result = np.argmax(pred)
                      predict_result = label_map[overall_result].split(":")[-1]
  
                      if predict_result == 'unknown': continue
  
                      content['prob'] = str(np.max(pred))
                      content['label'] = predict_result
                      content['filepath'] = str(_filepath[i], encoding='utf-8')
                      result.append(content)
  
          except tf.errors.OutOfRangeError:
              t2 = time.time()
              print("{} images processed, average time: {}s".format(count, (t2-t1)/count))
      return result
  </code></pre>
  
  <p>I tried to use <code>with tf.device('/gpu:{}'.format(i))</code> in the loading model part or the inference session part or the session part, <code>nvidia-smi</code> shows that only GPU0 is use to 100%, while other GPUs doesn't seem to work even when the memory is load.</p>
  
  <p>What should I do to make all the GPUs run at the same time to boost predicting speed?</p>
  
  <p>My code is under <a href="https://github.com/tzattack/image_classification_algorithms" rel="noreferrer">https://github.com/tzattack/image_classification_algorithms</a> .</p>