title: Can CPU code exist under "with tf.device(gpu_id) :" in multiple GPU case?
body: <p>Hi I am new to Tensorflow and I am assigned a task to change the "Demo.py" in a Github project "tf-faster-rcnn" to achieve multi-GPU inferencing. </p>
  
  <p>And this is generally what I plan to do (suppose the number of images I have is the same as the number of GPUs, and I will use a queue which is not illustrated here for simplicity reason):</p>
  
  <pre><code>for id, gpu in gpu_dict:
      with tf.device(gpu):
          im_detect(images[id])
  </code></pre>
  
  <p>The "im_detect" function is provided in the the source file (I can straight away call it) and it contains some non-GPU code (like conditions and data preparation)</p>
  
  <pre><code>def im_detect(sess, net, im):
    blobs, im_scales = _get_blobs(im)
    assert len(im_scales) == 1, "Only single-image batch implemented"
  
    im_blob = blobs['data']
    blobs['im_info'] = np.array([im_blob.shape[1], im_blob.shape[2], im_scales[0]], dtype=np.float32)
  
    _, scores, bbox_pred, rois = net.test_image(sess, blobs['data'], blobs['im_info'])
  
    boxes = rois[:, 1:5] / im_scales[0]
    scores = np.reshape(scores, [scores.shape[0], -1])
    bbox_pred = np.reshape(bbox_pred, [bbox_pred.shape[0], -1])
    if cfg.TEST.BBOX_REG:
      # Apply bounding-box regression deltas
      box_deltas = bbox_pred
      pred_boxes = bbox_transform_inv(boxes, box_deltas)
      pred_boxes = _clip_boxes(pred_boxes, im.shape)
    else:
      # Simply repeat the boxes, once for each class
      pred_boxes = np.tile(boxes, (1, scores.shape[1]))
  
    return scores, pred_boxes
  </code></pre>
  
  <p>Since I have never played with GPU before and I am a newbie to Tensorflow, I would like to ask is it okay to assign each GPU such a function call in Tensorflow? </p>
  
  <p>----------------below is updated------------------------</p>
  
  <p>I know there's a "alow_soft_placement" option in Tensorflow and that assigned those non-GPU code to CPU, but when there are multiple GPUs, how can one CPU handle those requests from all the GPUs? Should I create a CPU thread for each GPU?</p>