title: Tenserflow hangs when running inference with GPU enabled
body: <p>I am new to AI and TensorFlow and I am trying to use the TensorFlow object detection API on windows. <br>
  My current goal is to do real time human detection in a video stream. <br>
  For this I modified a python example from the TensorFlow Model Garden (<a href="https://github.com/tensorflow/models" rel="nofollow noreferrer">https://github.com/tensorflow/models</a>).<br>
  At the moment it detects all objects (not just humans) and shows the bounding boxes using opencv.</p>
  
  <p>It works fine when I have the GPU disabled (os.environ["CUDA_VISIBLE_DEVICES"] = "-1")<br>
  But when I enable the GPU and start the script it will hang on the first frame. </p>
  
  <p>Output:</p>
  
  <pre><code>2020-04-22 16:00:53.597492: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
  2020-04-22 16:00:56.942141: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library nvcuda.dll
  2020-04-22 16:00:56.976635: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
  pciBusID: 0000:01:00.0 name: GeForce GTX 960M computeCapability: 5.0
  coreClock: 1.176GHz coreCount: 5 deviceMemorySize: 2.00GiB deviceMemoryBandwidth: 74.65GiB/s
  2020-04-22 16:00:56.989129: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
  2020-04-22 16:00:57.000622: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll
  2020-04-22 16:00:57.012247: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cufft64_10.dll
  2020-04-22 16:00:57.020575: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library curand64_10.dll
  2020-04-22 16:00:57.031536: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusolver64_10.dll
  2020-04-22 16:00:57.042564: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusparse64_10.dll
  2020-04-22 16:00:57.066289: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
  2020-04-22 16:00:57.075760: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
  2020-04-22 16:00:59.239211: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
  2020-04-22 16:00:59.256577: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1f3f73cd670 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
  2020-04-22 16:00:59.264241: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
  2020-04-22 16:00:59.272280: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
  pciBusID: 0000:01:00.0 name: GeForce GTX 960M computeCapability: 5.0
  coreClock: 1.176GHz coreCount: 5 deviceMemorySize: 2.00GiB deviceMemoryBandwidth: 74.65GiB/s
  2020-04-22 16:00:59.281409: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
  2020-04-22 16:00:59.288204: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll
  2020-04-22 16:00:59.293112: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cufft64_10.dll
  2020-04-22 16:00:59.298222: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library curand64_10.dll
  2020-04-22 16:00:59.305446: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusolver64_10.dll
  2020-04-22 16:00:59.310590: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusparse64_10.dll
  2020-04-22 16:00:59.316250: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
  2020-04-22 16:00:59.324588: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
  2020-04-22 16:01:00.831569: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
  2020-04-22 16:01:00.839147: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108]      0
  2020-04-22 16:01:00.842279: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0:   N
  2020-04-22 16:01:00.846140: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1024 MB memory) -&gt; physical GPU (device: 0, name: GeForce GTX 960M, pci bus id: 0000:01:00.0, compute capability: 5.0)
  2020-04-22 16:01:00.865546: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1f39174cba0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
  2020-04-22 16:01:00.873656: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): GeForce GTX 960M, Compute Capability 5.0
  [&lt;tf.Tensor 'image_tensor:0' shape=(None, None, None, 3) dtype=uint8&gt;]
  2020-04-22 16:01:10.876733: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
  2020-04-22 16:01:11.814909: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: Invoking GPU asm compilation is supported on Cuda non-Windows platforms only
  Relying on driver to perform ptx compilation.
  Modify $PATH to customize ptxas location.
  This message will be only logged once.
  2020-04-22 16:01:11.852909: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll
  2020-04-22 16:01:12.149312: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.04GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
  2020-04-22 16:01:12.179484: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.04GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
  2020-04-22 16:01:12.209036: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
  2020-04-22 16:01:12.237205: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.05GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
  2020-04-22 16:01:12.266147: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.09GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
  2020-04-22 16:01:12.295182: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.08GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
  2020-04-22 16:01:12.325645: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.15GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
  2020-04-22 16:01:12.357550: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.15GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
  2020-04-22 16:01:12.405332: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.14GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
  2020-04-22 16:01:12.436336: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.27GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
  </code></pre>
  
  <p>This is the code I am using:</p>
  
  <pre><code>#!/usr/bin/env python
  # coding: utf-8
  
  import os
  import pathlib
  
  if "models" in pathlib.Path.cwd().parts:
    while "models" in pathlib.Path.cwd().parts:
      os.chdir('..')
  
  import numpy as np
  import os
  import six.moves.urllib as urllib
  import sys
  import tarfile
  import tensorflow as tf
  import zipfile
  
  from collections import defaultdict
  from io import StringIO
  from PIL import Image
  from IPython.display import display
  
  import cv2 
  cap = cv2.VideoCapture(1)
  
  from object_detection.utils import ops as utils_ops
  from object_detection.utils import label_map_util
  from object_detection.utils import visualization_utils as vis_util
  
  # patch tf1 into `utils.ops`
  utils_ops.tf = tf.compat.v1
  
  # Patch the location of gfile
  tf.gfile = tf.io.gfile
  
  # os.environ["CUDA_VISIBLE_DEVICES"] = "-1"
  
  def load_model(model_name):
    base_url = 'http://download.tensorflow.org/models/object_detection/'
    model_file = model_name + '.tar.gz'
    model_dir = tf.keras.utils.get_file(
      fname=model_name, 
      origin=base_url + model_file,
      untar=True)
  
    model_dir = pathlib.Path(model_dir)/"saved_model"
  
    model = tf.saved_model.load(str(model_dir))
    model = model.signatures['serving_default']
  
    return model
  
  # List of the strings that is used to add correct label for each box.
  PATH_TO_LABELS = 'models/research/object_detection/data/mscoco_label_map.pbtxt'
  category_index = label_map_util.create_category_index_from_labelmap(PATH_TO_LABELS, use_display_name=True)
  
  model_name = 'ssd_mobilenet_v1_coco_2017_11_17'
  # model_name= 'faster_rcnn_inception_v2_coco_2017_11_08';
  detection_model = load_model(model_name)
  
  print(detection_model.inputs)
  
  detection_model.output_dtypes
  detection_model.output_shapes
  
  def run_inference_for_single_image(model, image):
      image = np.asarray(image)
      # The input needs to be a tensor, convert it using `tf.convert_to_tensor`.
      input_tensor = tf.convert_to_tensor(image)
      # The model expects a batch of images, so add an axis with `tf.newaxis`.
      input_tensor = input_tensor[tf.newaxis,...]
  
      # Run inference (it hangs here)
      output_dict = model(input_tensor)
  
      # All outputs are batches tensors.
      # Convert to numpy arrays, and take index [0] to remove the batch dimension.
      # We're only interested in the first num_detections.
      num_detections = int(output_dict.pop('num_detections'))
      output_dict = {key:value[0, :num_detections].numpy() 
                   for key,value in output_dict.items()}
      output_dict['num_detections'] = num_detections
  
      # detection_classes should be ints.
      output_dict['detection_classes'] = output_dict['detection_classes'].astype(np.int64)
  
      # Handle models with masks:
      if 'detection_masks' in output_dict:
          # Reframe the the bbox mask to the image size.
          detection_masks_reframed = utils_ops.reframe_box_masks_to_image_masks(output_dict['detection_masks'], output_dict['detection_boxes'],image.shape[0], image.shape[1])      
          detection_masks_reframed = tf.cast(detection_masks_reframed &gt; 0.5,tf.uint8)
          output_dict['detection_masks_reframed'] = detection_masks_reframed.numpy()
  
      return output_dict
  
  def show_inference(model):
      # the array based representation of the image will be used later in order to prepare the
      # result image with boxes and labels on it.
      ret, image_np = cap.read()
  
      #percent by which the image is resized
      #scale_percent = 30
  
      #calculate the 50 percent of original dimensions
      #width = int(image_np.shape[1] * scale_percent / 100)
      #height = int(image_np.shape[0] * scale_percent / 100)
  
      # dsize
      #dsize = (width, height)
  
      # resize image
      #image_np = cv2.resize(image_np, dsize)
  
      # Actual detection.
      output_dict = run_inference_for_single_image(model, image_np)
  
      # Visualization of the results of a detection.
      vis_util.visualize_boxes_and_labels_on_image_array(
        image_np,
        output_dict['detection_boxes'],
        output_dict['detection_classes'],
        output_dict['detection_scores'],
        category_index,
        instance_masks=output_dict.get('detection_masks_reframed', None),
        use_normalized_coordinates=True,
        line_thickness=8)
  
      cv2.imshow('object detection', cv2.resize(image_np, (800,600)))
  
  while True:
    show_inference(detection_model)
    if cv2.waitKey(25) &amp; 0xFF == ord('q'):
      cv2.destroyAllWindows()
      break
  </code></pre>
  
  <p>I have installed the following versions:<br>
  Python: 3.7 64 bit <br>
  Tensorflow: 2.2.0-rc3 <br>
  Cuda: 10.1 <br>
  cudnn 7.6.5.32<br></p>
  
  <p>I tried this on 2 different machines:<br>
  Machine 1:<br>
   - CPU: i7-6700HQ<br>
   - RAM: 16 GB<br>
   - GPU: NVIDIA GeForce GTX 960M<br>
  <br>
  Machine 2:<br>
   - CPU: i5-6400 <br>
   - RAM: 16 GB<br>
   - GPU: NVIDIA GeForce GTX 960<br>
  <br>
  I am not sure how to debug this. I tried the same code on two different machine and the result was almost the same. <br>
  The only difference was the time it took for it to hang. Machine 1 would hang immediately and machine 2 took roughly 30 seconds. <br>
  Machine 2 was able to process the video and detect object up until the hang.<br></p>
  
  <p>I looked into the 'Allocator (GPU_0_bfc) ran out of memory' warnings.  <br>
  I tried some options that limited the available GPU memory size, but this did not help.  <br></p>
  
  <p>There where also multiple posts that sugested reducing the batch size.  <br>
  My interpretation was that this was only helpful when training your own model.  <br>
  And because I am using pre-trained models, this was not applicable.   <br></p>
  
  <p>I also tried to use different models: ssd_mobilenet_v1_coco_2017_11_17 and faster_rcnn_inception_v2_coco_2017_11_08. Both models have the same result. </p>
  
  <p>The last thing I tried was to reduce the image size before processing it. This also did not help.<br></p>
  
  <p>Any help would be much appreciated<br></p>
  
  <p><strong>Update</strong><br>
  I also tried it on a RTX2070 super GPU. There are no warnings about memory allocation. This is also not able to complete a single inference. 
  Just for completeness, this is the console output [The text 'inference start' is printed before running the inference. If the inference would complete, it would print 'inference end']:</p>
  
  <pre><code>2020-04-24 11:30:16.579805: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
  2020-04-24 11:30:18.916146: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library nvcuda.dll
  2020-04-24 11:30:18.941805: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties:
  pciBusID: 0000:01:00.0 name: GeForce RTX 2070 SUPER computeCapability: 7.5
  coreClock: 1.785GHz coreCount: 40 deviceMemorySize: 8.00GiB deviceMemoryBandwidth: 417.29GiB/s
  2020-04-24 11:30:18.946134: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
  2020-04-24 11:30:18.951172: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll
  2020-04-24 11:30:18.954809: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cufft64_10.dll
  2020-04-24 11:30:18.957258: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library curand64_10.dll
  2020-04-24 11:30:18.961662: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusolver64_10.dll
  2020-04-24 11:30:18.965553: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusparse64_10.dll
  2020-04-24 11:30:18.978671: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
  2020-04-24 11:30:18.980998: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
  2020-04-24 11:30:18.982226: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
  2020-04-24 11:30:18.984167: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties:
  pciBusID: 0000:01:00.0 name: GeForce RTX 2070 SUPER computeCapability: 7.5
  coreClock: 1.785GHz coreCount: 40 deviceMemorySize: 8.00GiB deviceMemoryBandwidth: 417.29GiB/s
  2020-04-24 11:30:18.987291: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
  2020-04-24 11:30:18.988809: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll
  2020-04-24 11:30:18.990303: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cufft64_10.dll
  2020-04-24 11:30:18.991792: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library curand64_10.dll
  2020-04-24 11:30:18.993320: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusolver64_10.dll
  2020-04-24 11:30:18.996960: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusparse64_10.dll
  2020-04-24 11:30:18.998497: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
  2020-04-24 11:30:19.000191: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
  2020-04-24 11:30:19.430864: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Device interconnect StreamExecutor with strength 1 edge matrix:
  2020-04-24 11:30:19.433076: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102]      0
  2020-04-24 11:30:19.434566: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 0:   N
  2020-04-24 11:30:19.436400: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6281 MB memory) -&gt; physical GPU (device: 0, name: GeForce RTX 2070 SUPER, pci bus id: 0000:01:00.0, compute capability: 7.5)
  [&lt;tf.Tensor 'image_tensor:0' shape=(None, None, None, 3) dtype=uint8&gt;]
  inference start
  2020-04-24 11:30:24.728554: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
  2020-04-24 11:30:25.608426: W tensorflow/stream_executor/gpu/redzone_allocator.cc:312] Internal: Invoking GPU asm compilation is supported on Cuda non-Windows platforms only
  Relying on driver to perform ptx compilation. This message will be only logged once.
  2020-04-24 11:30:25.625904: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll
  </code></pre>
  
  <p><strong>Update 2</strong><br>
  When Eager mode is disabled everything runs fine (even on GPU), but then I am not able to retreive the found objects.<br>
  The next thing I tried was running it with sessions (like TensorFlow 1 I think). Here the function session.run() blocks indefinitely on GPU. And again on CPU it works fine.</p>