title: Colab PRO: Resource exhausted: OOM when allocating tensor with shape and type float on /job:localhost/replica:0/task:0/device:GPU:0
body: <p>Unfortunately, I just paid for the monthly subscription on GoogleColab since I was having issues running the <code>fizyr/keras-retinanet</code> model. Before buying the subscription it was running on the following parameters: Batch 5, Step 5, Epochs 20. However, I believe the model is not being trained to its full potential. Increasing the Batch to 7 but keeping the other two parameters the same errors out and claims I'm using too much memory. The notebook has a GPU runtime, I checked multiple times. Unfortunately, google colab doesn't have an email for help nor a contact number. So if anyone has run into this issue and knows if there is something missing I need to add or it was just a waste of money, please let me know. Any help appreciated.</p>
  <p>Log snippet below:</p>
  <pre><code>2021-03-23 18:51:55.353329: I tensorflow/core/common_runtime/bfc_allocator.cc:1036] 13 Chunks of size 4194304 totalling 52.00MiB
  2021-03-23 18:51:55.353342: I tensorflow/core/common_runtime/bfc_allocator.cc:1036] 1 Chunks of size 4684544 totalling 4.47MiB
  2021-03-23 18:51:55.353355: I tensorflow/core/common_runtime/bfc_allocator.cc:1036] 1 Chunks of size 4980736 totalling 4.75MiB
  2021-03-23 18:51:55.353368: I tensorflow/core/common_runtime/bfc_allocator.cc:1036] 2 Chunks of size 6291456 totalling 12.00MiB
  2021-03-23 18:51:55.353381: I tensorflow/core/common_runtime/bfc_allocator.cc:1036] 3 Chunks of size 8388608 totalling 24.00MiB
  2021-03-23 18:51:55.353394: I tensorflow/core/common_runtime/bfc_allocator.cc:1036] 8 Chunks of size 9437184 totalling 72.00MiB
  2021-03-23 18:51:55.353407: I tensorflow/core/common_runtime/bfc_allocator.cc:1036] 1 Chunks of size 12402432 totalling 11.83MiB
  2021-03-23 18:51:55.353420: I tensorflow/core/common_runtime/bfc_allocator.cc:1036] 1 Chunks of size 13631488 totalling 13.00MiB
  2021-03-23 18:51:55.353433: I tensorflow/core/common_runtime/bfc_allocator.cc:1036] 3 Chunks of size 18874368 totalling 54.00MiB
  2021-03-23 18:51:55.353446: I tensorflow/core/common_runtime/bfc_allocator.cc:1036] 15 Chunks of size 33280000 totalling 476.07MiB
  2021-03-23 18:51:55.353458: I tensorflow/core/common_runtime/bfc_allocator.cc:1036] 1 Chunks of size 34467840 totalling 32.87MiB
  2021-03-23 18:51:55.353472: I tensorflow/core/common_runtime/bfc_allocator.cc:1036] 1 Chunks of size 49776128 totalling 47.47MiB
  2021-03-23 18:51:55.353483: I tensorflow/core/common_runtime/bfc_allocator.cc:1036] 1 Chunks of size 65372160 totalling 62.34MiB
  2021-03-23 18:51:55.353496: I tensorflow/core/common_runtime/bfc_allocator.cc:1036] 11 Chunks of size 66560000 totalling 698.24MiB
  2021-03-23 18:51:55.353508: I tensorflow/core/common_runtime/bfc_allocator.cc:1036] 3 Chunks of size 67742720 totalling 193.81MiB
  2021-03-23 18:51:55.353521: I tensorflow/core/common_runtime/bfc_allocator.cc:1036] 1 Chunks of size 80855552 totalling 77.11MiB
  2021-03-23 18:51:55.353534: I tensorflow/core/common_runtime/bfc_allocator.cc:1036] 1 Chunks of size 114268160 totalling 108.97MiB
  2021-03-23 18:51:55.353547: I tensorflow/core/common_runtime/bfc_allocator.cc:1036] 1 Chunks of size 131462912 totalling 125.37MiB
  2021-03-23 18:51:55.353560: I tensorflow/core/common_runtime/bfc_allocator.cc:1036] 2 Chunks of size 131937280 totalling 251.65MiB
  2021-03-23 18:51:55.353573: I tensorflow/core/common_runtime/bfc_allocator.cc:1036] 6 Chunks of size 133120000 totalling 761.72MiB
  2021-03-23 18:51:55.353587: I tensorflow/core/common_runtime/bfc_allocator.cc:1036] 1 Chunks of size 157906176 totalling 150.59MiB
  2021-03-23 18:51:55.353600: I tensorflow/core/common_runtime/bfc_allocator.cc:1036] 1 Chunks of size 217243648 totalling 207.18MiB
  2021-03-23 18:51:55.353614: I tensorflow/core/common_runtime/bfc_allocator.cc:1036] 2 Chunks of size 265728000 totalling 506.84MiB
  2021-03-23 18:51:55.353627: I tensorflow/core/common_runtime/bfc_allocator.cc:1036] 6 Chunks of size 266240000 totalling 1.49GiB
  2021-03-23 18:51:55.353640: I tensorflow/core/common_runtime/bfc_allocator.cc:1036] 1 Chunks of size 276045824 totalling 263.26MiB
  2021-03-23 18:51:55.353653: I tensorflow/core/common_runtime/bfc_allocator.cc:1040] Sum Total of in-use chunks: 5.80GiB
  2021-03-23 18:51:55.353665: I tensorflow/core/common_runtime/bfc_allocator.cc:1042] total_region_allocated_bytes_: 6282224896 memory_limit_: 6503147776 available bytes: 220922880 curr_region_allocation_bytes_: 8589934592
  2021-03-23 18:51:55.353683: I tensorflow/core/common_runtime/bfc_allocator.cc:1048] Stats: 
  Limit:                      6503147776
  InUse:                      6232547072
  MaxInUse:                   6233151488
  NumAllocs:                        1351
  MaxAllocSize:                975831040
  Reserved:                            0
  PeakReserved:                        0
  LargestFreeBlock:                    0
  
  2021-03-23 18:51:55.353744: W tensorflow/core/common_runtime/bfc_allocator.cc:441] ****************************************************************************************************
  2021-03-23 18:51:55.353802: W tensorflow/core/framework/op_kernel.cc:1763] OP_REQUIRES failed at fused_batch_norm_op.cc:1302 : Resource exhausted: OOM when allocating tensor with shape[5,512,100,130] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
  Traceback (most recent call last):
    File &quot;/usr/local/bin/retinanet-train&quot;, line 8, in &lt;module&gt;
      sys.exit(main())
    File &quot;/usr/local/lib/python3.7/dist-packages/keras_retinanet/bin/train.py&quot;, line 548, in main
      initial_epoch=args.initial_epoch
    File &quot;/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py&quot;, line 1861, in fit_generator
      initial_epoch=initial_epoch)
    File &quot;/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py&quot;, line 1100, in fit
      tmp_logs = self.train_function(iterator)
    File &quot;/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/def_function.py&quot;, line 828, in __call__
      result = self._call(*args, **kwds)
    File &quot;/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/def_function.py&quot;, line 888, in _call
      return self._stateless_fn(*args, **kwds)
    File &quot;/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/function.py&quot;, line 2943, in __call__
      filtered_flat_args, captured_inputs=graph_function.captured_inputs)  # pylint: disable=protected-access
    File &quot;/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/function.py&quot;, line 1919, in _call_flat
      ctx, args, cancellation_manager=cancellation_manager))
    File &quot;/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/function.py&quot;, line 560, in call
      ctx=ctx)
    File &quot;/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/execute.py&quot;, line 60, in quick_execute
      inputs, attrs, num_outputs)
  tensorflow.python.framework.errors_impl.ResourceExhaustedError:  OOM when allocating tensor with shape[5,100,130,512] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
       [[{{node retinanet/res3d_branch2c/Conv2D-0-0-TransposeNCHWToNHWC-LayoutOptimizer}}]]
  Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
   [Op:__inference_train_function_16419]
  
  Function call stack:
  train_function
  
  2021-03-23 18:51:55.488574: W tensorflow/core/kernels/data/generator_dataset_op.cc:107] Error occurred when finalizing GeneratorDataset iterator: Failed precondition: Python interpreter state is not initialized. The process may be terminated.
       [[{{node PyFunc}}]]
  </code></pre>