<p dir="auto"><strong>System information</strong></p>
<ul dir="auto">
<li>Have I written custom code (as opposed to using a stock example script provided in TensorFlow):</li>
</ul>
<ol dir="auto">
<li>Code in Golang</li>
<li>Used GO Tensorflow</li>
<li>MobileNet model</li>
</ol>
<ul dir="auto">
<li>OS Platform and Distribution (e.g., Linux Ubuntu 16.04):<br>
Raspberry Pi (1.4GHz, 1024 RAM, 4 Cores)</li>
<li>TensorFlow version (use command below):<br>
v1.12.0</li>
</ul>
<p dir="auto"><strong>Describe the current behavior</strong><br>
I have written a Face Classification code which uses MobileNet (depth 80) model. But When I ran the model after 8-10 min, tensorflow is throwing OOM allocation error.<br>
<strong>Describe the expected behavior</strong><br>
Because raspberry pi has limited memory. But as the model is very small it has to run perfectly.<br>
<strong>Code to reproduce the issue</strong></p>
<ol dir="auto">
<li>Run MobileNet model classification in Golang Env</li>
<li>Input tensor shape [5,224,224,3], tried other batch sizes as well 1 and 3 but no luck.<br>
<strong>Other info / logs</strong></li>
</ol>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="2019-02-20 10:52:26.403200: W tensorflow/core/framework/op_kernel.cc:1412] OP_REQUIRES failed at fused_batch_norm_op.cc:573 : Resource exhausted: OOM when allocating tensor with shape[5,112,112,64] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu
2019-02-20 10:52:26.478760: W tensorflow/core/framework/op_kernel.cc:1412] OP_REQUIRES failed at pad_op.cc:137 : Resource exhausted: OOM when allocating tensor with shape[5,113,113,64] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu
E0220 10:52:26.483588    3282 classifier.go:203] Failed to classify face: OOM when allocating tensor with shape[5,112,112,64] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu
	 [[{{node mobilenet_1.00_224/conv_pw_1_bn/FusedBatchNorm}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
"><pre class="notranslate"><code class="notranslate">2019-02-20 10:52:26.403200: W tensorflow/core/framework/op_kernel.cc:1412] OP_REQUIRES failed at fused_batch_norm_op.cc:573 : Resource exhausted: OOM when allocating tensor with shape[5,112,112,64] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu
2019-02-20 10:52:26.478760: W tensorflow/core/framework/op_kernel.cc:1412] OP_REQUIRES failed at pad_op.cc:137 : Resource exhausted: OOM when allocating tensor with shape[5,113,113,64] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu
E0220 10:52:26.483588    3282 classifier.go:203] Failed to classify face: OOM when allocating tensor with shape[5,112,112,64] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu
	 [[{{node mobilenet_1.00_224/conv_pw_1_bn/FusedBatchNorm}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc

</code></pre></div>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/poxvoculi/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/poxvoculi">@poxvoculi</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/penpornk/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/penpornk">@penpornk</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/nhasabni/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/nhasabni">@nhasabni</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/reedwm/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/reedwm">@reedwm</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/shashishekhar/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/shashishekhar">@shashishekhar</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/claynerobison/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/claynerobison">@claynerobison</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/andydavis1/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/andydavis1">@andydavis1</a><br>
Hi guys, I have checked some previous issues on <code class="notranslate">bad_allocation</code> and checked all the discussions and I could not reach to a conclusion how to resolve my <code class="notranslate">bad_allocation</code> issue. Below is my details for my issue with log. If I could get some insights, it will be very helpful.<br>
<strong>System information</strong></p>
<ul dir="auto">
<li>Have I written custom code (as opposed to using a stock example script provided in TensorFlow):<br>
Written custom code for face image classification. Model accepts 224x224x3 tensor. The model has 4,253,864 parameters and model depth is 88.</li>
<li>OS Platform and Distribution (e.g., Linux Ubuntu 16.04):<br>
Raspberry Pi, CPU: 1.4Ghz, 980 Mb RAM.</li>
<li>Programming Language<br>
Golang</li>
<li>Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy)<br>
No</li>
<li>TensorFlow installed from (source or binary):<br>
Build from source</li>
<li>TensorFlow version (use command below):<br>
v1.12.0</li>
<li>Python version:<br>
NO</li>
<li>Bazel version (if compiling from source):<br>
v0.15.2</li>
<li>GCC/Compiler version (if compiling from source):<br>
6.3</li>
<li>CUDA/cuDNN version:<br>
NO</li>
<li>GPU model and memory:<br>
NO</li>
</ul>
<p dir="auto"><strong>Describe the current behavior</strong><br>
I am using GoLang enviroment to load Tensorflow model and run. My Tensorflow model is MobileNet.<br>
It does face classification. I faced issue in memory allocation.<br>
<strong>Describe the expected behavior</strong><br>
Model loading is fine and runs perfectly for 15-20 min and after that, it throws <code class="notranslate">bad_allocation</code> error and exits.<br>
<strong>Code to reproduce the issue</strong></p>
<ol dir="auto">
<li>Generate libtensorflow.so and libtensorflow_framework.so artifact for raspberry pi.</li>
<li>Load tensorflow model into Golang Env.</li>
<li>While model is running and after 15-20 min this happens.</li>
</ol>
<p dir="auto"><strong>Other info / logs</strong><br>
2019-02-06 10:41:44.057443: W tensorflow/core/framework/op_kernel.cc:1412] OP_REQUIRES failed at conv_ops.cc:470 : Resource exhausted: OOM when allocating tensor with shape[1,112,112,64] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu<br>
E0206 10:41:44.128595    2719 classifier.go:148] Failed to classify face: OOM when allocating tensor with shape[1,112,112,64] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu<br>
[[{{node mobilenet_1.00_224/conv_pw_1/convolution}}]]<br>
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.<br>
2019-02-06 10:41:44.313010: W tensorflow/core/framework/op_kernel.cc:1412] OP_REQUIRES failed at conv_ops.cc:470 : Resource exhausted: OOM when allocating tensor with shape[1,112,112,64] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu<br>
E0206 10:41:44.314077    2719 classifier.go:148] Failed to classify face: OOM when allocating tensor with shape[1,112,112,64] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu<br>
[[{{node mobilenet_1.00_224/conv_pw_1/convolution}}]]<br>
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.<br>
2019-02-06 10:41:44.409438: W tensorflow/core/framework/op_kernel.cc:1412] OP_REQUIRES failed at conv_ops.cc:470 : Resource exhausted: OOM when allocating tensor with shape[1,112,112,64] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu<br>
E0206 10:41:44.411047    2719 classifier.go:148] Failed to classify face: OOM when allocating tensor with shape[1,112,112,64] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu<br>
[[{{node mobilenet_1.00_224/conv_pw_1/convolution}}]]<br>
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.<br>
2019-02-06 10:41:44.504644: W tensorflow/core/framework/op_kernel.cc:1412] OP_REQUIRES failed at conv_ops.cc:470 : Resource exhausted: OOM when allocating tensor with shape[1,112,112,64] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu<br>
E0206 10:41:44.505833    2719 classifier.go:148] Failed to classify face: OOM when allocating tensor with shape[1,112,112,64] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu<br>
[[{{node mobilenet_1.00_224/conv_pw_1/convolution}}]]<br>
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.<br>
2019-02-06 10:41:44.601160: W tensorflow/core/framework/op_kernel.cc:1412] OP_REQUIRES failed at conv_ops.cc:470 : Resource exhausted: OOM when allocating tensor with shape[1,112,112,64] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu<br>
E0206 10:41:44.602079    2719 classifier.go:148] Failed to classify face: OOM when allocating tensor with shape[1,112,112,64] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu<br>
[[{{node mobilenet_1.00_224/conv_pw_1/convolution}}]]<br>
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.<br>
I0206 10:41:48.258835    2719 camera_input.go:69] Captured Image, Len Queue 0<br>
I0206 10:41:48.898151    2719 runner.go:85] Detection time 638.594898ms<br>
2019-02-06 10:41:48.997640: W tensorflow/core/framework/op_kernel.cc:1412] OP_REQUIRES failed at conv_ops.cc:470 : Resource exhausted: OOM when allocating tensor with shape[1,112,112,64] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu<br>
E0206 10:41:48.999425    2719 classifier.go:148] Failed to classify face: OOM when allocating tensor with shape[1,112,112,64] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu<br>
[[{{node mobilenet_1.00_224/conv_pw_1/convolution}}]]<br>
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.<br>
2019-02-06 10:41:49.100121: W tensorflow/core/framework/op_kernel.cc:1412] OP_REQUIRES failed at conv_ops.cc:470 : Resource exhausted: OOM when allocating tensor with shape[1,112,112,64] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu<br>
E0206 10:41:49.101226    2719 classifier.go:148] Failed to classify face: OOM when allocating tensor with shape[1,112,112,64] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu<br>
[[{{node mobilenet_1.00_224/conv_pw_1/convolution}}]]<br>
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.<br>
2019-02-06 10:41:49.197890: W tensorflow/core/framework/op_kernel.cc:1412] OP_REQUIRES failed at conv_ops.cc:470 : Resource exhausted: OOM when allocating tensor with shape[1,112,112,64] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu<br>
E0206 10:41:49.199211    2719 classifier.go:148] Failed to classify face: OOM when allocating tensor with shape[1,112,112,64] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu<br>
[[{{node mobilenet_1.00_224/conv_pw_1/convolution}}]]<br>
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.<br>
terminate called after throwing an instance of 'std::bad_alloc'<br>
what():  std::bad_alloc<br>
Aborted</p>