{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/34188","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/34188/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/34188/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/34188/events","html_url":"https://github.com/tensorflow/tensorflow/issues/34188","id":521387289,"node_id":"MDU6SXNzdWU1MjEzODcyODk=","number":34188,"title":"TF2.0 GPU - Segmentation fault - Object Detection","user":{"login":"Icaro-Lima","id":27537731,"node_id":"MDQ6VXNlcjI3NTM3NzMx","avatar_url":"https://avatars0.githubusercontent.com/u/27537731?v=4","gravatar_id":"","url":"https://api.github.com/users/Icaro-Lima","html_url":"https://github.com/Icaro-Lima","followers_url":"https://api.github.com/users/Icaro-Lima/followers","following_url":"https://api.github.com/users/Icaro-Lima/following{/other_user}","gists_url":"https://api.github.com/users/Icaro-Lima/gists{/gist_id}","starred_url":"https://api.github.com/users/Icaro-Lima/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Icaro-Lima/subscriptions","organizations_url":"https://api.github.com/users/Icaro-Lima/orgs","repos_url":"https://api.github.com/users/Icaro-Lima/repos","events_url":"https://api.github.com/users/Icaro-Lima/events{/privacy}","received_events_url":"https://api.github.com/users/Icaro-Lima/received_events","type":"User","site_admin":false},"labels":[{"id":1160540362,"node_id":"MDU6TGFiZWwxMTYwNTQwMzYy","url":"https://api.github.com/repos/tensorflow/tensorflow/labels/TF%202.0","name":"TF 2.0","color":"e79b38","default":false},{"id":1105108936,"node_id":"MDU6TGFiZWwxMTA1MTA4OTM2","url":"https://api.github.com/repos/tensorflow/tensorflow/labels/comp:model","name":"comp:model","color":"0052cc","default":false}],"state":"open","locked":false,"assignee":{"login":"gadagashwini","id":48476109,"node_id":"MDQ6VXNlcjQ4NDc2MTA5","avatar_url":"https://avatars3.githubusercontent.com/u/48476109?v=4","gravatar_id":"","url":"https://api.github.com/users/gadagashwini","html_url":"https://github.com/gadagashwini","followers_url":"https://api.github.com/users/gadagashwini/followers","following_url":"https://api.github.com/users/gadagashwini/following{/other_user}","gists_url":"https://api.github.com/users/gadagashwini/gists{/gist_id}","starred_url":"https://api.github.com/users/gadagashwini/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gadagashwini/subscriptions","organizations_url":"https://api.github.com/users/gadagashwini/orgs","repos_url":"https://api.github.com/users/gadagashwini/repos","events_url":"https://api.github.com/users/gadagashwini/events{/privacy}","received_events_url":"https://api.github.com/users/gadagashwini/received_events","type":"User","site_admin":false},"assignees":[{"login":"gadagashwini","id":48476109,"node_id":"MDQ6VXNlcjQ4NDc2MTA5","avatar_url":"https://avatars3.githubusercontent.com/u/48476109?v=4","gravatar_id":"","url":"https://api.github.com/users/gadagashwini","html_url":"https://github.com/gadagashwini","followers_url":"https://api.github.com/users/gadagashwini/followers","following_url":"https://api.github.com/users/gadagashwini/following{/other_user}","gists_url":"https://api.github.com/users/gadagashwini/gists{/gist_id}","starred_url":"https://api.github.com/users/gadagashwini/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gadagashwini/subscriptions","organizations_url":"https://api.github.com/users/gadagashwini/orgs","repos_url":"https://api.github.com/users/gadagashwini/repos","events_url":"https://api.github.com/users/gadagashwini/events{/privacy}","received_events_url":"https://api.github.com/users/gadagashwini/received_events","type":"User","site_admin":false}],"milestone":null,"comments":0,"created_at":"2019-11-12T07:57:53Z","updated_at":"2019-11-12T11:52:21Z","closed_at":null,"author_association":"NONE","body":"**System information**\r\n- Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No\r\n- OS Platform and Distribution (e.g., Linux Ubuntu 16.04): 18.04\r\n- TensorFlow installed from (source or binary): binary\r\n- TensorFlow version (use command below): v2.0.0-rc2-26-g64c3d38 2.0.0\r\n- Python version: Python 3.7.5\r\n- CUDA/cuDNN version: 10.0.130/7.6.2\r\n- GPU model and memory: GeForce 940MX / 2004MiB\r\n\r\n**Describe the current behavior**\r\nWhen I try to run [this](https://github.com/tensorflow/models/blob/master/research/object_detection/object_detection_tutorial.ipynb) example, a segmentation fault happens during inference. I just formatted my Ubuntu 18.04 and did a clean install.\r\n\r\n**Describe the expected behavior**\r\nThe images marked with the boxes.\r\n\r\n**Code to reproduce the issue**\r\n```\r\n...\r\n# Run inference\r\n  output_dict = model(input_tensor)\r\n...\r\n```\r\n\r\n**Other info / logs**\r\n```\r\n2019-11-12 04:53:51.925933: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1\r\n2019-11-12 04:53:51.935688: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\r\n2019-11-12 04:53:51.935937: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: \r\nname: GeForce 940MX major: 5 minor: 0 memoryClockRate(GHz): 1.189\r\npciBusID: 0000:01:00.0\r\n2019-11-12 04:53:51.936117: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0\r\n2019-11-12 04:53:51.937181: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0\r\n2019-11-12 04:53:51.938180: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0\r\n2019-11-12 04:53:51.938498: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0\r\n2019-11-12 04:53:51.939671: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0\r\n2019-11-12 04:53:51.940624: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0\r\n2019-11-12 04:53:51.943488: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7\r\n2019-11-12 04:53:51.943615: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\r\n2019-11-12 04:53:51.944225: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\r\n2019-11-12 04:53:51.944738: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0\r\n2019-11-12 04:53:53.716082: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA\r\n2019-11-12 04:53:53.741945: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2904000000 Hz\r\n2019-11-12 04:53:53.742225: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55fe34207b30 executing computations on platform Host. Devices:\r\n2019-11-12 04:53:53.742249: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Host, Default Version\r\n2019-11-12 04:53:53.792897: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\r\n2019-11-12 04:53:53.793392: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55fe33fa3880 executing computations on platform CUDA. Devices:\r\n2019-11-12 04:53:53.793424: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): GeForce 940MX, Compute Capability 5.0\r\n2019-11-12 04:53:53.793636: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\r\n2019-11-12 04:53:53.794038: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: \r\nname: GeForce 940MX major: 5 minor: 0 memoryClockRate(GHz): 1.189\r\npciBusID: 0000:01:00.0\r\n2019-11-12 04:53:53.794108: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0\r\n2019-11-12 04:53:53.794141: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0\r\n2019-11-12 04:53:53.794163: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0\r\n2019-11-12 04:53:53.794182: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0\r\n2019-11-12 04:53:53.794203: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0\r\n2019-11-12 04:53:53.794224: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0\r\n2019-11-12 04:53:53.794245: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7\r\n2019-11-12 04:53:53.794346: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\r\n2019-11-12 04:53:53.794887: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\r\n2019-11-12 04:53:53.795348: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0\r\n2019-11-12 04:53:53.795437: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0\r\n2019-11-12 04:53:53.796297: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:\r\n2019-11-12 04:53:53.796327: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165]      0 \r\n2019-11-12 04:53:53.796337: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0:   N \r\n2019-11-12 04:53:53.796644: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\r\n2019-11-12 04:53:53.796975: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\r\n2019-11-12 04:53:53.797255: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1306 MB memory) -> physical GPU (device: 0, name: GeForce 940MX, pci bus id: 0000:01:00.0, compute capability: 5.0)\r\n2019-11-12 04:54:11.319157: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7\r\n2019-11-12 04:54:12.995202: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.\r\n2019-11-12 04:54:13.169311: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.09GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.\r\n2019-11-12 04:54:13.328776: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.15GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.\r\n2019-11-12 04:54:13.366495: W tensorflow/core/common_runtime/bfc_allocator.cc:305] Garbage collection: deallocate free memory regions (i.e., allocations) so that we can re-allocate a larger region to avoid OOM due to memory fragmentation. If you see this message frequently, you are running near the threshold of the available device memory and re-allocation may incur great performance overhead. You may try smaller batch sizes to observe the performance impact. Set TF_ENABLE_GPU_GARBAGE_COLLECTION=false if you'd like to disable this feature.\r\n2019-11-12 04:54:13.786988: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.27GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.\r\n2019-11-12 04:54:13.848769: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.22GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.\r\nSegmentation fault (core dumped)\r\n```"},"comment":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/comments/552861866","html_url":"https://github.com/tensorflow/tensorflow/issues/34188#issuecomment-552861866","issue_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/34188","id":552861866,"node_id":"MDEyOklzc3VlQ29tbWVudDU1Mjg2MTg2Ng==","user":{"login":"gadagashwini","id":48476109,"node_id":"MDQ6VXNlcjQ4NDc2MTA5","avatar_url":"https://avatars3.githubusercontent.com/u/48476109?v=4","gravatar_id":"","url":"https://api.github.com/users/gadagashwini","html_url":"https://github.com/gadagashwini","followers_url":"https://api.github.com/users/gadagashwini/followers","following_url":"https://api.github.com/users/gadagashwini/following{/other_user}","gists_url":"https://api.github.com/users/gadagashwini/gists{/gist_id}","starred_url":"https://api.github.com/users/gadagashwini/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gadagashwini/subscriptions","organizations_url":"https://api.github.com/users/gadagashwini/orgs","repos_url":"https://api.github.com/users/gadagashwini/repos","events_url":"https://api.github.com/users/gadagashwini/events{/privacy}","received_events_url":"https://api.github.com/users/gadagashwini/received_events","type":"User","site_admin":false},"created_at":"2019-11-12T11:52:21Z","updated_at":"2019-11-12T11:52:21Z","author_association":"NONE","body":"@Icaro-Lima, Its working fine on Colab. Can you check once and let us know. Thanks!"}}
{"actor":{"display_login":"gadagashwini"}}