{"action":"opened","issue":{"url":"https://api.github.com/repos/ROCmSoftwarePlatform/tensorflow-upstream/issues/250","repository_url":"https://api.github.com/repos/ROCmSoftwarePlatform/tensorflow-upstream","labels_url":"https://api.github.com/repos/ROCmSoftwarePlatform/tensorflow-upstream/issues/250/labels{/name}","comments_url":"https://api.github.com/repos/ROCmSoftwarePlatform/tensorflow-upstream/issues/250/comments","events_url":"https://api.github.com/repos/ROCmSoftwarePlatform/tensorflow-upstream/issues/250/events","html_url":"https://github.com/ROCmSoftwarePlatform/tensorflow-upstream/issues/250","id":383249957,"node_id":"MDU6SXNzdWUzODMyNDk5NTc=","number":250,"title":"HSA_STATUS_ERROR_OUT_OF_RESOURCES with num_gpus=4 in tensorflow/benchmarks","user":{"login":"jamilbk","id":167144,"node_id":"MDQ6VXNlcjE2NzE0NA==","avatar_url":"https://avatars0.githubusercontent.com/u/167144?v=4","gravatar_id":"","url":"https://api.github.com/users/jamilbk","html_url":"https://github.com/jamilbk","followers_url":"https://api.github.com/users/jamilbk/followers","following_url":"https://api.github.com/users/jamilbk/following{/other_user}","gists_url":"https://api.github.com/users/jamilbk/gists{/gist_id}","starred_url":"https://api.github.com/users/jamilbk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jamilbk/subscriptions","organizations_url":"https://api.github.com/users/jamilbk/orgs","repos_url":"https://api.github.com/users/jamilbk/repos","events_url":"https://api.github.com/users/jamilbk/events{/privacy}","received_events_url":"https://api.github.com/users/jamilbk/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-11-21T18:44:41Z","updated_at":"2018-11-21T18:44:41Z","closed_at":null,"author_association":"NONE","body":"<em>Please make sure that this is a bug. As per our [GitHub Policy](https://github.com/tensorflow/tensorflow/blob/master/ISSUES.md), we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:bug_template</em>\r\n\r\n**System information**\r\n- Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No\r\n- OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 18.04\r\n- Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:\r\n- TensorFlow installed from (source or binary): Binary (pip package)\r\n- TensorFlow version (use command below): 1.12\r\n- Python version: 3.6.7\r\n- Bazel version (if compiling from source):\r\n- GCC/Compiler version (if compiling from source):\r\n- CUDA/cuDNN version:\r\n- GPU model and memory: 4x Radeon Vega FE on Gigabyte X399 and TR1950x\r\n\r\nRocm lib versions:\r\n\r\n```\r\nrocm-clang-ocl/Ubuntu 16.04,now 0.3.0-7997136 amd64 [installed,automatic]\r\nrocm-dev/Ubuntu 16.04,now 1.9.307 amd64 [installed,automatic]\r\nrocm-device-libs/Ubuntu 16.04,now 0.0.1 amd64 [installed]\r\nrocm-dkms/Ubuntu 16.04,now 1.9.307 amd64 [installed]\r\nrocm-libs/Ubuntu 16.04,now 1.9.307 amd64 [installed]\r\nrocm-opencl/Ubuntu 16.04,now 1.2.0-2018111340 amd64 [installed,automatic]\r\nrocm-opencl-dev/Ubuntu 16.04,now 1.2.0-2018111340 amd64 [installed,automatic]\r\nrocm-profiler/Ubuntu 16.04,now 5.4.6878 amd64 [installed]\r\nrocm-smi/Ubuntu 16.04,now 1.0.0-72-gec1da05 amd64 [installed,automatic]\r\nrocm-utils/Ubuntu 16.04,now 1.9.307 amd64 [installed,automatic]\r\nrocminfo/Ubuntu 16.04,now 1.0.0 amd64 [installed,automatic]\r\n```\r\n\r\n**Describe the current behavior**\r\nThe tensorflow benchmark suite crashes with `HSA_STATUS_ERROR_OUT_OF_RESOURCES` when more than 1 GPU is used. This does not occur with one GPU. This is a new issue with tensorflow-rocm 1.12 and occurs regardless of the `batch_size` used.\r\n\r\n**Describe the expected behavior**\r\nI expect the benchmarks to complete successfully and, in an ideal world, achieve a speedup that scales (mostly) linearly with the number of GPUs used.\r\n\r\n**Code to reproduce the issue**\r\n```\r\n$ git clone https://github.com/tensorflow/benchmarks\r\n$ cd benchmarks/scripts/tf_cnn_benchmarks\r\n$ gco cnn_tf_v1.12_compatible\r\n$ python tf_cnn_benchmarks.py --num_gpus=4 --batch_size=1 --model=resnet50\r\n```\r\n\r\n\r\n**Other info / logs**\r\n\r\nHere is the full output:\r\n\r\n[output.txt](https://github.com/ROCmSoftwarePlatform/tensorflow-upstream/files/2605408/output.txt)\r\n\r\n"}}
{"actor":{"display_login":"jamilbk"}}