{"action":"created","issue":{"url":"https://api.github.com/repos/lukeiwanski/tensorflow/issues/252","repository_url":"https://api.github.com/repos/lukeiwanski/tensorflow","labels_url":"https://api.github.com/repos/lukeiwanski/tensorflow/issues/252/labels{/name}","comments_url":"https://api.github.com/repos/lukeiwanski/tensorflow/issues/252/comments","events_url":"https://api.github.com/repos/lukeiwanski/tensorflow/issues/252/events","html_url":"https://github.com/lukeiwanski/tensorflow/issues/252","id":343324171,"node_id":"MDU6SXNzdWUzNDMzMjQxNzE=","number":252,"title":"TensorFlow does not see all available GPUs in my system","user":{"login":"lu4","id":1069295,"node_id":"MDQ6VXNlcjEwNjkyOTU=","avatar_url":"https://avatars3.githubusercontent.com/u/1069295?v=4","gravatar_id":"","url":"https://api.github.com/users/lu4","html_url":"https://github.com/lu4","followers_url":"https://api.github.com/users/lu4/followers","following_url":"https://api.github.com/users/lu4/following{/other_user}","gists_url":"https://api.github.com/users/lu4/gists{/gist_id}","starred_url":"https://api.github.com/users/lu4/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lu4/subscriptions","organizations_url":"https://api.github.com/users/lu4/orgs","repos_url":"https://api.github.com/users/lu4/repos","events_url":"https://api.github.com/users/lu4/events{/privacy}","received_events_url":"https://api.github.com/users/lu4/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-07-21T12:45:51Z","updated_at":"2018-07-22T21:02:47Z","closed_at":null,"author_association":"NONE","body":"### System information\r\n- **Have I written custom code (as opposed to using a stock example script provided in TensorFlow)**: N/A\r\n- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Ubuntu 16.0.4 LTS x86_64\r\n- **TensorFlow installed from (source or binary)**: source\r\n- **TensorFlow version (use command below)**: ComputeCpp-v0.6.0-4212-gb29ac8a  1.8.0-rc1\r\n- **Python version**: 3.5.2\r\n- **Bazel version (if compiling from source)**: 0.15.0, build timestamp - 1530015019\r\n- **GCC/Compiler version (if compiling from source)**: 5.4.0 20160609\r\n- **CUDA/cuDNN version**: N/A\r\n- **GPU model and memory**: Sapphire  Radeon RX470, 8Gbytes\r\n- **Exact command to reproduce**: see below\r\n\r\n```\r\nfrom tensorflow.python.client import device_lib\r\ndevice_lib.list_local_devices()\r\n```\r\n\r\nHere's info from environment capture script:\r\n```\r\n== cat /etc/issue ===============================================\r\nLinux custom 4.16.0-rc6-smos+ #1 SMP Wed Mar 21 13:23:56 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux\r\nVERSION=\"16.04.3 LTS (Xenial Xerus)\"\r\nVERSION_ID=\"16.04\"\r\nVERSION_CODENAME=xenial\r\n\r\n== are we in docker =============================================\r\nNo\r\n\r\n== compiler =====================================================\r\nc++ (Ubuntu 5.4.0-6ubuntu1~16.04.10) 5.4.0 20160609\r\nCopyright (C) 2015 Free Software Foundation, Inc.\r\nThis is free software; see the source for copying conditions.  There is NO\r\nwarranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.\r\n\r\n\r\n== uname -a =====================================================\r\nLinux custom 4.16.0-rc6-smos+ #1 SMP Wed Mar 21 13:23:56 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux\r\n\r\n== check pips ===================================================\r\nnumpy               1.14.5\r\nprotobuf            3.6.0\r\ntensorflow          1.8.0rc1\r\n\r\n== check for virtualenv =========================================\r\nFalse\r\n\r\n== tensorflow import ============================================\r\nTraceback (most recent call last):\r\n  File \"<string>\", line 1, in <module>\r\nImportError: No module named tensorflow\r\n\r\n== env ==========================================================\r\nLD_LIBRARY_PATH /usr/local/lib:/usr/local/computecpp/lib:/usr/local/lib:/usr/local/computecpp/lib:\r\nDYLD_LIBRARY_PATH is unset\r\n\r\n== nvidia-smi ===================================================\r\nNVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.\r\n\r\n\r\n== cuda libs  ===================================================\r\n/usr/local/lib/libcudart.so.9.0.103\r\n\r\n== cat /etc/issue ===============================================\r\nLinux custom 4.16.0-rc6-smos+ #1 SMP Wed Mar 21 13:23:56 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux\r\nVERSION=\"16.04.3 LTS (Xenial Xerus)\"\r\nVERSION_ID=\"16.04\"\r\nVERSION_CODENAME=xenial\r\n\r\n== are we in docker =============================================\r\nNo\r\n\r\n== compiler =====================================================\r\nc++ (Ubuntu 5.4.0-6ubuntu1~16.04.10) 5.4.0 20160609\r\nCopyright (C) 2015 Free Software Foundation, Inc.\r\nThis is free software; see the source for copying conditions.  There is NO\r\nwarranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.\r\n\r\n\r\n== uname -a =====================================================\r\nLinux custom 4.16.0-rc6-smos+ #1 SMP Wed Mar 21 13:23:56 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux\r\n\r\n== check pips ===================================================\r\nnumpy               1.14.5\r\nprotobuf            3.6.0\r\ntensorflow          1.8.0rc1\r\n\r\n== check for virtualenv =========================================\r\nFalse\r\n\r\n== tensorflow import ============================================\r\ntf.VERSION = 1.8.0-rc1\r\ntf.GIT_VERSION = b'ComputeCpp-v0.6.0-4212-gb29ac8a'\r\ntf.COMPILER_VERSION = b'ComputeCpp-v0.6.0-4212-gb29ac8a'\r\nSanity check: array([1], dtype=int32)\r\n\r\n== env ==========================================================\r\nLD_LIBRARY_PATH /usr/local/lib:/usr/local/computecpp/lib:/usr/local/lib:/usr/local/computecpp/lib:\r\nDYLD_LIBRARY_PATH is unset\r\n\r\n== nvidia-smi ===================================================\r\nNVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.\r\n\r\n\r\n== cuda libs  ===================================================\r\n/usr/local/lib/libcudart.so.9.0.103\r\n```\r\n\r\n### Describe the problem\r\nTensorflow built on top of SYCL refuses to list and use all available GPUs in the system. I'm using the following commands to get list of devices:\r\n\r\n_(please note that TensorFlow's in-line log presents 8 devices, but the actual resulting variable contains just two CPU and one GPU available through \"/device:SYCL:0\" name)_\r\n```\r\n>>> from tensorflow.python.client import device_lib\r\n>>> device_lib.list_local_devices()\r\n2018-07-21 14:21:08.328612: I ./tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2\r\n2018-07-21 14:21:09.308907: I ./tensorflow/core/common_runtime/sycl/sycl_device.h:70] Found following OpenCL devices:\r\n2018-07-21 14:21:09.308981: I ./tensorflow/core/common_runtime/sycl/sycl_device.h:72] id: 0, type: GPU, name: Ellesmere, vendor: Advanced Micro Devices, Inc., profile: FULL_PROFILE\r\n2018-07-21 14:21:09.309001: I ./tensorflow/core/common_runtime/sycl/sycl_device.h:72] id: 1, type: GPU, name: Ellesmere, vendor: Advanced Micro Devices, Inc., profile: FULL_PROFILE\r\n2018-07-21 14:21:09.309019: I ./tensorflow/core/common_runtime/sycl/sycl_device.h:72] id: 2, type: GPU, name: Ellesmere, vendor: Advanced Micro Devices, Inc., profile: FULL_PROFILE\r\n2018-07-21 14:21:09.309034: I ./tensorflow/core/common_runtime/sycl/sycl_device.h:72] id: 3, type: GPU, name: Ellesmere, vendor: Advanced Micro Devices, Inc., profile: FULL_PROFILE\r\n2018-07-21 14:21:09.309052: I ./tensorflow/core/common_runtime/sycl/sycl_device.h:72] id: 4, type: GPU, name: Ellesmere, vendor: Advanced Micro Devices, Inc., profile: FULL_PROFILE\r\n2018-07-21 14:21:09.309068: I ./tensorflow/core/common_runtime/sycl/sycl_device.h:72] id: 5, type: GPU, name: Ellesmere, vendor: Advanced Micro Devices, Inc., profile: FULL_PROFILE\r\n2018-07-21 14:21:09.309085: I ./tensorflow/core/common_runtime/sycl/sycl_device.h:72] id: 6, type: GPU, name: Ellesmere, vendor: Advanced Micro Devices, Inc., profile: FULL_PROFILE\r\n2018-07-21 14:21:09.309101: I ./tensorflow/core/common_runtime/sycl/sycl_device.h:72] id: 7, type: GPU, name: Ellesmere, vendor: Advanced Micro Devices, Inc., profile: FULL_PROFILE\r\n[name: \"/device:CPU:0\"\r\ndevice_type: \"CPU\"\r\nmemory_limit: 268435456\r\nlocality {\r\n}\r\nincarnation: 911408516298923653\r\n, name: \"/device:SYCL:0\"\r\ndevice_type: \"SYCL\"\r\nmemory_limit: 268435456\r\nlocality {\r\n}\r\nincarnation: 161138719697210983\r\nphysical_device_desc: \"id: 0, type: GPU, name: Ellesmere, vendor: Advanced Micro Devices, Inc., profile: FULL_PROFILE\"\r\n]\r\n\r\n```\r\n\r\nI confirm that all devices are functional and available to OpenCL (visible to clinfo) and are operable through another 3-rd party package (ArrayFire). Also I confirm that SYCL itself sees all available devices, in order to test that purpose I've updated SYCL's 'custom-device-selector' example to following code:\r\n\r\n```\r\n/***************************************************************************\r\n *\r\n *  Copyright (C) 2016 Codeplay Software Limited\r\n *  Licensed under the Apache License, Version 2.0 (the \"License\");\r\n *  you may not use this file except in compliance with the License.\r\n *  You may obtain a copy of the License at\r\n *\r\n *      http://www.apache.org/licenses/LICENSE-2.0\r\n *\r\n *  For your convenience, a copy of the License has been included in this\r\n *  repository.\r\n *\r\n *  Unless required by applicable law or agreed to in writing, software\r\n *  distributed under the License is distributed on an \"AS IS\" BASIS,\r\n *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\r\n *  See the License for the specific language governing permissions and\r\n *  limitations under the License.\r\n *\r\n *  Codeplay's ComputeCpp SDK\r\n *\r\n *  custom-device-selector.cpp\r\n *\r\n *  Description:\r\n *    Sample code that shows how to write a custom device selector in SYCL.\r\n *\r\n **************************************************************************/\r\n\r\n#include <CL/sycl.hpp>\r\n#include <iostream>\r\n\r\nusing namespace cl::sycl;\r\nusing namespace std;\r\n\r\n/* Classes can inherit from the device_selector class to allow users\r\n * to dictate the criteria for choosing a device from those that might be\r\n * present on a system. This example looks for a device with SPIR support\r\n * and prefers GPUs over CPUs. */\r\nclass custom_selector : public device_selector {\r\n public:\r\n  custom_selector() : device_selector() {}\r\n\r\n  /* The selection is performed via the () operator in the base\r\n   * selector class.This method will be called once per device in each\r\n   * platform. Note that all platforms are evaluated whenever there is\r\n   * a device selection. */\r\n  int operator()(const device& device) const override {\r\n    cout << device.get_info<cl::sycl::info::device::vendor>() << \": \" << device.get_info<cl::sycl::info::device::name>() << std::endl; // << \"(\" << device.get_info<cl::sycl::info::device::device_type>() << \")\"\r\n    cout << '\\t' << \"max_work_group_size : \" << device.get_info<cl::sycl::info::device::max_work_group_size>() << std::endl;\r\n//    cout << '\\t' << \"max_work_item_sizes : \" << device.get_info<cl::sycl::info::device::max_work_item_sizes>() << std::endl;\r\n    cout << '\\t' << \"max_compute_units   : \" << device.get_info<cl::sycl::info::device::max_compute_units>() << std::endl;\r\n    cout << '\\t' << \"local_mem_size      : \" << device.get_info<cl::sycl::info::device::local_mem_size>() << std::endl;\r\n    cout << '\\t' << \"max_mem_alloc_size  : \" << device.get_info<cl::sycl::info::device::max_mem_alloc_size>() << std::endl;\r\n    cout << '\\t' << \"profile             : \" << device.get_info<cl::sycl::info::device::profile>() << std::endl;\r\n    cout << \"----------------------------------------------------------------------------------------------\" <<  std::endl << std::endl << std::endl;\r\n\r\n    /* We only give a valid score to devices that support SPIR. */\r\n    if (device.has_extension(cl::sycl::string_class(\"cl_khr_spir\"))) {\r\n      if (device.get_info<info::device::device_type>() ==\r\n          info::device_type::cpu) {\r\n        return 50;\r\n      }\r\n\r\n      if (device.get_info<info::device::device_type>() ==\r\n          info::device_type::gpu) {\r\n        return 100;\r\n      }\r\n    }\r\n    /* Devices with a negative score will never be chosen. */\r\n    return -1;\r\n  }\r\n};\r\n\r\nint main() {\r\n  const int dataSize = 64;\r\n  int ret = -1;\r\n  float data[dataSize] = {0.f};\r\n\r\n  range<1> dataRange(dataSize);\r\n  buffer<float, 1> buf(data, dataRange);\r\n\r\n  /* We create an object of custom_selector type and use it\r\n   * like any other selector. */\r\n  custom_selector selector;\r\n  queue myQueue(selector);\r\n\r\n  myQueue.submit([&](handler& cgh) {\r\n    auto ptr = buf.get_access<access::mode::read_write>(cgh);\r\n\r\n    cgh.parallel_for<class example_kernel>(dataRange, [=](item<1> item) {\r\n      size_t idx = item.get_linear_id();\r\n      ptr[item.get_linear_id()] = static_cast<float>(idx);\r\n    });\r\n  });\r\n\r\n  /* A host accessor can be used to force an update from the device to the\r\n   * host, allowing the data to be checked. */\r\n  accessor<float, 1, access::mode::read_write, access::target::host_buffer>\r\n      hostPtr(buf);\r\n\r\n  if (hostPtr[10] == 10.0f) {\r\n    ret = 0;\r\n  }\r\n\r\n  return ret;\r\n}\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/lukeiwanski/tensorflow/issues/comments/406896491","html_url":"https://github.com/lukeiwanski/tensorflow/issues/252#issuecomment-406896491","issue_url":"https://api.github.com/repos/lukeiwanski/tensorflow/issues/252","id":406896491,"node_id":"MDEyOklzc3VlQ29tbWVudDQwNjg5NjQ5MQ==","user":{"login":"mirh","id":9366725,"node_id":"MDQ6VXNlcjkzNjY3MjU=","avatar_url":"https://avatars1.githubusercontent.com/u/9366725?v=4","gravatar_id":"","url":"https://api.github.com/users/mirh","html_url":"https://github.com/mirh","followers_url":"https://api.github.com/users/mirh/followers","following_url":"https://api.github.com/users/mirh/following{/other_user}","gists_url":"https://api.github.com/users/mirh/gists{/gist_id}","starred_url":"https://api.github.com/users/mirh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mirh/subscriptions","organizations_url":"https://api.github.com/users/mirh/orgs","repos_url":"https://api.github.com/users/mirh/repos","events_url":"https://api.github.com/users/mirh/events{/privacy}","received_events_url":"https://api.github.com/users/mirh/received_events","type":"User","site_admin":false},"created_at":"2018-07-22T21:02:47Z","updated_at":"2018-07-22T21:02:47Z","author_association":"NONE","body":"Try computecpp 0.9.0 for starters?"}}
{"actor":{"display_login":"mirh"}}