{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/22619","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/22619/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/22619/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/22619/events","html_url":"https://github.com/tensorflow/tensorflow/issues/22619","id":365149052,"node_id":"MDU6SXNzdWUzNjUxNDkwNTI=","number":22619,"title":"Unable to use multiple CPU cores in TensorFlow","user":{"login":"elliottslaughter","id":3129,"node_id":"MDQ6VXNlcjMxMjk=","avatar_url":"https://avatars0.githubusercontent.com/u/3129?v=4","gravatar_id":"","url":"https://api.github.com/users/elliottslaughter","html_url":"https://github.com/elliottslaughter","followers_url":"https://api.github.com/users/elliottslaughter/followers","following_url":"https://api.github.com/users/elliottslaughter/following{/other_user}","gists_url":"https://api.github.com/users/elliottslaughter/gists{/gist_id}","starred_url":"https://api.github.com/users/elliottslaughter/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/elliottslaughter/subscriptions","organizations_url":"https://api.github.com/users/elliottslaughter/orgs","repos_url":"https://api.github.com/users/elliottslaughter/repos","events_url":"https://api.github.com/users/elliottslaughter/events{/privacy}","received_events_url":"https://api.github.com/users/elliottslaughter/received_events","type":"User","site_admin":false},"labels":[{"id":386191887,"node_id":"MDU6TGFiZWwzODYxOTE4ODc=","url":"https://api.github.com/repos/tensorflow/tensorflow/labels/stat:awaiting%20response","name":"stat:awaiting response","color":"5319e7","default":false}],"state":"open","locked":false,"assignee":{"login":"harshini-gadige","id":42781361,"node_id":"MDQ6VXNlcjQyNzgxMzYx","avatar_url":"https://avatars1.githubusercontent.com/u/42781361?v=4","gravatar_id":"","url":"https://api.github.com/users/harshini-gadige","html_url":"https://github.com/harshini-gadige","followers_url":"https://api.github.com/users/harshini-gadige/followers","following_url":"https://api.github.com/users/harshini-gadige/following{/other_user}","gists_url":"https://api.github.com/users/harshini-gadige/gists{/gist_id}","starred_url":"https://api.github.com/users/harshini-gadige/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/harshini-gadige/subscriptions","organizations_url":"https://api.github.com/users/harshini-gadige/orgs","repos_url":"https://api.github.com/users/harshini-gadige/repos","events_url":"https://api.github.com/users/harshini-gadige/events{/privacy}","received_events_url":"https://api.github.com/users/harshini-gadige/received_events","type":"User","site_admin":false},"assignees":[{"login":"harshini-gadige","id":42781361,"node_id":"MDQ6VXNlcjQyNzgxMzYx","avatar_url":"https://avatars1.githubusercontent.com/u/42781361?v=4","gravatar_id":"","url":"https://api.github.com/users/harshini-gadige","html_url":"https://github.com/harshini-gadige","followers_url":"https://api.github.com/users/harshini-gadige/followers","following_url":"https://api.github.com/users/harshini-gadige/following{/other_user}","gists_url":"https://api.github.com/users/harshini-gadige/gists{/gist_id}","starred_url":"https://api.github.com/users/harshini-gadige/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/harshini-gadige/subscriptions","organizations_url":"https://api.github.com/users/harshini-gadige/orgs","repos_url":"https://api.github.com/users/harshini-gadige/repos","events_url":"https://api.github.com/users/harshini-gadige/events{/privacy}","received_events_url":"https://api.github.com/users/harshini-gadige/received_events","type":"User","site_admin":false}],"milestone":null,"comments":1,"created_at":"2018-09-29T17:01:59Z","updated_at":"2018-10-01T23:44:33Z","closed_at":null,"author_association":"NONE","body":"This issue was previously asked here on StackOverflow (with no answers at the time of this issue): https://stackoverflow.com/q/52507748/188046\r\n\r\n------------------------\r\n\r\n### System information\r\n- **Have I written custom code**: custom Python code, but no custom ops (Python is linked below)\r\n- **OS Platform and Distribution**: Ubuntu 16.04.5\r\n- **Mobile device**: N/A\r\n- **TensorFlow installed from**: binary, from `tensorflow` PyPI package via pip (also tried from conda with same result)\r\n- **TensorFlow version**:  v1.11.0-0-gc19e29306c 1.11.0\r\n- **Python version**: 3.6.6 from conda\r\n- **Bazel version**: N/A\r\n- **GCC/Compiler version**: N/A\r\n- **CUDA/cuDNN version**: N/A, problem occurs on CPU\r\n- **GPU model and memory**: N/A, problem occurs on CPU\r\n- **CPU model**: Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz (2 sockets, 10 cores per socket)\r\n- **Exact command to reproduce**: See below\r\n\r\n### Describe the problem\r\n\r\nI am unable to configure TensorFlow to use multiple CPU cores for inter-op parallelism on my machine. As described in my [StackOverflow question](https://stackoverflow.com/q/52507748/188046), I have read other answers extensively, and scrubbed the first page of Google search results for several keywords, and tried everything I've seen suggested, and I just can't get this to work.\r\n\r\nI have included a program below that demonstrates the problem. The program calls `matmul` once per core (i.e. weak scaling). I would expect that as the number of cores increases, the running time would stay roughly constant. Instead the running time seems to increase linearly with the core count, indicating that the `matmul` ops are running sequentially, not in parallel.\r\n\r\nI have also confirmed via `htop` that there is only one core on my CPU that is in use when the program is running. The system is otherwise idle. `htop` has the capability to show multiple threads within a process, but I do not even see these (or they are not using enough CPU to show up on the first page of results when sorted by CPU usage).\r\n\r\nHow can I get TensorFlow to execute different operations on different cores in parallel?\r\n\r\nNote:\r\n\r\n  * I am creating a session with multiple CPU devices. I have also tried only creating a single CPU device, and relying entirely on `inter_op_parallelism_threads`. Nothing I have tried has been able to use multiple cores.\r\n  * I can comment out the line `with tf.device(d):`, and it makes no difference.\r\n  * I have tried tracing (see the commented out lines), and the trace seems to reflect what I'd expect. Ops are being assigned to the CPUs like I want them to be. However, they still don't run in parallel.\r\n  * I have also tried generating a Chrome trace (commented out lines at the very bottom). The Chrome trace doesn't seem to be working properly, or at least the reported running times are way off what they should be. So I'm not sure how much this information can be relied upon. Perhaps I'm doing something wrong.\r\n\r\n### Source code / logs\r\n\r\nSource for test_cores.py: https://gist.github.com/elliottslaughter/750a27c832782f4daec8686281027de8\r\n\r\nSample output:\r\n\r\n```\r\n$ python test_cores.py \r\n2018-09-29 09:40:34.489657: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary \r\nwas not compiled to use: AVX2 FMA\r\nRunning on 1 cores\r\n  Assigning matmul to /cpu:0\r\n  Duration (via time.perf_counter()): 3.209691 (693823.014392 - 693819.804701)\r\n  Clock (via time.clock()): 3.205479 (8.912035 - 5.706556)\r\nRunning on 2 cores\r\n  Assigning matmul to /cpu:0\r\n  Assigning matmul to /cpu:1\r\n  Duration (via time.perf_counter()): 6.452124 (693829.493906 - 693823.041782)\r\n  Clock (via time.clock()): 6.449224 (15.388567 - 8.939343)\r\n```"},"comment":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/comments/426099811","html_url":"https://github.com/tensorflow/tensorflow/issues/22619#issuecomment-426099811","issue_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/22619","id":426099811,"node_id":"MDEyOklzc3VlQ29tbWVudDQyNjA5OTgxMQ==","user":{"login":"elliottslaughter","id":3129,"node_id":"MDQ6VXNlcjMxMjk=","avatar_url":"https://avatars0.githubusercontent.com/u/3129?v=4","gravatar_id":"","url":"https://api.github.com/users/elliottslaughter","html_url":"https://github.com/elliottslaughter","followers_url":"https://api.github.com/users/elliottslaughter/followers","following_url":"https://api.github.com/users/elliottslaughter/following{/other_user}","gists_url":"https://api.github.com/users/elliottslaughter/gists{/gist_id}","starred_url":"https://api.github.com/users/elliottslaughter/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/elliottslaughter/subscriptions","organizations_url":"https://api.github.com/users/elliottslaughter/orgs","repos_url":"https://api.github.com/users/elliottslaughter/repos","events_url":"https://api.github.com/users/elliottslaughter/events{/privacy}","received_events_url":"https://api.github.com/users/elliottslaughter/received_events","type":"User","site_admin":false},"created_at":"2018-10-01T23:44:33Z","updated_at":"2018-10-01T23:44:33Z","author_association":"NONE","body":"@harshini-gadige Thanks for the suggestion. Though in my real use case, I'm not especially interested in intra-op parallelism, you're right that this is an interesting data point for debugging purposes.\r\n\r\nUnfortunately, changing the value of `intra_op_parallelism` doesn't seem to make a difference. E.g. when I set `intra_op_parallelism_threads=n_cpus` I still get the exact same running times as when I set `intra_op_parallelism_threads=1`. I also get the same running times if I hard-code the value to e.g. `4`."}}
{"actor":{"display_login":"elliottslaughter"}}