{"action":"created","issue":{"url":"https://api.github.com/repos/dask/dask-jobqueue/issues/181","repository_url":"https://api.github.com/repos/dask/dask-jobqueue","labels_url":"https://api.github.com/repos/dask/dask-jobqueue/issues/181/labels{/name}","comments_url":"https://api.github.com/repos/dask/dask-jobqueue/issues/181/comments","events_url":"https://api.github.com/repos/dask/dask-jobqueue/issues/181/events","html_url":"https://github.com/dask/dask-jobqueue/issues/181","id":372752428,"node_id":"MDU6SXNzdWUzNzI3NTI0Mjg=","number":181,"title":"Multiple cores per process/thread","user":{"login":"dgasmith","id":1769841,"node_id":"MDQ6VXNlcjE3Njk4NDE=","avatar_url":"https://avatars2.githubusercontent.com/u/1769841?v=4","gravatar_id":"","url":"https://api.github.com/users/dgasmith","html_url":"https://github.com/dgasmith","followers_url":"https://api.github.com/users/dgasmith/followers","following_url":"https://api.github.com/users/dgasmith/following{/other_user}","gists_url":"https://api.github.com/users/dgasmith/gists{/gist_id}","starred_url":"https://api.github.com/users/dgasmith/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dgasmith/subscriptions","organizations_url":"https://api.github.com/users/dgasmith/orgs","repos_url":"https://api.github.com/users/dgasmith/repos","events_url":"https://api.github.com/users/dgasmith/events{/privacy}","received_events_url":"https://api.github.com/users/dgasmith/received_events","type":"User","site_admin":false},"labels":[{"id":853197927,"node_id":"MDU6TGFiZWw4NTMxOTc5Mjc=","url":"https://api.github.com/repos/dask/dask-jobqueue/labels/enhancement","name":"enhancement","color":"a2eeef","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":22,"created_at":"2018-10-23T00:25:00Z","updated_at":"2018-11-07T16:07:04Z","closed_at":null,"author_association":"NONE","body":"We have a use case where we would like to dask.distributed to parallelize a python-bound C++ program that would work best if it could consume 8-32 threads depending on the problem size and will manage threading internally. Normally, a Dask worker is run on a node that has 32 cores with the worker using 2 processes at 1 thread each so that we can give each process 16 cores.\r\n\r\nLooking at the code currently, it seems that `nthreads = ncores / nprocesses` without exceptions, is there a canonical way to change this so that we can orchestrate our normal Dask worker operation with dask-jobqueue?"},"comment":{"url":"https://api.github.com/repos/dask/dask-jobqueue/issues/comments/436677069","html_url":"https://github.com/dask/dask-jobqueue/issues/181#issuecomment-436677069","issue_url":"https://api.github.com/repos/dask/dask-jobqueue/issues/181","id":436677069,"node_id":"MDEyOklzc3VlQ29tbWVudDQzNjY3NzA2OQ==","user":{"login":"lesteve","id":1680079,"node_id":"MDQ6VXNlcjE2ODAwNzk=","avatar_url":"https://avatars1.githubusercontent.com/u/1680079?v=4","gravatar_id":"","url":"https://api.github.com/users/lesteve","html_url":"https://github.com/lesteve","followers_url":"https://api.github.com/users/lesteve/followers","following_url":"https://api.github.com/users/lesteve/following{/other_user}","gists_url":"https://api.github.com/users/lesteve/gists{/gist_id}","starred_url":"https://api.github.com/users/lesteve/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lesteve/subscriptions","organizations_url":"https://api.github.com/users/lesteve/orgs","repos_url":"https://api.github.com/users/lesteve/repos","events_url":"https://api.github.com/users/lesteve/events{/privacy}","received_events_url":"https://api.github.com/users/lesteve/received_events","type":"User","site_admin":false},"created_at":"2018-11-07T16:07:04Z","updated_at":"2018-11-07T16:07:04Z","author_association":"COLLABORATOR","body":"I am still unconvinced that dask-jobqueue has to change to accommodate this use case. One reason I am not a big fan of the general direction taken by this issue: suppose some of your tasks are doing multi-threading internally but some of your tasks don't (e.g. in the OP case, once your C++ code has run you need to run additional analysis code in Python). Basically you have booked some resources that your dask-worker is unable to use during the Python analysis phase.\r\n\r\nI only recently realised that worker resources could be used for the original OP's use case, see this related [SO question and answer](https://stackoverflow.com/q/45052535).\r\n\r\n"}}
{"actor":{"display_login":"lesteve"}}