{"action":"created","comment":{"url":"https://api.github.com/repos/docker/swarmkit/pulls/comments/74528780","id":74528780,"diff_hunk":"@@ -696,3 +696,9 @@ message ManagerStatus {\n \t// Reachability specifies whether this node is reachable.\n \tRaftMemberStatus.Reachability reachability = 4;\n }\n+\n+message Device {\n+\tstring source = 1;\n+\tstring target = 2;\n+\tstring permissions = 3;","path":"api/types.proto","position":8,"original_position":8,"commit_id":"3b4c9af3e6c0953c387db776b9285116231926db","original_commit_id":"3b4c9af3e6c0953c387db776b9285116231926db","user":{"login":"flx42","id":3645581,"avatar_url":"https://avatars.githubusercontent.com/u/3645581?v=3","gravatar_id":"","url":"https://api.github.com/users/flx42","html_url":"https://github.com/flx42","followers_url":"https://api.github.com/users/flx42/followers","following_url":"https://api.github.com/users/flx42/following{/other_user}","gists_url":"https://api.github.com/users/flx42/gists{/gist_id}","starred_url":"https://api.github.com/users/flx42/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/flx42/subscriptions","organizations_url":"https://api.github.com/users/flx42/orgs","repos_url":"https://api.github.com/users/flx42/repos","events_url":"https://api.github.com/users/flx42/events{/privacy}","received_events_url":"https://api.github.com/users/flx42/received_events","type":"User","site_admin":false},"body":"It's actually more complicated than that. If you have 4 GPUs on your machine, the available devices will be:\r\n```\r\n$  ls /dev/nvidia*\r\n/dev/nvidia0  /dev/nvidia1  /dev/nvidia2  /dev/nvidia3  /dev/nvidiactl  /dev/nvidia-modeset  /dev/nvidia-uvm  /dev/nvidia-uvm-tools\r\n```\r\nDevices `/dev/nvidia[0-3]` are the individual GPUs that can be selected, but the others are control devices that **must** be shared between all containers, so it will not fit this class abstraction and will require additional logic.\r\n\r\nOther considerations:\r\n- For the sake of completeness, let's list other potential interesting devices apart from GPUs:\r\ndisks (`/dev/sd*`), Infiniband (`/dev/infiniband/*`), other types of accelerators (FPGAs, Xeon Phi).\r\n- Not all the GPUs are necessarily the same on a single machine, `/dev/nvidia0` could be 2x faster than `/dev/nvidia1`. While this consideration might be advanced, it's interesting to keep it in mind, if we want to enable that in the future.\r\n- If we consider disks, would it make sense to have a `/dev/sd*` class? Does it make sense to have a service exclusively owning `/dev/sda`? I'm not sure the \"class\" abstraction could extend well to other types of devices.\r\n\r\nThe question of supporting additional resources and having a scheduler aware of these resources is a complex one. So, here's what I'm suggesting:\r\n1) We should allow devices in services, because a services might need a device, any device, no questions asked. It will be the responsibility of the user to make sure the device exists on all nodes.\r\n2) We can start a longer discussion regarding supporting additional resource types (not only GPUs) in Swarm.\r\n\r\n","created_at":"2016-08-12T00:50:53Z","updated_at":"2016-08-12T00:50:53Z","html_url":"https://github.com/docker/swarmkit/pull/1355#discussion_r74528780","pull_request_url":"https://api.github.com/repos/docker/swarmkit/pulls/1355","_links":{"self":{"href":"https://api.github.com/repos/docker/swarmkit/pulls/comments/74528780"},"html":{"href":"https://github.com/docker/swarmkit/pull/1355#discussion_r74528780"},"pull_request":{"href":"https://api.github.com/repos/docker/swarmkit/pulls/1355"}}},"pull_request":{"url":"https://api.github.com/repos/docker/swarmkit/pulls/1355","id":80720736,"html_url":"https://github.com/docker/swarmkit/pull/1355","diff_url":"https://github.com/docker/swarmkit/pull/1355.diff","patch_url":"https://github.com/docker/swarmkit/pull/1355.patch","issue_url":"https://api.github.com/repos/docker/swarmkit/issues/1355","number":1355,"state":"open","locked":false,"title":"[WIP] api: add support for devices","user":{"login":"flx42","id":3645581,"avatar_url":"https://avatars.githubusercontent.com/u/3645581?v=3","gravatar_id":"","url":"https://api.github.com/users/flx42","html_url":"https://github.com/flx42","followers_url":"https://api.github.com/users/flx42/followers","following_url":"https://api.github.com/users/flx42/following{/other_user}","gists_url":"https://api.github.com/users/flx42/gists{/gist_id}","starred_url":"https://api.github.com/users/flx42/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/flx42/subscriptions","organizations_url":"https://api.github.com/users/flx42/orgs","repos_url":"https://api.github.com/users/flx42/repos","events_url":"https://api.github.com/users/flx42/events{/privacy}","received_events_url":"https://api.github.com/users/flx42/received_events","type":"User","site_admin":false},"body":"See #1244, sorry for the delay.\r\nIn the end, I moved the devices to `ContainerSpec` as @stevvooe suggested [here](https://github.com/docker/swarmkit/issues/1244#issuecomment-235775570)\r\n\r\nUsing this patch, I'm able to mimic what [nvidia-docker](https://github.com/nvidia/nvidia-docker) is doing and I can launch DIGITS:\r\n```\r\nbin/swarmctl service create --device /dev/nvidia-uvm --device /dev/nvidiactl --device /dev/nvidia0 --bind /var/lib/nvidia-docker/volumes/nvidia_driver/367.35:/usr/local/nvidia --image nvidia/digits:4.0 --name digits\r\n```\r\n\r\nOr, if you want to test and since you might not have a NVIDIA GPU, you can also use one of the image from @jfrazelle for testing, for instance VLC in Swarm (wat):\r\n```\r\nbin/swarmctl service create --env DISPLAY=unix$DISPLAY --device /dev/snd --bind /tmp/.X11-unix:/tmp/.X11-unix --image jess/vlc --name vlc\r\n```\r\nAnd then play a network stream to verify sound works.\r\n\r\nIf this is accepted, I will need help regarding enabling support in the engine too.","created_at":"2016-08-10T01:16:28Z","updated_at":"2016-08-12T00:50:53Z","closed_at":null,"merged_at":null,"merge_commit_sha":"7317173e99314c0b278c7e462b8c749247101e25","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/docker/swarmkit/pulls/1355/commits","review_comments_url":"https://api.github.com/repos/docker/swarmkit/pulls/1355/comments","review_comment_url":"https://api.github.com/repos/docker/swarmkit/pulls/comments{/number}","comments_url":"https://api.github.com/repos/docker/swarmkit/issues/1355/comments","statuses_url":"https://api.github.com/repos/docker/swarmkit/statuses/3b4c9af3e6c0953c387db776b9285116231926db","head":{"label":"flx42:service-device-support","ref":"service-device-support","sha":"3b4c9af3e6c0953c387db776b9285116231926db","user":{"login":"flx42","id":3645581,"avatar_url":"https://avatars.githubusercontent.com/u/3645581?v=3","gravatar_id":"","url":"https://api.github.com/users/flx42","html_url":"https://github.com/flx42","followers_url":"https://api.github.com/users/flx42/followers","following_url":"https://api.github.com/users/flx42/following{/other_user}","gists_url":"https://api.github.com/users/flx42/gists{/gist_id}","starred_url":"https://api.github.com/users/flx42/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/flx42/subscriptions","organizations_url":"https://api.github.com/users/flx42/orgs","repos_url":"https://api.github.com/users/flx42/repos","events_url":"https://api.github.com/users/flx42/events{/privacy}","received_events_url":"https://api.github.com/users/flx42/received_events","type":"User","site_admin":false},"repo":{"id":64349632,"name":"swarmkit","full_name":"flx42/swarmkit","owner":{"login":"flx42","id":3645581,"avatar_url":"https://avatars.githubusercontent.com/u/3645581?v=3","gravatar_id":"","url":"https://api.github.com/users/flx42","html_url":"https://github.com/flx42","followers_url":"https://api.github.com/users/flx42/followers","following_url":"https://api.github.com/users/flx42/following{/other_user}","gists_url":"https://api.github.com/users/flx42/gists{/gist_id}","starred_url":"https://api.github.com/users/flx42/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/flx42/subscriptions","organizations_url":"https://api.github.com/users/flx42/orgs","repos_url":"https://api.github.com/users/flx42/repos","events_url":"https://api.github.com/users/flx42/events{/privacy}","received_events_url":"https://api.github.com/users/flx42/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/flx42/swarmkit","description":"A toolkit for orchestrating distributed systems at any scale. It includes primitives for node discovery, raft-based consensus, task scheduling and more.","fork":true,"url":"https://api.github.com/repos/flx42/swarmkit","forks_url":"https://api.github.com/repos/flx42/swarmkit/forks","keys_url":"https://api.github.com/repos/flx42/swarmkit/keys{/key_id}","collaborators_url":"https://api.github.com/repos/flx42/swarmkit/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/flx42/swarmkit/teams","hooks_url":"https://api.github.com/repos/flx42/swarmkit/hooks","issue_events_url":"https://api.github.com/repos/flx42/swarmkit/issues/events{/number}","events_url":"https://api.github.com/repos/flx42/swarmkit/events","assignees_url":"https://api.github.com/repos/flx42/swarmkit/assignees{/user}","branches_url":"https://api.github.com/repos/flx42/swarmkit/branches{/branch}","tags_url":"https://api.github.com/repos/flx42/swarmkit/tags","blobs_url":"https://api.github.com/repos/flx42/swarmkit/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/flx42/swarmkit/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/flx42/swarmkit/git/refs{/sha}","trees_url":"https://api.github.com/repos/flx42/swarmkit/git/trees{/sha}","statuses_url":"https://api.github.com/repos/flx42/swarmkit/statuses/{sha}","languages_url":"https://api.github.com/repos/flx42/swarmkit/languages","stargazers_url":"https://api.github.com/repos/flx42/swarmkit/stargazers","contributors_url":"https://api.github.com/repos/flx42/swarmkit/contributors","subscribers_url":"https://api.github.com/repos/flx42/swarmkit/subscribers","subscription_url":"https://api.github.com/repos/flx42/swarmkit/subscription","commits_url":"https://api.github.com/repos/flx42/swarmkit/commits{/sha}","git_commits_url":"https://api.github.com/repos/flx42/swarmkit/git/commits{/sha}","comments_url":"https://api.github.com/repos/flx42/swarmkit/comments{/number}","issue_comment_url":"https://api.github.com/repos/flx42/swarmkit/issues/comments{/number}","contents_url":"https://api.github.com/repos/flx42/swarmkit/contents/{+path}","compare_url":"https://api.github.com/repos/flx42/swarmkit/compare/{base}...{head}","merges_url":"https://api.github.com/repos/flx42/swarmkit/merges","archive_url":"https://api.github.com/repos/flx42/swarmkit/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/flx42/swarmkit/downloads","issues_url":"https://api.github.com/repos/flx42/swarmkit/issues{/number}","pulls_url":"https://api.github.com/repos/flx42/swarmkit/pulls{/number}","milestones_url":"https://api.github.com/repos/flx42/swarmkit/milestones{/number}","notifications_url":"https://api.github.com/repos/flx42/swarmkit/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/flx42/swarmkit/labels{/name}","releases_url":"https://api.github.com/repos/flx42/swarmkit/releases{/id}","deployments_url":"https://api.github.com/repos/flx42/swarmkit/deployments","created_at":"2016-07-27T23:56:56Z","updated_at":"2016-07-27T23:56:57Z","pushed_at":"2016-08-10T17:04:45Z","git_url":"git://github.com/flx42/swarmkit.git","ssh_url":"git@github.com:flx42/swarmkit.git","clone_url":"https://github.com/flx42/swarmkit.git","svn_url":"https://github.com/flx42/swarmkit","homepage":"","size":8911,"stargazers_count":0,"watchers_count":0,"language":"Go","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"docker:master","ref":"master","sha":"abb15df5e4f634445329fbb6f887a56b9b399789","user":{"login":"docker","id":5429470,"avatar_url":"https://avatars.githubusercontent.com/u/5429470?v=3","gravatar_id":"","url":"https://api.github.com/users/docker","html_url":"https://github.com/docker","followers_url":"https://api.github.com/users/docker/followers","following_url":"https://api.github.com/users/docker/following{/other_user}","gists_url":"https://api.github.com/users/docker/gists{/gist_id}","starred_url":"https://api.github.com/users/docker/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/docker/subscriptions","organizations_url":"https://api.github.com/users/docker/orgs","repos_url":"https://api.github.com/users/docker/repos","events_url":"https://api.github.com/users/docker/events{/privacy}","received_events_url":"https://api.github.com/users/docker/received_events","type":"Organization","site_admin":false},"repo":{"id":51556828,"name":"swarmkit","full_name":"docker/swarmkit","owner":{"login":"docker","id":5429470,"avatar_url":"https://avatars.githubusercontent.com/u/5429470?v=3","gravatar_id":"","url":"https://api.github.com/users/docker","html_url":"https://github.com/docker","followers_url":"https://api.github.com/users/docker/followers","following_url":"https://api.github.com/users/docker/following{/other_user}","gists_url":"https://api.github.com/users/docker/gists{/gist_id}","starred_url":"https://api.github.com/users/docker/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/docker/subscriptions","organizations_url":"https://api.github.com/users/docker/orgs","repos_url":"https://api.github.com/users/docker/repos","events_url":"https://api.github.com/users/docker/events{/privacy}","received_events_url":"https://api.github.com/users/docker/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/docker/swarmkit","description":"A toolkit for orchestrating distributed systems at any scale. It includes primitives for node discovery, raft-based consensus, task scheduling and more.","fork":false,"url":"https://api.github.com/repos/docker/swarmkit","forks_url":"https://api.github.com/repos/docker/swarmkit/forks","keys_url":"https://api.github.com/repos/docker/swarmkit/keys{/key_id}","collaborators_url":"https://api.github.com/repos/docker/swarmkit/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/docker/swarmkit/teams","hooks_url":"https://api.github.com/repos/docker/swarmkit/hooks","issue_events_url":"https://api.github.com/repos/docker/swarmkit/issues/events{/number}","events_url":"https://api.github.com/repos/docker/swarmkit/events","assignees_url":"https://api.github.com/repos/docker/swarmkit/assignees{/user}","branches_url":"https://api.github.com/repos/docker/swarmkit/branches{/branch}","tags_url":"https://api.github.com/repos/docker/swarmkit/tags","blobs_url":"https://api.github.com/repos/docker/swarmkit/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/docker/swarmkit/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/docker/swarmkit/git/refs{/sha}","trees_url":"https://api.github.com/repos/docker/swarmkit/git/trees{/sha}","statuses_url":"https://api.github.com/repos/docker/swarmkit/statuses/{sha}","languages_url":"https://api.github.com/repos/docker/swarmkit/languages","stargazers_url":"https://api.github.com/repos/docker/swarmkit/stargazers","contributors_url":"https://api.github.com/repos/docker/swarmkit/contributors","subscribers_url":"https://api.github.com/repos/docker/swarmkit/subscribers","subscription_url":"https://api.github.com/repos/docker/swarmkit/subscription","commits_url":"https://api.github.com/repos/docker/swarmkit/commits{/sha}","git_commits_url":"https://api.github.com/repos/docker/swarmkit/git/commits{/sha}","comments_url":"https://api.github.com/repos/docker/swarmkit/comments{/number}","issue_comment_url":"https://api.github.com/repos/docker/swarmkit/issues/comments{/number}","contents_url":"https://api.github.com/repos/docker/swarmkit/contents/{+path}","compare_url":"https://api.github.com/repos/docker/swarmkit/compare/{base}...{head}","merges_url":"https://api.github.com/repos/docker/swarmkit/merges","archive_url":"https://api.github.com/repos/docker/swarmkit/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/docker/swarmkit/downloads","issues_url":"https://api.github.com/repos/docker/swarmkit/issues{/number}","pulls_url":"https://api.github.com/repos/docker/swarmkit/pulls{/number}","milestones_url":"https://api.github.com/repos/docker/swarmkit/milestones{/number}","notifications_url":"https://api.github.com/repos/docker/swarmkit/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/docker/swarmkit/labels{/name}","releases_url":"https://api.github.com/repos/docker/swarmkit/releases{/id}","deployments_url":"https://api.github.com/repos/docker/swarmkit/deployments","created_at":"2016-02-12T00:02:15Z","updated_at":"2016-08-11T19:26:10Z","pushed_at":"2016-08-12T00:44:33Z","git_url":"git://github.com/docker/swarmkit.git","ssh_url":"git@github.com:docker/swarmkit.git","clone_url":"https://github.com/docker/swarmkit.git","svn_url":"https://github.com/docker/swarmkit","homepage":"","size":19933,"stargazers_count":874,"watchers_count":874,"language":"Go","has_issues":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":155,"mirror_url":null,"open_issues_count":161,"forks":155,"open_issues":161,"watchers":874,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/docker/swarmkit/pulls/1355"},"html":{"href":"https://github.com/docker/swarmkit/pull/1355"},"issue":{"href":"https://api.github.com/repos/docker/swarmkit/issues/1355"},"comments":{"href":"https://api.github.com/repos/docker/swarmkit/issues/1355/comments"},"review_comments":{"href":"https://api.github.com/repos/docker/swarmkit/pulls/1355/comments"},"review_comment":{"href":"https://api.github.com/repos/docker/swarmkit/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/docker/swarmkit/pulls/1355/commits"},"statuses":{"href":"https://api.github.com/repos/docker/swarmkit/statuses/3b4c9af3e6c0953c387db776b9285116231926db"}}}}
{"actor":{"display_login":"flx42"}}