{"action":"created","issue":{"url":"https://api.github.com/repos/docker/swarmkit/issues/2682","repository_url":"https://api.github.com/repos/docker/swarmkit","labels_url":"https://api.github.com/repos/docker/swarmkit/issues/2682/labels{/name}","comments_url":"https://api.github.com/repos/docker/swarmkit/issues/2682/comments","events_url":"https://api.github.com/repos/docker/swarmkit/issues/2682/events","html_url":"https://github.com/docker/swarmkit/issues/2682","id":337678704,"node_id":"MDU6SXNzdWUzMzc2Nzg3MDQ=","number":2682,"title":"Proposal: Device Support","user":{"login":"dperny","id":2367858,"node_id":"MDQ6VXNlcjIzNjc4NTg=","avatar_url":"https://avatars3.githubusercontent.com/u/2367858?v=4","gravatar_id":"","url":"https://api.github.com/users/dperny","html_url":"https://github.com/dperny","followers_url":"https://api.github.com/users/dperny/followers","following_url":"https://api.github.com/users/dperny/following{/other_user}","gists_url":"https://api.github.com/users/dperny/gists{/gist_id}","starred_url":"https://api.github.com/users/dperny/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dperny/subscriptions","organizations_url":"https://api.github.com/users/dperny/orgs","repos_url":"https://api.github.com/users/dperny/repos","events_url":"https://api.github.com/users/dperny/events{/privacy}","received_events_url":"https://api.github.com/users/dperny/received_events","type":"User","site_admin":false},"labels":[{"id":388007593,"node_id":"MDU6TGFiZWwzODgwMDc1OTM=","url":"https://api.github.com/repos/docker/swarmkit/labels/kind/feature","name":"kind/feature","color":"AAB3AB","default":false},{"id":660837429,"node_id":"MDU6TGFiZWw2NjA4Mzc0Mjk=","url":"https://api.github.com/repos/docker/swarmkit/labels/kind/proposal","name":"kind/proposal","color":"1d76db","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2018-07-02T22:00:47Z","updated_at":"2018-07-09T22:21:41Z","closed_at":null,"author_association":"MEMBER","body":"This is a rough overview of a proposed design for device support in Swarm. This is a possible implementation of #1244. The objective is to implement, in a way sensible to the cluster, support for devices. **Please note** that this is not yet on the road map; this is an early-stage proposal.\r\n\r\n### For community members, even if you don't or haven't contributed directly to swarmkit:\r\n\r\nDoes this meet or exceed the community's needs for device support? Is the UI flexible, ergonomic, and easy to use? Feel free to leave a comment explaining what is good and bad about this proposal.\r\n\r\n# Overview\r\nDevices will be added as a first-class feature of swarm. The user will be able to define device classes, to which devices belong to. The user will be able to register devices on specific nodes, indicating to what class the device belongs to and what path the device is located at. The user can then specify device classes that a task needs to execute, and the swarmkit scheduler will assign a device to the task and place the task on the node with that device.\r\n\r\n## Goals\r\nThe goal of this proposal is to implement the most basic device-aware scheduling system, to swarmkit to fully support devices in a clustered environment.\r\n\r\n## Non-Goals\r\n\r\nNon-goals of this proposal are to support things like security profiles or permissions. Additionally, though the device management workflow presented in this PR is a bit onerous and requires manual registration of devices, implementing automatic device detection and registration is out of scope.\r\n\r\n# Detailed Design\r\n\r\n## Data Model\r\nThe basic data model of devices is as follows:\r\n\r\n1. Device Classes represent a set of interchangeable and equivalent devices equally suited for scheduling. All devices belong to exactly one device class. \r\n2. Individual devices will be registered belonging to a class on a per-node basis. Once registered, a task may be assigned to use them.\r\n3. Task Specs will be updated to include the desired device classes and attachment options.\r\n\r\nDevices are host-specific resources, but different devices on the same or different hosts may possibly be treated as interchangeable or equivalent. For example, many nodes in the cluster could possibly be attached to some GPU. Though the actual GPU on different nodes may be different, and there may even be more than one GPU per node, their functionality is equivalent, and any of these nodes is an equally suitable candidate for scheduling. Further, some devices should only be used by one task in the cluster, whereas others can be shared between as many tasks as needed.\r\n\r\nDevice classes are the object that represents the top-level concept of a device. Tasks can only specify devices in terms of device classes they desire. The specific device chosen is the prerogative of the swarmkit scheduler.\r\n\r\nThe individual devices available are a property of the node. A node may have as many devices specified as necessary. In keeping with the security pattern of not trusting workers, devices are always registered through the swarmkit manager, never self-reported or self-discovered.\r\n\r\nTask Specs will include a list of device classes and options desired, including where in the task’s file system to place the device. Tasks must be prepared to accept any device in the class as equivalent. When a task is created, it will have the full run-time device parameter included in the object.\r\n\r\n## User Interface\r\n\r\nAdding devices will introduce a new command and subcommands for the management of devices. The first command, and the biggest change, will be to add new subcommands to manage device classes:\r\n\r\n```\r\nUsage: docker swarm device COMMAND\r\n\r\nManage Swarm devices\r\n\r\nCommands:\r\n  add      Add a new device class to the swarm\r\n  ls       List device classes on this swarm\r\n  inspect  Show information about a device class and its devices\r\n  rm       Remove a device class from the swarm\r\n```\r\n\r\nThe add command adds a new device to the the swarm:\r\n\r\n```\r\nUsage: docker swarm device add [OPTIONS] CLASS\r\n\r\nAdd a new device class to the swarm\r\n\r\nOptions:\r\n     --shared      Allow this class to be shared between tasks\r\n     --label list  Set metadata on this device class\r\n```\r\n\r\nThe ls command will allow listing all available device classes\r\n\r\n```\r\nUsage: docker swarm device ls [OPTIONS]\r\n\r\nList device classes on this swarm\r\n\r\nOptions:\r\n  -q, --quiet   Only display IDs\r\n  -f, --filter  Filter output based on conditions provided\r\n```\r\n\r\nThe inspect command will allow showing full information about device classes, as well as allowing the user to include all devices currently registered belonging to a device class.\r\n\r\n```\r\nUsage: docker swarm device inspect [OPTIONS] CLASS [CLASS]\r\n\r\nDisplay detailed information one one or more device classes\r\n\r\nOptions:\r\n  -f, --format string   Format the output\r\n      --pretty          Print the information human friendly\r\n      --devices         Include devices belonging to this class\r\n```\r\n\r\nThe remove command is similar to all other rm commands, and its usage is obvious, with the caveat that removal of a device class will be disallowed if a device is in use by task. There is no update command, as device classes will not be treated as updateable.\r\n\r\nTo manage particular devices on nodes, the existing node update command will receive new flags:\r\n\r\n```\r\n--device-add device  Register a device on a node with the swarm\r\n--device-rm device   Deregister a device with the swarm\r\n```\r\n\r\nSimilar to other options like ports and volumes, devices will accept both short- and long-form versions.\r\n\r\nThe short form will take the format target:class, where path is the path of device on the host, and class is the device class to register with. as such\r\n\r\n```\r\n--device-add /dev/nvidia0:gpu\r\n```\r\n\r\nThe long form of the command allows specifying these options independently, and allows future expansion of options for devices (such as host-specific cgroup options):\r\n\r\n```\r\n--device-add target=”/dev/nvidia0”,class=”gpu”\r\n```\r\n\r\nThe device rm option for node update acts as expected, but will disallow removing a device that is in use.\r\n\r\nServices would also support new flags. Service create will have a new option, --device, with both a long form and a short form. The short form will be reciprocal of the the --device flag on the node, taking the form class:path. It will also optionally support a third rwm field, mirroring the --device flag on docker run. The long form will take discrete arguments, and allow the user to specify cgroup options as supported in th\r\n\r\nThe short form, for mounting a GPU:\r\n\r\n```\r\n--device gpu:/dev/nvidia0\r\n```\r\n\r\nServices would also support a long form of the command:\r\n\r\n```\r\n--device class=”gpu”,path=”/dev/nvidia0”\r\n```\r\n\r\nNote: the long form of the command could possibly support further cgroup options, as allowed in the docker REST API for container creation.\r\n\r\nService update would include `--device-add` and `--device-rm` flags. `--device-add` syntax will be equivalent to the `--device` flag of create.  Because a task may have more than one device of a class mounted into its running container, `--device-rm` would require both the class and path of the device to disambiguate the specific device that is to be removed.\r\n\r\n```\r\n--device-rm class=”gpu”,path=”/dev/nvidia0”\r\n```\r\n\r\n## REST API\r\nThe Docker engine REST API would require a new set of endpoint to accommodate the concept of device classes. These endpoints would return the JSON representation of the objects described in the example Protocol Buffers. These endpoints would be as follows:\r\n\r\n```\r\nGET    /devices             List device classes\r\nPOST   /devices/create      Create a new device class\r\nGET    /devices/{id}        Inspect a device class\r\nPOST   /devices/{id}/update Update a device class\r\nDELETE /devices/{id}        Delete a device class\r\n```\r\n\r\n## Protocol Buffers\r\nIn swarm, protocol buffers define the internal API and object structure.\r\n\r\nThe `DeviceClass` proto will form a new top-level type, like a `Network` or a `Service`. It will have an ID and a name.\r\n\r\n```proto\r\n// DeviceClass is a specification for a particular device, zero or more of\r\n// which may be available on the cluster. It refers to the general class of\r\n// devices that the user wishes to be assumed as interchangeably usable. For\r\n// example, a cluster may have many possible block devices on many nodes, but\r\n// any of them are valid. The specific implementation of a specific device on a\r\n// node is provided by the node. A particular device may only belong to one\r\n// device class.\r\nmessage DeviceClass {\r\n  string id = 1;\r\n\r\n  Meta meta = 2 [(gogoproto.nullable = false];\r\n\r\n  // Shared represents whether this device can be shared between many tasks, or\r\n  // whether it should be uniquely mapped to a particular task. Shared devices\r\n  // may have any number of tasks assigned to them.\r\n  //\r\n  // Note that Shared has strong security risks; shared devices may be used by\r\n  // tasks to communicate with one another.\r\n  bool shared = 3;\r\n}\r\n```\r\n\r\nThe `Device` proto is included as a repeated field on `Node` specs. It defines a particular available device belonging to a class.\r\n\r\n```proto\r\n// Device represents a particular available device on a node. It is one\r\n// particular instance of a DeviceClass, and is interchangeable with other\r\n// devices in the DeviceClass\r\nmessage Device {\r\n  // DeviceClassID is the ID of the device class that this device belongs to.\r\n  string device_class_id = 1;\r\n\r\n  // PathOnHost is the path in the host's filesystem that this device should be\r\n  // mounted from.  For example, a block device may have this value as\r\n  // \"/dev/sda\". A particular device may belong to only 1 device class;\r\n  // assigning a device to more than one class may cause it to be conflictingly\r\n  // scheduled.\r\n  string path_on_host = 2;\r\n}\r\n```\r\n\r\nThe DeviceAttachmentSpec is a repeated field found in the TaskSpec proto, and defines the devices that a task should be attached to.\r\n\r\n```proto\r\n// DeviceAttachmentSpec represents the spec for a device attachment\r\nmessage DeviceAttachmentSpec {\r\n  // DeviceClass is the ID or name of the device class that is to be used for\r\n  // this spec. The actual device may be any device of this class on any node.\r\n  string device_class = 1;\r\n\r\n  // Path represents the path in the task's filesystem that this device should\r\n  // be mounted at.\r\n  string path = 2;\r\n \r\n  // DeviceCgroupRules represents the cgroup rules that should be applied to\r\n  // this device.\r\n  repeated string device_cgroup_rules = 16;\r\n}\r\n```\r\n\r\nThe DeviceAttachment is a repeated field on Tasks which defines specifically the run-time parameters of a device attachment for a particular task.\r\n\r\n\r\n```proto\r\n// DeviceAttachment represents the run-time configuration of a device in use.\r\n// It includes both the path on the host and the path in the Task of the\r\n// device, because a Task may have many devices of the same class reserved, and\r\n// those reservations would be otherwise indistinguishable.\r\nmessage DeviceAttachment {\r\n  // DeviceClassID is the ID of the device class used for this device\r\n  string device_class_id = 1;\r\n\r\n  // PathOnHost is the path on the host's filesystem of the device to be used\r\n  // by the task.\r\n  string path_on_host = 2;\r\n\r\n  // PathInTask is the path in the task's filesystem that the device will be\r\n  // mounted at.\r\n  string path_in_task = 3;\r\n\r\n  // DeviceCgroupRules represents the Cgroup rules that should be applied to\r\n  // this device.\r\n  repeated string device_cgroup_rules = 16;\r\n}\r\n```\r\n\r\n## Swarmkit Implementation\r\nThe device allocator will be implemented as a sub-component of the Scheduler. It will be created when a scheduler is created, and keep track of the available devices in the cluster. Scheduling for available devices forms part of the constraint-solving portion of the scheduler.\r\n\r\nTask updates present a difficulty for devices. If devices in the class can be shared between tasks (marked `--shared`), then there is not problem. However, the start-first update strategy would fail if there were not at least one device in a class available, such that the new task could start with a fresh device, allowing the old task to shut down and free its in-use device. There is no easy solution for this, I think. We should instead document thoroughly that using `start-first` with devices may cause trouble.\r\n\r\n## Error Handling\r\n\r\nBecause of the nature of distrusting the workers, it is difficult or impossible for swarm to “prove” that a given device exists on a node, or performs as the user expects. Swarm will therefore make no attempts to verify the correctness of provided user data. If a device is mistakenly assigned to the wrong class, or if it does not exist at all, the task is expected to fail to start. It should enter a terminal state of FAILED and should include an error message explaining that the errant device is at fault.\r\n\r\nNotably, in this proposal, there will be no attempt to “downweight” or otherwise attempt to avoid a node with a failing device. This functionality may come later, but not as part of this proposal.\r\n\r\n## Security\r\n\r\nIt must be understood that once on the host, swarmkit has no control over how a task uses devices. If improperly used, devices can be an extreme security hole for swarm tasks. For example, mounting block devices may allow read or write access to all of their contents. If the host’s primary block device were mounted into a task, that task could have full access to the host filesystem.\r\n"},"comment":{"url":"https://api.github.com/repos/docker/swarmkit/issues/comments/403638939","html_url":"https://github.com/docker/swarmkit/issues/2682#issuecomment-403638939","issue_url":"https://api.github.com/repos/docker/swarmkit/issues/2682","id":403638939,"node_id":"MDEyOklzc3VlQ29tbWVudDQwMzYzODkzOQ==","user":{"login":"dperny","id":2367858,"node_id":"MDQ6VXNlcjIzNjc4NTg=","avatar_url":"https://avatars3.githubusercontent.com/u/2367858?v=4","gravatar_id":"","url":"https://api.github.com/users/dperny","html_url":"https://github.com/dperny","followers_url":"https://api.github.com/users/dperny/followers","following_url":"https://api.github.com/users/dperny/following{/other_user}","gists_url":"https://api.github.com/users/dperny/gists{/gist_id}","starred_url":"https://api.github.com/users/dperny/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dperny/subscriptions","organizations_url":"https://api.github.com/users/dperny/orgs","repos_url":"https://api.github.com/users/dperny/repos","events_url":"https://api.github.com/users/dperny/events{/privacy}","received_events_url":"https://api.github.com/users/dperny/received_events","type":"User","site_admin":false},"created_at":"2018-07-09T22:21:41Z","updated_at":"2018-07-09T22:21:41Z","author_association":"MEMBER","body":"I'm poking at how to leverage the existing GenericResource code, and it's honestly not that sensible. The use case is too different. The amount of mogrification to the GenericResource concept that one would have to do is untenable.\r\n\r\nHonestly... GenericResource isn't a super sensible implementation anyway. It totally decouples a task's resource demands from the actual use of resources, which is a serious problem. If a Task reserves a resource, but does not have any way to use it, the resource is wasted. However, if a Task specifies how to use a resource, but no such reservation was made, then the Task will fail in strange ways.\r\n\r\nI think, despite the slight duplication of efforts, the use case for _actually using_ devices is sufficiently different to warrant a separate design."}}
{"actor":{"display_login":"dperny"}}