{"action":"created","issue":{"url":"https://api.github.com/repos/moby/moby/issues/38321","repository_url":"https://api.github.com/repos/moby/moby","labels_url":"https://api.github.com/repos/moby/moby/issues/38321/labels{/name}","comments_url":"https://api.github.com/repos/moby/moby/issues/38321/comments","events_url":"https://api.github.com/repos/moby/moby/issues/38321/events","html_url":"https://github.com/moby/moby/issues/38321","id":387749139,"node_id":"MDU6SXNzdWUzODc3NDkxMzk=","number":38321,"title":"dispatcher-heartbeat option not persisted after docker update","user":{"login":"Zelphy","id":4152786,"node_id":"MDQ6VXNlcjQxNTI3ODY=","avatar_url":"https://avatars2.githubusercontent.com/u/4152786?v=4","gravatar_id":"","url":"https://api.github.com/users/Zelphy","html_url":"https://github.com/Zelphy","followers_url":"https://api.github.com/users/Zelphy/followers","following_url":"https://api.github.com/users/Zelphy/following{/other_user}","gists_url":"https://api.github.com/users/Zelphy/gists{/gist_id}","starred_url":"https://api.github.com/users/Zelphy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Zelphy/subscriptions","organizations_url":"https://api.github.com/users/Zelphy/orgs","repos_url":"https://api.github.com/users/Zelphy/repos","events_url":"https://api.github.com/users/Zelphy/events{/privacy}","received_events_url":"https://api.github.com/users/Zelphy/received_events","type":"User","site_admin":false},"labels":[{"id":392678049,"node_id":"MDU6TGFiZWwzOTI2NzgwNDk=","url":"https://api.github.com/repos/moby/moby/labels/area/swarm","name":"area/swarm","color":"fbca04","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-12-05T13:08:41Z","updated_at":"2018-12-27T12:06:49Z","closed_at":null,"author_association":"NONE","body":"**Description**\r\n\r\nThe dispatcher-heartbeat option does not seems to be persisted after docker update.\r\n\r\n**Steps to reproduce the issue:**\r\n\r\n1. Install a version of docker-ce on a Centos VM (not the latest, I've tried with 18.06.0)\r\n2. Create a swarm via `docker swarm init`\r\n3. Update the swarm dispatcher-heartbeat parameter via `docker swarm update --dispatcher-heartbeat 30s`\r\n4. Update Docker on this manager (`yum update docker-ce`)\r\n\r\n**Describe the results you received:**\r\n\r\n1. The heartbeat period is 5s after the update and not 30s\r\n2. The swarm dispatcher-heartbeat parameter cannot be updated to 30s anymore\r\n\r\n**Describe the results you expected:**\r\n\r\nThe heartbeat period is 30s after the update\r\n\r\n**Additional information you deem important (e.g. issue happens only occasionally):**\r\n\r\nI've also tried to initialize the swarm and set the heartbeat period at this step via `docker swarm init --dispatcher-heartbeat 30s` but we had the same behaviour.\r\n\r\nCurrently, after an update and as a workaround, we do:\r\n\r\n `docker swarm update --dispatcher-heartbeat Xs`\r\n `docker swarm update --dispatcher-heartbeat 30s`\r\n\r\nwith `X != 30`\r\n\r\n**Output of `docker version`:**\r\n\r\nBefore the update:\r\n\r\n```\r\nClient:\r\n Version:           18.06.0-ce\r\n API version:       1.38\r\n Go version:        go1.10.3\r\n Git commit:        0ffa825\r\n Built:             Wed Jul 18 19:08:18 2018\r\n OS/Arch:           linux/amd64\r\n Experimental:      false\r\n\r\nServer:\r\n Engine:\r\n  Version:          18.06.0-ce\r\n  API version:      1.38 (minimum version 1.12)\r\n  Go version:       go1.10.3\r\n  Git commit:       0ffa825\r\n  Built:            Wed Jul 18 19:10:42 2018\r\n  OS/Arch:          linux/amd64\r\n  Experimental:     false\r\n```\r\n\r\nAfter the update:\r\n\r\n```\r\nClient:\r\n Version:           18.09.1-beta2\r\n API version:       1.39\r\n Go version:        go1.10.5\r\n Git commit:        74239bf\r\n Built:             Tue Dec  4 01:08:56 2018\r\n OS/Arch:           linux/amd64\r\n Experimental:      false\r\n\r\nServer: Docker Engine - Community\r\n Engine:\r\n  Version:          18.09.1-beta2\r\n  API version:      1.39 (minimum version 1.12)\r\n  Go version:       go1.10.5\r\n  Git commit:       74239bf\r\n  Built:            Tue Dec  4 00:33:49 2018\r\n  OS/Arch:          linux/amd64\r\n  Experimental:     false\r\n```\r\n\r\n**Output of `docker info`:**\r\n\r\nBefore the update\r\n\r\n```\r\nContainers: 0\r\n Running: 0\r\n Paused: 0\r\n Stopped: 0\r\nImages: 0\r\nServer Version: 18.06.0-ce\r\nStorage Driver: overlay2\r\n Backing Filesystem: extfs\r\n Supports d_type: true\r\n Native Overlay Diff: true\r\nLogging Driver: json-file\r\nCgroup Driver: cgroupfs\r\nPlugins:\r\n Volume: local\r\n Network: bridge host macvlan null overlay\r\n Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog\r\nSwarm: inactive\r\nRuntimes: runc\r\nDefault Runtime: runc\r\nInit Binary: docker-init\r\ncontainerd version: d64c661f1d51c48782c9cec8fda7604785f93587\r\nrunc version: 69663f0bd4b60df09991c08812a60108003fa340\r\ninit version: fec3683\r\nSecurity Options:\r\n seccomp\r\n  Profile: default\r\nKernel Version: 4.19.6-1.el7.elrepo.x86_64\r\nOperating System: CentOS Linux 7 (Core)\r\nOSType: linux\r\nArchitecture: x86_64\r\nCPUs: 2\r\nTotal Memory: 1.923GiB\r\nName: temp1\r\nID: CRQF:VSWG:C7N2:CPFP:BJNW:JE3J:ORMX:WOA7:GMHK:IH7K:7IOG:IU6B\r\nDocker Root Dir: /var/lib/docker\r\nDebug Mode (client): false\r\nDebug Mode (server): true\r\n File Descriptors: 23\r\n Goroutines: 50\r\n System Time: 2018-12-05T11:03:38.456933195+01:00\r\n EventsListeners: 0\r\nUsername: nadisolutionsjenkins\r\nRegistry: https://index.docker.io/v1/\r\nLabels:\r\nExperimental: false\r\nInsecure Registries:\r\n 127.0.0.0/8\r\nLive Restore Enabled: false\r\n\r\nWARNING: bridge-nf-call-ip6tables is disabled\r\n```\r\n\r\nAfter the update:\r\n\r\n```\r\nContainers: 0\r\n Running: 0\r\n Paused: 0\r\n Stopped: 0\r\nImages: 0\r\nServer Version: 18.09.1-beta2\r\nStorage Driver: overlay2\r\n Backing Filesystem: extfs\r\n Supports d_type: true\r\n Native Overlay Diff: true\r\nLogging Driver: json-file\r\nCgroup Driver: cgroupfs\r\nPlugins:\r\n Volume: local\r\n Network: bridge host macvlan null overlay\r\n Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog\r\nSwarm: active\r\n NodeID: csu3w0hw0xayyr3yap5etfblb\r\n Is Manager: true\r\n ClusterID: jtm26tt1z4lz5zdty07zsc7xb\r\n Managers: 1\r\n Nodes: 1\r\n Default Address Pool: 10.0.0.0/8  \r\n SubnetSize: 24\r\n Orchestration:\r\n  Task History Retention Limit: 5\r\n Raft:\r\n  Snapshot Interval: 10000\r\n  Number of Old Snapshots to Retain: 0\r\n  Heartbeat Tick: 1\r\n  Election Tick: 10\r\n Dispatcher:\r\n  Heartbeat Period: 30 seconds\r\n CA Configuration:\r\n  Expiry Duration: 3 months\r\n  Force Rotate: 0\r\n Autolock Managers: false\r\n Root Rotation In Progress: false\r\n Node Address: 172.16.1.45\r\n Manager Addresses:\r\n  172.16.1.45:2377\r\nRuntimes: runc\r\nDefault Runtime: runc\r\nInit Binary: docker-init\r\ncontainerd version: de1f167ab96338a9f5c2b17347abf84bdf1dd411\r\nrunc version: 10d38b660a77168360df3522881e2dc2be5056bd\r\ninit version: fec3683\r\nSecurity Options:\r\n seccomp\r\n  Profile: default\r\nKernel Version: 4.19.6-1.el7.elrepo.x86_64\r\nOperating System: CentOS Linux 7 (Core)\r\nOSType: linux\r\nArchitecture: x86_64\r\nCPUs: 2\r\nTotal Memory: 1.923GiB\r\nName: temp1\r\nID: CRQF:VSWG:C7N2:CPFP:BJNW:JE3J:ORMX:WOA7:GMHK:IH7K:7IOG:IU6B\r\nDocker Root Dir: /var/lib/docker\r\nDebug Mode (client): false\r\nDebug Mode (server): true\r\n File Descriptors: 39\r\n Goroutines: 167\r\n System Time: 2018-12-05T11:16:12.924472222+01:00\r\n EventsListeners: 0\r\nUsername: nadisolutionsjenkins\r\nRegistry: https://index.docker.io/v1/\r\nLabels:\r\nExperimental: false\r\nInsecure Registries:\r\n 127.0.0.0/8\r\nLive Restore Enabled: false\r\nProduct License: Community Engine\r\n\r\nWARNING: bridge-nf-call-ip6tables is disabled\r\n```\r\n\r\n**Additional environment details (AWS, VirtualBox, physical, etc.):**\r\n\r\nOS: Centos 7\r\n\r\n```\r\ntemp1> uname -a\r\nLinux temp1 4.19.1-1.el7.elrepo.x86_64 #1 SMP Sun Nov 4 18:04:04 EST 2018 x86_64 x86_64 x86_64 GNU/Linux\r\n```\r\n\r\nThank you!"},"comment":{"url":"https://api.github.com/repos/moby/moby/issues/comments/450138988","html_url":"https://github.com/moby/moby/issues/38321#issuecomment-450138988","issue_url":"https://api.github.com/repos/moby/moby/issues/38321","id":450138988,"node_id":"MDEyOklzc3VlQ29tbWVudDQ1MDEzODk4OA==","user":{"login":"Zelphy","id":4152786,"node_id":"MDQ6VXNlcjQxNTI3ODY=","avatar_url":"https://avatars2.githubusercontent.com/u/4152786?v=4","gravatar_id":"","url":"https://api.github.com/users/Zelphy","html_url":"https://github.com/Zelphy","followers_url":"https://api.github.com/users/Zelphy/followers","following_url":"https://api.github.com/users/Zelphy/following{/other_user}","gists_url":"https://api.github.com/users/Zelphy/gists{/gist_id}","starred_url":"https://api.github.com/users/Zelphy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Zelphy/subscriptions","organizations_url":"https://api.github.com/users/Zelphy/orgs","repos_url":"https://api.github.com/users/Zelphy/repos","events_url":"https://api.github.com/users/Zelphy/events{/privacy}","received_events_url":"https://api.github.com/users/Zelphy/received_events","type":"User","site_admin":false},"created_at":"2018-12-27T12:06:49Z","updated_at":"2018-12-27T12:06:49Z","author_association":"NONE","body":"We are running Docker Swarm on multiple virtual machines created via VMware vSphere (cluster of servers).\r\nNetwork components are virtualized using [VMware NSX](https://docs.vmware.com/en/VMware-NSX-Data-Center-for-vSphere/6.3/com.vmware.nsx.install.doc/GUID-2482B032-F420-432F-A6D0-6CD91506BFCC.html)\r\n\r\nWhen there is changes in this infrastructure, like:\r\n\r\n- a Docker node is migrated from an host to another via vSphere live migration process\r\n- NSX component is updated and must reload his configuration\r\n- NSX component is live migrated from an host to another\r\n- other?\r\n\r\n... the swarm manager sometimes detects that the node is down, put the node in \"NODE DOWN\" state and start the process of finding another node to start the service.\r\n\r\nThen, the node is back up because the vSphere migration is done (or the network component configuration reload is done) so the node is UP again but the swarm manager was already informed that the node was down so services on the node are restarted.\r\n\r\nWe though that the threshold of `5s` for `--dispatcher-heartbeat` was a bit sensitive regarding our specific configuration (mainly virtualized network component and Docker nodes live migration) so we've increased it and made some tests. It seems to work great with `30s` for our use case.\r\n\r\nIn order to understand the behavior of the `--dispatcher-heartbeat` flag, we have done multiple tests with different values for this parameter.\r\n\r\n[dock_temp_20sec.txt](https://github.com/moby/moby/files/2712363/dock_temp_20sec.txt)\r\n*dispatcher-heartbeat* is set to 20s. We DOWN the node, wait 60sec, UP the node, wait 10s, then we redo it 5 times.\r\n\r\n[dock_temp_30sec.txt](https://github.com/moby/moby/files/2712364/dock_temp_30sec.txt)\r\n*dispatcher-heartbeat* is set to 30s. We DOWN the node, wait 90sec, UP the node, wait 10s, then we redo it 5 times.\r\n\r\n[dock_temp_60sec.txt](https://github.com/moby/moby/files/2712365/dock_temp_60sec.txt)\r\n*dispatcher-heartbeat* is set to 60s. We DOWN the node, wait 180sec, UP the node, wait 10s, then we redo it 5 times.\r\n\r\nFollowing the result at the end of each file and the [code](github.com/docker/swarmkit/blob/master/manager/dispatcher/dispatcher.go#L32), we think that Docker Swarm wait between `2*dispatcher-heartbeat` and `3*dispatcher-heartbeat` before setting a node DOWN.\r\n\r\nAfter the change, this gif should represents the behavior inside a container/on a node when a network problem happens (the timeout is increased here so the node is not set to DOWN, the ping can continue and the service is not restarted):\r\n\r\n![peek 2018-07-19 17-45](https://user-images.githubusercontent.com/4152786/50479544-66deb200-09d7-11e9-9f5c-b7649076fa94.gif)\r\n\r\n"}}
{"actor":{"display_login":"Zelphy"}}