{"action":"created","issue":{"url":"https://api.github.com/repos/docker/docker/issues/25608","repository_url":"https://api.github.com/repos/docker/docker","labels_url":"https://api.github.com/repos/docker/docker/issues/25608/labels{/name}","comments_url":"https://api.github.com/repos/docker/docker/issues/25608/comments","events_url":"https://api.github.com/repos/docker/docker/issues/25608/events","html_url":"https://github.com/docker/docker/issues/25608","id":170609120,"number":25608,"title":"Native Swarm in 1.12 - panic: runtime error: index out of range","user":{"login":"b0ch3nski","id":1633387,"avatar_url":"https://avatars.githubusercontent.com/u/1633387?v=3","gravatar_id":"","url":"https://api.github.com/users/b0ch3nski","html_url":"https://github.com/b0ch3nski","followers_url":"https://api.github.com/users/b0ch3nski/followers","following_url":"https://api.github.com/users/b0ch3nski/following{/other_user}","gists_url":"https://api.github.com/users/b0ch3nski/gists{/gist_id}","starred_url":"https://api.github.com/users/b0ch3nski/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/b0ch3nski/subscriptions","organizations_url":"https://api.github.com/users/b0ch3nski/orgs","repos_url":"https://api.github.com/users/b0ch3nski/repos","events_url":"https://api.github.com/users/b0ch3nski/events{/privacy}","received_events_url":"https://api.github.com/users/b0ch3nski/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/docker/docker/labels/area/swarm","name":"area/swarm","color":"fbca04"},{"url":"https://api.github.com/repos/docker/docker/labels/group/networking","name":"group/networking","color":"F6FBFF"},{"url":"https://api.github.com/repos/docker/docker/labels/kind/bug","name":"kind/bug","color":"FF8C94"},{"url":"https://api.github.com/repos/docker/docker/labels/priority/P1","name":"priority/P1","color":"eb6420"},{"url":"https://api.github.com/repos/docker/docker/labels/version/1.12","name":"version/1.12","color":"ededed"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/docker/docker/milestones/58","html_url":"https://github.com/docker/docker/milestones/1.12.2","labels_url":"https://api.github.com/repos/docker/docker/milestones/58/labels","id":1973095,"number":58,"title":"1.12.2","description":"","creator":{"login":"tiborvass","id":827131,"avatar_url":"https://avatars.githubusercontent.com/u/827131?v=3","gravatar_id":"","url":"https://api.github.com/users/tiborvass","html_url":"https://github.com/tiborvass","followers_url":"https://api.github.com/users/tiborvass/followers","following_url":"https://api.github.com/users/tiborvass/following{/other_user}","gists_url":"https://api.github.com/users/tiborvass/gists{/gist_id}","starred_url":"https://api.github.com/users/tiborvass/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tiborvass/subscriptions","organizations_url":"https://api.github.com/users/tiborvass/orgs","repos_url":"https://api.github.com/users/tiborvass/repos","events_url":"https://api.github.com/users/tiborvass/events{/privacy}","received_events_url":"https://api.github.com/users/tiborvass/received_events","type":"User","site_admin":false},"open_issues":36,"closed_issues":2,"state":"open","created_at":"2016-08-30T16:26:40Z","updated_at":"2016-08-31T20:06:07Z","due_on":null,"closed_at":null},"comments":8,"created_at":"2016-08-11T09:56:45Z","updated_at":"2016-08-31T21:02:38Z","closed_at":null,"body":"**Output of `docker version`:**\r\n\r\n```\r\nClient:\r\n Version:      1.12.0\r\n API version:  1.24\r\n Go version:   go1.6.3\r\n Git commit:   8eab29e\r\n Built:        \r\n OS/Arch:      linux/amd64\r\n\r\nServer:\r\n Version:      1.12.0\r\n API version:  1.24\r\n Go version:   go1.6.3\r\n Git commit:   8eab29e\r\n Built:        \r\n OS/Arch:      linux/amd64\r\n```\r\n\r\n\r\n**Output of `docker info`:**\r\n\r\n```\r\nContainers: 1\r\n Running: 0\r\n Paused: 0\r\n Stopped: 1\r\nImages: 2\r\nServer Version: 1.12.0\r\nStorage Driver: devicemapper\r\n Pool Name: docker-253:0-1970762-pool\r\n Pool Blocksize: 65.54 kB\r\n Base Device Size: 10.74 GB\r\n Backing Filesystem: xfs\r\n Data file: /dev/loop0\r\n Metadata file: /dev/loop1\r\n Data Space Used: 10.46 GB\r\n Data Space Total: 107.4 GB\r\n Data Space Available: 28.07 GB\r\n Metadata Space Used: 6.238 MB\r\n Metadata Space Total: 2.147 GB\r\n Metadata Space Available: 2.141 GB\r\n Thin Pool Minimum Free Space: 10.74 GB\r\n Udev Sync Supported: true\r\n Deferred Removal Enabled: false\r\n Deferred Deletion Enabled: false\r\n Deferred Deleted Device Count: 0\r\n Data loop file: /mnt/docker/devicemapper/devicemapper/data\r\n WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.\r\n Metadata loop file: /mnt/docker/devicemapper/devicemapper/metadata\r\n Library Version: 1.02.107-RHEL7 (2016-06-09)\r\nLogging Driver: json-file\r\nCgroup Driver: cgroupfs\r\nPlugins:\r\n Volume: local\r\n Network: overlay bridge null host\r\nSwarm: active\r\n NodeID: 2vyiapx1ngzr5s0lj87x3nqxd\r\n Is Manager: false\r\n Node Address: 172.28.128.18\r\nRuntimes: runc\r\nDefault Runtime: runc\r\nSecurity Options: seccomp\r\nKernel Version: 3.10.0-327.22.2.el7.x86_64\r\nOperating System: CentOS Linux 7 (Core)\r\nOSType: linux\r\nArchitecture: x86_64\r\nCPUs: 1\r\nTotal Memory: 993 MiB\r\nName: dt-dev-2\r\nID: 7IDM:SHKD:MMTJ:USU6:U7BD:CZAK:P3QO:IXFM:UCM3:S2LB:OIJP:GW6P\r\nDocker Root Dir: /mnt/docker\r\nDebug Mode (client): false\r\nDebug Mode (server): false\r\nRegistry: https://index.docker.io/v1/\r\nWARNING: bridge-nf-call-ip6tables is disabled\r\nInsecure Registries:\r\n 127.0.0.0/8\r\n```\r\n\r\n\r\n**Additional environment details (AWS, VirtualBox, physical, etc.):**\r\n\r\nCentOS7 @ VirtualBox (via Vagrant)\r\n\r\n\r\n**Additional information:**\r\n\r\nI had Swarm setup with 2 nodes (and 2 different services started). I've restarted one of nodes and I was surprised that now the restarted node hosts both services (they were auto-balanced before).\r\nTurns out that Docker at machine that was NOT restarted died - this is the stacktrace:\r\n\r\n```\r\nAug 11 06:13:28 dt-dev-2 dockerd[17100]: time=\"2016-08-11T06:13:28.138941967Z\" level=warning msg=\"cannot find proper key indices while processing key upda\r\nAug 11 08:31:24 dt-dev-2 dockerd[17100]: time=\"2016-08-11T08:31:24.552409257Z\" level=error msg=\"agent: session failed\" error=\"rpc error: code = 13 desc = \r\nAug 11 08:31:24 dt-dev-2 dockerd[17100]: time=\"2016-08-11T08:31:24Z\" level=info msg=\"Firewalld running: false\"\r\nAug 11 08:31:24 dt-dev-2 dockerd[17100]: time=\"2016-08-11T08:31:24Z\" level=info msg=\"Firewalld running: false\"\r\nAug 11 08:31:27 dt-dev-2 dockerd[17100]: time=\"2016-08-11T08:31:27.274591339Z\" level=info msg=\"2016/08/11 08:31:27 [INFO] memberlist: Suspect dt-dev-1 has\r\nAug 11 08:31:29 dt-dev-2 dockerd[17100]: time=\"2016-08-11T08:31:29.274701535Z\" level=info msg=\"2016/08/11 08:31:29 [INFO] memberlist: Suspect dt-dev-1 has\r\nAug 11 08:31:29 dt-dev-2 dockerd[17100]: time=\"2016-08-11T08:31:29.274859714Z\" level=error msg=\"periodic bulk sync failure for network 1ms79jc1lu6z29l9ra0\r\nAug 11 08:31:29 dt-dev-2 dockerd[17100]: time=\"2016-08-11T08:31:29.625202513Z\" level=error msg=\"agent: session failed\" error=\"session initiation timed out\r\nAug 11 08:31:31 dt-dev-2 dockerd[17100]: time=\"2016-08-11T08:31:31.274195993Z\" level=info msg=\"2016/08/11 08:31:31 [INFO] memberlist: Suspect dt-dev-1 has\r\nAug 11 08:31:32 dt-dev-2 dockerd[17100]: time=\"2016-08-11T08:31:32.274488016Z\" level=warning msg=\"cannot find proper key indices while processing key upda\r\nAug 11 08:31:32 dt-dev-2 dockerd[17100]: time=\"2016-08-11T08:31:32.275195889Z\" level=info msg=\"2016/08/11 08:31:32 [INFO] memberlist: Suspect dt-dev-1 has\r\nAug 11 08:31:32 dt-dev-2 dockerd[17100]: time=\"2016-08-11T08:31:32.275233408Z\" level=info msg=\"2016/08/11 08:31:32 [INFO] memberlist: Marking dt-dev-1 as \r\nAug 11 08:31:32 dt-dev-2 dockerd[17100]: time=\"2016-08-11T08:31:32.286804614Z\" level=error msg=\"container status unavailable\" error=\"context canceled\" mod\r\nAug 11 08:43:35 dt-dev-2 dockerd[17100]: time=\"2016-08-11T08:43:35.005839511Z\" level=error msg=\"agent: session failed\" error=\"rpc error: code = 13 desc = \r\nAug 11 08:43:40 dt-dev-2 dockerd[17100]: time=\"2016-08-11T08:43:40.990620593Z\" level=error msg=\"agent: session failed\" error=\"session initiation timed out\r\nAug 11 08:43:42 dt-dev-2 dockerd[17100]: time=\"2016-08-11T08:43:42.520821682Z\" level=warning msg=\"cannot find proper key indices while processing key upda\r\nAug 11 08:43:42 dt-dev-2 dockerd[17100]: time=\"2016-08-11T08:43:42.555826038Z\" level=error msg=\"container status unavailable\" error=\"context canceled\" mod\r\nAug 11 08:47:25 dt-dev-2 dockerd[17100]: time=\"2016-08-11T08:47:25.336033855Z\" level=error msg=\"agent: session failed\" error=\"rpc error: code = 13 desc = \r\nAug 11 08:47:31 dt-dev-2 dockerd[17100]: time=\"2016-08-11T08:47:31.229495064Z\" level=error msg=\"agent: session failed\" error=\"session initiation timed out\r\nAug 11 08:47:32 dt-dev-2 dockerd[17100]: panic: runtime error: index out of range\r\nAug 11 08:47:32 dt-dev-2 dockerd[17100]: goroutine 870 [running]:\r\nAug 11 08:47:32 dt-dev-2 dockerd[17100]: panic(0x1a6fa00, 0xc82000e020)\r\nAug 11 08:47:32 dt-dev-2 dockerd[17100]: /usr/local/go/src/runtime/panic.go:481 +0x3e6\r\nAug 11 08:47:32 dt-dev-2 dockerd[17100]: github.com/docker/libnetwork/drivers/overlay.updateNodeKey(0xc820c300f0, 0x10, 0x10, 0xc820c30100, 0x10, 0x10, 0x\r\nAug 11 08:47:32 dt-dev-2 dockerd[17100]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/docker/libnetwork/drivers/overlay/encryption.go:496 +0x1\r\nAug 11 08:47:32 dt-dev-2 dockerd[17100]: github.com/docker/libnetwork/drivers/overlay.(*driver).updateKeys.func1(0xc820970e20, 0xd, 0xc820be9400, 0x3, 0x3\r\nAug 11 08:47:32 dt-dev-2 dockerd[17100]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/docker/libnetwork/drivers/overlay/encryption.go:444 +0x1\r\nAug 11 08:47:32 dt-dev-2 dockerd[17100]: github.com/docker/libnetwork/drivers/overlay.(*driver).secMapWalk(0xc820082e00, 0xc821b9cfc0, 0x0, 0x0)\r\nAug 11 08:47:32 dt-dev-2 dockerd[17100]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/docker/libnetwork/drivers/overlay/encryption.go:382 +0x1\r\nAug 11 08:47:32 dt-dev-2 dockerd[17100]: github.com/docker/libnetwork/drivers/overlay.(*driver).updateKeys(0xc820082e00, 0xc820aba1e0, 0xc820aba200, 0xc82\r\nAug 11 08:47:32 dt-dev-2 dockerd[17100]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/docker/libnetwork/drivers/overlay/encryption.go:445 +0x8\r\nAug 11 08:47:32 dt-dev-2 dockerd[17100]: github.com/docker/libnetwork/drivers/overlay.(*driver).DiscoverNew(0xc820082e00, 0x4, 0x1b9a4e0, 0xc821442c00, 0x\r\nAug 11 08:47:32 dt-dev-2 dockerd[17100]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/docker/libnetwork/drivers/overlay/overlay.go:351 +0xbbc\r\nAug 11 08:47:32 dt-dev-2 dockerd[17100]: github.com/docker/libnetwork.(*controller).handleKeyChange.func1(0x1d53098, 0x7, 0x7f7542cd07e8, 0xc820082e00, 0x\r\nAug 11 08:47:32 dt-dev-2 dockerd[17100]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/docker/libnetwork/agent.go:151 +0x94\r\nAug 11 08:47:32 dt-dev-2 dockerd[17100]: github.com/docker/libnetwork/drvregistry.(*DrvRegistry).WalkDrivers(0xc8201cf1d0, 0xc821b9d6e0)\r\nAug 11 08:47:32 dt-dev-2 dockerd[17100]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/docker/libnetwork/drvregistry/drvregistry.go:107 +0x2ce\r\nAug 11 08:47:32 dt-dev-2 dockerd[17100]: github.com/docker/libnetwork.(*controller).handleKeyChange(0xc8203f43c0, 0xc8210b9840, 0x6, 0x8, 0x0, 0x0)\r\nAug 11 08:47:32 dt-dev-2 dockerd[17100]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/docker/libnetwork/agent.go:156 +0x836\r\nAug 11 08:47:32 dt-dev-2 dockerd[17100]: github.com/docker/libnetwork.(*controller).SetKeys(0xc8203f43c0, 0xc8210b9840, 0x6, 0x8, 0x0, 0x0)\r\nAug 11 08:47:32 dt-dev-2 dockerd[17100]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/docker/libnetwork/controller.go:275 +0x35f\r\nAug 11 08:47:32 dt-dev-2 dockerd[17100]: github.com/docker/docker/daemon.(*Daemon).SetNetworkBootstrapKeys(0xc8203656c0, 0xc8210b9840, 0x6, 0x8, 0x0, 0x0)\r\nAug 11 08:47:32 dt-dev-2 dockerd[17100]: /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/daemon/network.go:185 +0x6d\r\nAug 11 08:47:32 dt-dev-2 dockerd[17100]: github.com/docker/docker/daemon/cluster/executor/container.(*executor).SetNetworkBootstrapKeys(0xc820c10fb0, 0xc8\r\nAug 11 08:47:32 dt-dev-2 dockerd[17100]: /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/daemon/cluster/executor/container/executo\r\nAug 11 08:47:32 dt-dev-2 dockerd[17100]: github.com/docker/swarmkit/agent.(*Agent).handleSessionMessage(0xc8209c6850, 0x7f7542cc6a98, 0xc820bb9dd0, 0xc820\r\nAug 11 08:47:32 dt-dev-2 dockerd[17100]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/docker/swarmkit/agent/agent.go:272 +0x5ca\r\nAug 11 08:47:32 dt-dev-2 dockerd[17100]: github.com/docker/swarmkit/agent.(*Agent).run(0xc8209c6850, 0x7f7542cc6a98, 0xc820bb9dd0)\r\nAug 11 08:47:32 dt-dev-2 dockerd[17100]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/docker/swarmkit/agent/agent.go:170 +0xbe6\r\nAug 11 08:47:32 dt-dev-2 dockerd[17100]: created by github.com/docker/swarmkit/agent.(*Agent).Start\r\nAug 11 08:47:32 dt-dev-2 dockerd[17100]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/docker/swarmkit/agent/agent.go:81 +0x37d\r\nAug 11 08:47:32 dt-dev-2 systemd[1]: docker.service: main process exited, code=exited, status=2/INVALIDARGUMENT\r\nAug 11 08:47:32 dt-dev-2 systemd[1]: Unit docker.service entered failed state.\r\nAug 11 08:47:32 dt-dev-2 systemd[1]: docker.service failed.\r\n```"},"comment":{"url":"https://api.github.com/repos/docker/docker/issues/comments/243900663","html_url":"https://github.com/docker/docker/issues/25608#issuecomment-243900663","issue_url":"https://api.github.com/repos/docker/docker/issues/25608","id":243900663,"user":{"login":"aboch","id":10080882,"avatar_url":"https://avatars.githubusercontent.com/u/10080882?v=3","gravatar_id":"","url":"https://api.github.com/users/aboch","html_url":"https://github.com/aboch","followers_url":"https://api.github.com/users/aboch/followers","following_url":"https://api.github.com/users/aboch/following{/other_user}","gists_url":"https://api.github.com/users/aboch/gists{/gist_id}","starred_url":"https://api.github.com/users/aboch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aboch/subscriptions","organizations_url":"https://api.github.com/users/aboch/orgs","repos_url":"https://api.github.com/users/aboch/repos","events_url":"https://api.github.com/users/aboch/events{/privacy}","received_events_url":"https://api.github.com/users/aboch/received_events","type":"User","site_admin":false},"created_at":"2016-08-31T21:02:38Z","updated_at":"2016-08-31T21:02:38Z","body":"@b0ch3nski \r\nI tried multiple times today with the debug logs enabled and was not able to reproduce.\r\nBut eventually I hit the issue when I was not expecting it, I had no debug logs enabled.\r\nI am testing with latest code with a change in keymanager which increases the key rotation frequency.\r\nATM, I still don't know how to trigger the issue."}}
{"actor":{"display_login":"aboch"}}