{"action":"created","issue":{"url":"https://api.github.com/repos/docker/docker/issues/26356","repository_url":"https://api.github.com/repos/docker/docker","labels_url":"https://api.github.com/repos/docker/docker/issues/26356/labels{/name}","comments_url":"https://api.github.com/repos/docker/docker/issues/26356/comments","events_url":"https://api.github.com/repos/docker/docker/issues/26356/events","html_url":"https://github.com/docker/docker/issues/26356","id":175394769,"number":26356,"title":"dockerd: panic: runtime error: invalid memory address or nil pointer dereference","user":{"login":"JamesJJ","id":813929,"avatar_url":"https://avatars.githubusercontent.com/u/813929?v=3","gravatar_id":"","url":"https://api.github.com/users/JamesJJ","html_url":"https://github.com/JamesJJ","followers_url":"https://api.github.com/users/JamesJJ/followers","following_url":"https://api.github.com/users/JamesJJ/following{/other_user}","gists_url":"https://api.github.com/users/JamesJJ/gists{/gist_id}","starred_url":"https://api.github.com/users/JamesJJ/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JamesJJ/subscriptions","organizations_url":"https://api.github.com/users/JamesJJ/orgs","repos_url":"https://api.github.com/users/JamesJJ/repos","events_url":"https://api.github.com/users/JamesJJ/events{/privacy}","received_events_url":"https://api.github.com/users/JamesJJ/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/docker/docker/labels/area/networking","name":"area/networking","color":"fbca04"},{"url":"https://api.github.com/repos/docker/docker/labels/kind/bug","name":"kind/bug","color":"FF8C94"},{"url":"https://api.github.com/repos/docker/docker/labels/priority/P1","name":"priority/P1","color":"eb6420"},{"url":"https://api.github.com/repos/docker/docker/labels/version/1.12","name":"version/1.12","color":"ededed"}],"state":"open","locked":false,"assignee":{"login":"sanimej","id":10071849,"avatar_url":"https://avatars.githubusercontent.com/u/10071849?v=3","gravatar_id":"","url":"https://api.github.com/users/sanimej","html_url":"https://github.com/sanimej","followers_url":"https://api.github.com/users/sanimej/followers","following_url":"https://api.github.com/users/sanimej/following{/other_user}","gists_url":"https://api.github.com/users/sanimej/gists{/gist_id}","starred_url":"https://api.github.com/users/sanimej/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sanimej/subscriptions","organizations_url":"https://api.github.com/users/sanimej/orgs","repos_url":"https://api.github.com/users/sanimej/repos","events_url":"https://api.github.com/users/sanimej/events{/privacy}","received_events_url":"https://api.github.com/users/sanimej/received_events","type":"User","site_admin":false},"assignees":[{"login":"sanimej","id":10071849,"avatar_url":"https://avatars.githubusercontent.com/u/10071849?v=3","gravatar_id":"","url":"https://api.github.com/users/sanimej","html_url":"https://github.com/sanimej","followers_url":"https://api.github.com/users/sanimej/followers","following_url":"https://api.github.com/users/sanimej/following{/other_user}","gists_url":"https://api.github.com/users/sanimej/gists{/gist_id}","starred_url":"https://api.github.com/users/sanimej/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sanimej/subscriptions","organizations_url":"https://api.github.com/users/sanimej/orgs","repos_url":"https://api.github.com/users/sanimej/repos","events_url":"https://api.github.com/users/sanimej/events{/privacy}","received_events_url":"https://api.github.com/users/sanimej/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/docker/docker/milestones/58","html_url":"https://github.com/docker/docker/milestones/1.12.2","labels_url":"https://api.github.com/repos/docker/docker/milestones/58/labels","id":1973095,"number":58,"title":"1.12.2","description":"","creator":{"login":"tiborvass","id":827131,"avatar_url":"https://avatars.githubusercontent.com/u/827131?v=3","gravatar_id":"","url":"https://api.github.com/users/tiborvass","html_url":"https://github.com/tiborvass","followers_url":"https://api.github.com/users/tiborvass/followers","following_url":"https://api.github.com/users/tiborvass/following{/other_user}","gists_url":"https://api.github.com/users/tiborvass/gists{/gist_id}","starred_url":"https://api.github.com/users/tiborvass/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tiborvass/subscriptions","organizations_url":"https://api.github.com/users/tiborvass/orgs","repos_url":"https://api.github.com/users/tiborvass/repos","events_url":"https://api.github.com/users/tiborvass/events{/privacy}","received_events_url":"https://api.github.com/users/tiborvass/received_events","type":"User","site_admin":false},"open_issues":18,"closed_issues":46,"state":"open","created_at":"2016-08-30T16:26:40Z","updated_at":"2016-09-21T22:11:45Z","due_on":null,"closed_at":null},"comments":9,"created_at":"2016-09-07T01:56:50Z","updated_at":"2016-09-22T08:36:41Z","closed_at":null,"body":"\r\n**Output of `docker version`:**\r\n\r\n```\r\nClient:\r\n Version:      1.12.1\r\n API version:  1.24\r\n Go version:   go1.6.3\r\n Git commit:   23cf638\r\n Built:        \r\n OS/Arch:      linux/amd64\r\n\r\nServer:\r\n Version:      1.12.1\r\n API version:  1.24\r\n Go version:   go1.6.3\r\n Git commit:   23cf638\r\n Built:        \r\n OS/Arch:      linux/amd64\r\n```\r\n\r\n\r\n**Output of `docker info`:**\r\n\r\n*... after `systemctl restart  docker.service`*\r\n\r\n```\r\nContainers: 5\r\n Running: 5\r\n Paused: 0\r\n Stopped: 0\r\nImages: 8\r\nServer Version: 1.12.1\r\nStorage Driver: devicemapper\r\n Pool Name: docker_lvm_vg-docker_lvm\r\n Pool Blocksize: 524.3 kB\r\n Base Device Size: 8.59 GB\r\n Backing Filesystem: xfs\r\n Data file: \r\n Metadata file: \r\n Data Space Used: 4.733 GB\r\n Data Space Total: 52.94 GB\r\n Data Space Available: 48.2 GB\r\n Metadata Space Used: 1.913 MB\r\n Metadata Space Total: 209.7 MB\r\n Metadata Space Available: 207.8 MB\r\n Thin Pool Minimum Free Space: 3.705 GB\r\n Udev Sync Supported: true\r\n Deferred Removal Enabled: true\r\n Deferred Deletion Enabled: false\r\n Deferred Deleted Device Count: 0\r\n Library Version: 1.02.107-RHEL7 (2016-06-09)\r\nLogging Driver: json-file\r\nCgroup Driver: cgroupfs\r\nPlugins:\r\n Volume: local\r\n Network: overlay null bridge host\r\nSwarm: inactive\r\nRuntimes: runc\r\nDefault Runtime: runc\r\nSecurity Options: seccomp\r\nKernel Version: 3.10.0-327.28.3.el7.x86_64\r\nOperating System: CentOS Linux 7 (Core)\r\nOSType: linux\r\nArchitecture: x86_64\r\nCPUs: 2\r\nTotal Memory: 7.389 GiB\r\nName: ip-10-12-15-126.domain.name\r\nID: 62CV:56GM:HC7A:HS6X:PVII:5HMY:HQNW:Q3PL:7L3D:SRFF:EUDH:BPNX\r\nDocker Root Dir: /var/lib/docker\r\nDebug Mode (client): false\r\nDebug Mode (server): false\r\nRegistry: https://index.docker.io/v1/\r\nWARNING: bridge-nf-call-iptables is disabled\r\nWARNING: bridge-nf-call-ip6tables is disabled\r\nInsecure Registries:\r\n 127.0.0.0/8\r\n```\r\n\r\n**Additional environment details (AWS, VirtualBox, physical, etc.):**\r\n\r\nAWS / Centos 7, fully updated / Docker from Docker's Yum repo\r\n\r\n\r\n**Steps to reproduce the issue:**\r\n1. [unknown if relevant] Start some containers\r\n2. [unknown if relevant] One container includes `HEALTHCHECK ...` directive\r\n3. Wait for /time/ (in this case the machine and containers had been running for about 3 days)\r\n4. Docker and containers all dead. `/var/log/messages` shows below panic message\r\n\r\n\r\n**Describe the results you received:**\r\n\r\n*From /var/log/messages*\r\n\r\n```\r\nSep  6 20:48:54 ip-10-12-15-126 dockerd: time=\"2016-09-06T20:48:54.855732287Z\" level=warning msg=\"Health check error: rpc error: code = 2 desc = oci runtime error: exec failed: process_linux.go:75: starting setns process caused \\\"fork/exec /proc/self/exe: no such file or directory\\\"\"\r\nSep  6 20:48:58 ip-10-12-15-126 dockerd: time=\"2016-09-06T20:48:58.875034743Z\" level=warning msg=\"Health check error: rpc error: code = 2 desc = oci runtime error: exec failed: process_linux.go:75: starting setns process caused \\\"fork/exec /proc/self/exe: no such file or directory\\\"\"\r\nSep  6 20:49:02 ip-10-12-15-126 dockerd: time=\"2016-09-06T20:49:02.894231701Z\" level=warning msg=\"Health check error: rpc error: code = 2 desc = oci runtime error: exec failed: process_linux.go:75: starting setns process caused \\\"fork/exec /proc/self/exe: no such file or directory\\\"\"\r\nSep  6 20:49:06 ip-10-12-15-126 dockerd: time=\"2016-09-06T20:49:06.913581331Z\" level=warning msg=\"Health check error: rpc error: code = 2 desc = oci runtime error: exec failed: process_linux.go:75: starting setns process caused \\\"fork/exec /proc/self/exe: no such file or directory\\\"\"\r\nSep  6 20:49:10 ip-10-12-15-126 dockerd: time=\"2016-09-06T20:49:10.933518882Z\" level=warning msg=\"Health check error: rpc error: code = 2 desc = oci runtime error: exec failed: process_linux.go:75: starting setns process caused \\\"fork/exec /proc/self/exe: no such file or directory\\\"\"\r\nSep  6 20:49:14 ip-10-12-15-126 dockerd: time=\"2016-09-06T20:49:14.952690867Z\" level=warning msg=\"Health check error: rpc error: code = 2 desc = oci runtime error: exec failed: process_linux.go:75: starting setns process caused \\\"fork/exec /proc/self/exe: no such file or directory\\\"\"\r\nSep  6 20:49:18 ip-10-12-15-126 dockerd: time=\"2016-09-06T20:49:18.971882492Z\" level=warning msg=\"Health check error: rpc error: code = 2 desc = oci runtime error: exec failed: process_linux.go:75: starting setns process caused \\\"fork/exec /proc/self/exe: no such file or directory\\\"\"\r\nSep  6 20:49:19 ip-10-12-15-126 dockerd: panic: runtime error: invalid memory address or nil pointer dereference\r\nSep  6 20:49:19 ip-10-12-15-126 dockerd: [signal 0xb code=0x1 addr=0x28 pc=0x8b6fa0]\r\nSep  6 20:49:19 ip-10-12-15-126 dockerd: goroutine 583379 [running]:\r\nSep  6 20:49:19 ip-10-12-15-126 dockerd: panic(0x1a7edc0, 0xc82000a080)\r\nSep  6 20:49:19 ip-10-12-15-126 dockerd: /usr/local/go/src/runtime/panic.go:481 +0x3e6\r\nSep  6 20:49:19 ip-10-12-15-126 dockerd: github.com/docker/libnetwork.(*resolver).ServeDNS(0xc8200882d0, 0x7f7a522a7898, 0xc82649ea10, 0xc8a2b88360)\r\nSep  6 20:49:19 ip-10-12-15-126 dockerd: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/docker/libnetwork/resolver.go:363 +0x7b0\r\nSep  6 20:49:19 ip-10-12-15-126 dockerd: github.com/miekg/dns.(*Server).serve(0xc820e944e0, 0x7f7a522a3508, 0xc8adfb9050, 0x7f7a522a3580, 0xc8200882d0, 0xc8981df800, 0x23, 0x200, 0xc82001c0d0, 0xc8a76f6940, ...)\r\nSep  6 20:49:19 ip-10-12-15-126 dockerd: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/miekg/dns/server.go:535 +0x7c8\r\nSep  6 20:49:19 ip-10-12-15-126 dockerd: created by github.com/miekg/dns.(*Server).serveUDP\r\nSep  6 20:49:19 ip-10-12-15-126 dockerd: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/miekg/dns/server.go:489 +0x3d5\r\nSep  6 20:49:19 ip-10-12-15-126 systemd: docker.service: main process exited, code=exited, status=2/INVALIDARGUMENT\r\nSep  6 20:49:19 ip-10-12-15-126 systemd: Unit docker.service entered failed state.\r\nSep  6 20:49:19 ip-10-12-15-126 systemd: docker.service failed.\r\n\r\n```\r\n**Describe the results you expected:**\r\n\r\nNo panic / service uninterrupted \r\n\r\n\r\n**Additional information you deem important (e.g. issue happens only occasionally):**\r\n"},"comment":{"url":"https://api.github.com/repos/docker/docker/issues/comments/248843241","html_url":"https://github.com/docker/docker/issues/26356#issuecomment-248843241","issue_url":"https://api.github.com/repos/docker/docker/issues/26356","id":248843241,"user":{"login":"JamesJJ","id":813929,"avatar_url":"https://avatars.githubusercontent.com/u/813929?v=3","gravatar_id":"","url":"https://api.github.com/users/JamesJJ","html_url":"https://github.com/JamesJJ","followers_url":"https://api.github.com/users/JamesJJ/followers","following_url":"https://api.github.com/users/JamesJJ/following{/other_user}","gists_url":"https://api.github.com/users/JamesJJ/gists{/gist_id}","starred_url":"https://api.github.com/users/JamesJJ/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JamesJJ/subscriptions","organizations_url":"https://api.github.com/users/JamesJJ/orgs","repos_url":"https://api.github.com/users/JamesJJ/repos","events_url":"https://api.github.com/users/JamesJJ/events{/privacy}","received_events_url":"https://api.github.com/users/JamesJJ/received_events","type":"User","site_admin":false},"created_at":"2016-09-22T08:36:40Z","updated_at":"2016-09-22T08:36:40Z","body":"> What command do you have in the healthcheck ? Does it work normally ?\r\n\r\n`HEALTHCHECK --interval=4s CMD curl --max-time 3 --connect-timeout 2 -sSf http://127.0.0.1:9080/health || exit 1`\r\n\r\nIt works as expected: Usually starting-> healthy, at container start. Sometimes first check will fail if dependency container is not up yet, but that is only a brief condition (maybe 10 seconds). Only one of the containers has HEALTHCHECK ....  the others do not.\r\n\r\n> This check happens once, typically when the daemon comes up. So before this panic was there any daemon restart ?\r\n\r\nThe `Firewalld` message starts from damon first start [2 days before panic] and continues until panic ... full log:\r\n[panic-log.txt](https://github.com/docker/docker/files/486772/panic-log.txt)\r\n(PID does not change)\r\n\r\n> Do you have live-restore enabled\r\n\r\nNo\r\n\r\n> ... have containers with restart policies\r\n\r\nYes: All containers have `restart: always` in `docker-compose.yml`\r\n\r\n--\r\nMany thanks for your help."}}
{"actor":{"display_login":"JamesJJ"}}