{"action":"opened","issue":{"url":"https://api.github.com/repos/coreos/bugs/issues/1681","repository_url":"https://api.github.com/repos/coreos/bugs","labels_url":"https://api.github.com/repos/coreos/bugs/issues/1681/labels{/name}","comments_url":"https://api.github.com/repos/coreos/bugs/issues/1681/comments","events_url":"https://api.github.com/repos/coreos/bugs/issues/1681/events","html_url":"https://github.com/coreos/bugs/issues/1681","id":192225762,"number":1681,"title":"Docker daemon hangs","user":{"login":"pjknkda","id":4986524,"avatar_url":"https://avatars.githubusercontent.com/u/4986524?v=3","gravatar_id":"","url":"https://api.github.com/users/pjknkda","html_url":"https://github.com/pjknkda","followers_url":"https://api.github.com/users/pjknkda/followers","following_url":"https://api.github.com/users/pjknkda/following{/other_user}","gists_url":"https://api.github.com/users/pjknkda/gists{/gist_id}","starred_url":"https://api.github.com/users/pjknkda/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pjknkda/subscriptions","organizations_url":"https://api.github.com/users/pjknkda/orgs","repos_url":"https://api.github.com/users/pjknkda/repos","events_url":"https://api.github.com/users/pjknkda/events{/privacy}","received_events_url":"https://api.github.com/users/pjknkda/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2016-11-29T09:10:49Z","updated_at":"2016-11-29T09:10:49Z","closed_at":null,"body":"# Issue Report #\r\n\r\n## Bug ##\r\n\r\n### CoreOS Version ###\r\n\r\n```\r\n$ cat /etc/os-release\r\nNAME=CoreOS\r\nID=coreos\r\nVERSION=1235.0.0\r\nVERSION_ID=1235.0.0\r\nBUILD_ID=2016-11-17-0416\r\nPRETTY_NAME=\"CoreOS 1235.0.0 (MoreOS)\"\r\nANSI_COLOR=\"1;32\"\r\nHOME_URL=\"https://coreos.com/\"\r\nBUG_REPORT_URL=\"https://github.com/coreos/bugs/issues\"\r\n```\r\n\r\n### Environment ###\r\n\r\nKVM on CentOS 7 (3.10.0-327.36.3.el7.x86_64)\r\n\r\n### Expected Behavior ###\r\n\r\nDocker daemon should be responsible for commands (such as ps, run, stop, etc.).\r\n\r\n### Actual Behavior ###\r\n\r\nDocker daemon should not hang on commands.\r\n\r\n### Reproduction Steps ###\r\n\r\nI cannot find exact reproduction step yet (I'm making simplified version of my real service, but yet success to reproduce the issue). However, it occurs with 100% probability when I repeat creating and removing containers a lot (more than 2,000 containers and each container lives 1s ~ 120s).\r\n\r\n### Other Information ###\r\n\r\nStacktrace of dockerd : [sigusr1.txt](https://github.com/coreos/bugs/files/618230/sigusr1.txt)\r\n\r\nSystemd log,\r\n\r\n```\r\nNov 28 07:25:54 localhost docker[3564]: [Worker 0] Accpet request (Id: 2076410)\r\nNov 28 07:25:54 localhost docker[3564]: Get (Id: 2076410)\r\nNov 28 07:25:55 localhost docker[3564]: New container: aad1a33df311 (Elapsed: 46ms)\r\nNov 28 07:25:55 localhost systemd-udevd[14298]: Could not generate persistent MAC address for veth208af43: No such file or directory\r\nNov 28 07:25:55 localhost systemd-udevd[14299]: Could not generate persistent MAC address for veth34418f0: No such file or directory\r\nNov 28 07:25:55 localhost systemd-networkd[3188]: veth208af43: IPv6 enabled for interface: Success\r\nNov 28 07:25:55 localhost kernel: IPv6: ADDRCONF(NETDEV_UP): veth208af43: link is not ready\r\nov 28 07:25:55 localhost systemd-networkd[3188]: veth34418f0: IPv6 enabled for interface: Success\r\nNov 28 07:25:55 localhost kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth208af43: link becomes ready\r\nNov 28 07:25:55 localhost systemd-timesyncd[3123]: Network configuration changed, trying to establish connection.\r\nNov 28 07:25:55 localhost systemd-networkd[3188]: veth34418f0: Gained carrier\r\nNov 28 07:25:55 localhost systemd-networkd[3188]: veth208af43: Gained carrier\r\nNov 28 07:25:55 localhost kernel: docker0: port 3(veth34418f0) entered blocking state\r\nNov 28 07:25:55 localhost kernel: docker0: port 3(veth34418f0) entered disabled state\r\nNov 28 07:25:55 localhost kernel: device veth34418f0 entered promiscuous mode\r\nNov 28 07:25:55 localhost kernel: docker0: port 3(veth34418f0) entered blocking state\r\nNov 28 07:25:55 localhost kernel: docker0: port 3(veth34418f0) entered forwarding state\r\nNov 28 07:25:57 localhost systemd-networkd[3188]: veth208af43: Gained IPv6LL\r\nNov 28 07:25:57 localhost systemd-networkd[3188]: veth34418f0: Gained IPv6LL\r\n\r\nNov 28 07:26:06 localhost kernel: INFO: task bash:14181 blocked for more than 120 seconds.\r\nNov 28 07:26:06 localhost kernel:       Not tainted 4.8.6-coreos #1\r\nNov 28 07:26:06 localhost kernel: \"echo 0 > /proc/sys/kernel/hung_task_timeout_secs\" disables this message.\r\nNov 28 07:26:06 localhost kernel: bash            D ffff8b963e5a7c48     0 14181  14166 0x00000182\r\nNov 28 07:26:06 localhost kernel:  ffff8b963e5a7c48 0000000000000000 ffffffff96a0d540 ffff8b960ded1d40\r\nNov 28 07:26:06 localhost kernel:  ffffffff9607bb30 ffff8b963e5a8000 ffff8b960bd33420 0000000000000001\r\nNov 28 07:26:06 localhost kernel:  ffff8b96091f65a8 0000000000000008 ffff8b963e5a7c60 ffffffff965b9bf5\r\nNov 28 07:26:06 localhost kernel: Call Trace:\r\nNov 28 07:26:06 localhost kernel:  [<ffffffff9607bb30>] ? task_stopped_code+0x50/0x50\r\nNov 28 07:26:06 localhost kernel:  [<ffffffff965b9bf5>] schedule+0x35/0x80\r\nNov 28 07:26:06 localhost kernel:  [<ffffffff9611318f>] zap_pid_ns_processes+0x13f/0x1a0\r\nNov 28 07:26:06 localhost kernel:  [<ffffffff9607dcd6>] do_exit+0xa96/0xb10\r\nNov 28 07:26:06 localhost kernel:  [<ffffffff9607ddd5>] do_group_exit+0x45/0xb0\r\nNov 28 07:26:06 localhost kernel:  [<ffffffff96088ff2>] get_signal+0x282/0x610\r\nNov 28 07:26:06 localhost kernel:  [<ffffffff9602e197>] do_signal+0x37/0x6c0\r\nNov 28 07:26:06 localhost kernel:  [<ffffffff961ff9bf>] ? __vfs_read+0xdf/0x130\r\nNov 28 07:26:06 localhost kernel:  [<ffffffff9611bcf1>] ? __audit_syscall_entry+0xb1/0x100\r\nNov 28 07:26:06 localhost kernel:  [<ffffffff9600357c>] exit_to_usermode_loop+0x8c/0xd0\r\nNov 28 07:26:06 localhost kernel:  [<ffffffff96003bc9>] do_syscall_64+0xa9/0x160\r\n```\r\n\r\nI first tried with coreos stable(1185.3.0) and suffered from another issue, and moved to coreos alpha (1235.0.0) and then this issue is occurred. Here is the error log what I got in stable channel.\r\n\r\n```\r\nNov 28 05:26:32 localhost dockerd[1748]: 2016/11/28 05:26:32 http: panic serving @: runtime error: index out of range\r\nNov 28 05:26:32 localhost dockerd[1748]: goroutine 55942 [running]:\r\nNov 28 05:26:32 localhost dockerd[1748]: net/http.(*conn).serve.func1(0xc822768700)\r\nNov 28 05:26:32 localhost dockerd[1748]:         /usr/lib/go1.6/src/net/http/server.go:1389 +0xc1\r\nNov 28 05:26:32 localhost dockerd[1748]: panic(0x1532c80, 0xc82000e040)\r\nNov 28 05:26:32 localhost dockerd[1748]:         /usr/lib/go1.6/src/runtime/panic.go:443 +0x4e9\r\nNov 28 05:26:32 localhost dockerd[1748]: github.com/docker/docker/docker/hack.(*MalformedHostHeaderOverrideConn).Read(0xc823434300, 0xc826507000, 0x1000, 0x1000, 0x4, 0x0, 0x0)\r\nNov 28 05:26:32 localhost dockerd[1748]:         /build/amd64-usr/var/tmp/portage/app-emulation/docker-1.11.2-r5/work/docker-1.11.2/.gopath/src/github.com/docker/docker/docker/hack/malformed_host_override.go:58 +0x7b5\r\nNov 28 05:26:32 localhost dockerd[1748]: net/http.(*connReader).Read(0xc823434400, 0xc826507000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\r\nNov 28 05:26:32 localhost dockerd[1748]:         /usr/lib/go1.6/src/net/http/server.go:526 +0x196\r\nNov 28 05:26:32 localhost dockerd[1748]: bufio.(*Reader).fill(0xc822f8daa0)\r\nNov 28 05:26:32 localhost dockerd[1748]:         /usr/lib/go1.6/src/bufio/bufio.go:97 +0x1e9\r\nNov 28 05:26:32 localhost dockerd[1748]: bufio.(*Reader).ReadSlice(0xc822f8daa0, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0)\r\nNov 28 05:26:32 localhost dockerd[1748]:         /usr/lib/go1.6/src/bufio/bufio.go:328 +0x21a\r\nNov 28 05:26:32 localhost dockerd[1748]: bufio.(*Reader).ReadLine(0xc822f8daa0, 0x0, 0x0, 0x0, 0x1709400, 0x0, 0x0)\r\nNov 28 05:26:32 localhost dockerd[1748]:         /usr/lib/go1.6/src/bufio/bufio.go:357 +0x53\r\nNov 28 05:26:32 localhost dockerd[1748]: net/textproto.(*Reader).readLineSlice(0xc822f75200, 0x0, 0x0, 0x0, 0x0, 0x0)\r\nNov 28 05:26:32 localhost dockerd[1748]:         /usr/lib/go1.6/src/net/textproto/reader.go:55 +0x81\r\nNov 28 05:26:32 localhost dockerd[1748]: net/textproto.(*Reader).ReadLine(0xc822f75200, 0x0, 0x0, 0x0, 0x0)\r\nNov 28 05:26:32 localhost dockerd[1748]:         /usr/lib/go1.6/src/net/textproto/reader.go:36 +0x40\r\nNov 28 05:26:32 localhost dockerd[1748]: net/http.readRequest(0xc822f8daa0, 0xc81fe0c500, 0xc82585fb20, 0x0, 0x0)\r\nNov 28 05:26:32 localhost dockerd[1748]:         /usr/lib/go1.6/src/net/http/request.go:721 +0xb6\r\nNov 28 05:26:32 localhost dockerd[1748]: net/http.(*conn).readRequest(0xc822768700, 0x0, 0x0, 0x0)\r\nNov 28 05:26:32 localhost dockerd[1748]:         /usr/lib/go1.6/src/net/http/server.go:705 +0x359\r\nNov 28 05:26:32 localhost dockerd[1748]: net/http.(*conn).serve(0xc822768700)\r\nNov 28 05:26:32 localhost dockerd[1748]:         /usr/lib/go1.6/src/net/http/server.go:1425 +0x947\r\nNov 28 05:26:32 localhost dockerd[1748]: created by net/http.(*Server).Serve\r\nNov 28 05:26:32 localhost dockerd[1748]:         /usr/lib/go1.6/src/net/http/server.go:2137 +0x44e\r\n```\r\n"}}
{"actor":{"display_login":"pjknkda"}}