{"action":"created","issue":{"url":"https://api.github.com/repos/docker/docker/issues/22226","repository_url":"https://api.github.com/repos/docker/docker","labels_url":"https://api.github.com/repos/docker/docker/issues/22226/labels{/name}","comments_url":"https://api.github.com/repos/docker/docker/issues/22226/comments","events_url":"https://api.github.com/repos/docker/docker/issues/22226/events","html_url":"https://github.com/docker/docker/issues/22226","id":150061429,"number":22226,"title":"Sporadic RPC error \"containerd: container did not start before the specified timeout\"","user":{"login":"talon-hartmann","id":16796014,"avatar_url":"https://avatars.githubusercontent.com/u/16796014?v=3","gravatar_id":"","url":"https://api.github.com/users/talon-hartmann","html_url":"https://github.com/talon-hartmann","followers_url":"https://api.github.com/users/talon-hartmann/followers","following_url":"https://api.github.com/users/talon-hartmann/following{/other_user}","gists_url":"https://api.github.com/users/talon-hartmann/gists{/gist_id}","starred_url":"https://api.github.com/users/talon-hartmann/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/talon-hartmann/subscriptions","organizations_url":"https://api.github.com/users/talon-hartmann/orgs","repos_url":"https://api.github.com/users/talon-hartmann/repos","events_url":"https://api.github.com/users/talon-hartmann/events{/privacy}","received_events_url":"https://api.github.com/users/talon-hartmann/received_events","type":"User","site_admin":false},"labels":[{"id":190258329,"url":"https://api.github.com/repos/docker/docker/labels/kind/bug","name":"kind/bug","color":"FF8C94","default":false},{"id":375217742,"url":"https://api.github.com/repos/docker/docker/labels/version/1.11","name":"version/1.11","color":"ededed","default":false}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/docker/docker/milestones/53","html_url":"https://github.com/docker/docker/milestones/1.12.0","labels_url":"https://api.github.com/repos/docker/docker/milestones/53/labels","id":1657301,"number":53,"title":"1.12.0","description":"feature/code freeze: June 14th, 2016","creator":{"login":"thaJeztah","id":1804568,"avatar_url":"https://avatars.githubusercontent.com/u/1804568?v=3","gravatar_id":"","url":"https://api.github.com/users/thaJeztah","html_url":"https://github.com/thaJeztah","followers_url":"https://api.github.com/users/thaJeztah/followers","following_url":"https://api.github.com/users/thaJeztah/following{/other_user}","gists_url":"https://api.github.com/users/thaJeztah/gists{/gist_id}","starred_url":"https://api.github.com/users/thaJeztah/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thaJeztah/subscriptions","organizations_url":"https://api.github.com/users/thaJeztah/orgs","repos_url":"https://api.github.com/users/thaJeztah/repos","events_url":"https://api.github.com/users/thaJeztah/events{/privacy}","received_events_url":"https://api.github.com/users/thaJeztah/received_events","type":"User","site_admin":false},"open_issues":23,"closed_issues":909,"state":"closed","created_at":"2016-03-21T13:47:53Z","updated_at":"2016-11-08T12:00:39Z","due_on":"2016-07-28T07:00:00Z","closed_at":"2016-07-29T00:47:52Z"},"comments":51,"created_at":"2016-04-21T12:36:11Z","updated_at":"2016-11-10T16:55:50Z","closed_at":"2016-06-15T20:08:01Z","body":"<!--\nIf you are reporting a new issue, make sure that we do not have any duplicates\nalready open. You can ensure this by searching the issue list for this\nrepository. If there is a duplicate, please close your issue and add a comment\nto the existing issue instead.\n\nIf you suspect your issue is a bug, please edit your issue description to\ninclude the BUG REPORT INFORMATION shown below. If you fail to provide this\ninformation within 7 days, we cannot debug your issue and will close it. We\nwill, however, reopen it if you later provide the information.\n\nFor more information about reporting issues, see\nhttps://github.com/docker/docker/blob/master/CONTRIBUTING.md#reporting-other-issues\n\n---------------------------------------------------\nBUG REPORT INFORMATION\n---------------------------------------------------\nUse the commands below to provide key information from your environment:\nYou do NOT have to include this information if this is a FEATURE REQUEST\n-->\n\n**Output of `docker version`:**\n\n```\nClient:\n Version:      1.11.0\n API version:  1.23\n Go version:   go1.5.4\n Git commit:   4dc5990\n Built:        Wed Apr 13 18:17:17 2016\n OS/Arch:      linux/amd64\n\nServer:\n Version:      1.11.0\n API version:  1.23\n Go version:   go1.5.4\n Git commit:   4dc5990\n Built:        Wed Apr 13 18:17:17 2016\n OS/Arch:      linux/amd64\n```\n\n**Output of `docker info`:**\n\n```\nContainers: 38\n Running: 23\n Paused: 0\n Stopped: 15\nImages: 37\nServer Version: 1.11.0\nStorage Driver: overlay\n Backing Filesystem: extfs\nLogging Driver: json-file\nCgroup Driver: cgroupfs\nPlugins: \n Volume: local\n Network: bridge null host\nKernel Version: 4.4.0-0.bpo.1-amd64\nOperating System: Debian GNU/Linux 8 (jessie)\nOSType: linux\nArchitecture: x86_64\nCPUs: 8\nTotal Memory: 31.24 GiB\nName: talon.one\nID: 4S3Y:6F4A:EYIB:DWKQ:C5C7:5RLX:YAM6:O426:BLGU:HP47:KN6J:FA5N\nDocker Root Dir: /var/lib/docker\nDebug mode (client): false\nDebug mode (server): false\nRegistry: https://index.docker.io/v1/\nWARNING: No kernel memory limit support\n```\n\n**Additional environment details (AWS, VirtualBox, physical, etc.):**\nPhysical box:\nDebian 8.4\nLinux talon.one 4.4.0-0.bpo.1-amd64 #1 SMP Debian 4.4.6-1~bpo8+1 (2016-03-20) x86_64 GNU/Linux\n\n**Steps to reproduce the issue:**\n\nWe build our image using Drone, and at the end of the build the new docker image is automatically deployed on the same server:\n\n```\n$ docker run --restart=always --name talon-master-api -e POSTGRES_PORT_5432_TCP_ADDR=talon-master-postgres --env-file=/home/talonone/secrets/salesforce --net=talon-master-nw --expose=9000 -d docker.talon.one/talon-api/master:latest\n27625ea8e6ca8d59c2b501451846d476a0ce50f8c6d87a7e465be255d5a3de7a\n```\n\nSometimes we get the response:\n\n```\ndocker: Error response from daemon: rpc error: code = 2 desc = \"containerd: container did not start before the specified timeout\".\n```\n\nThe issue is similar to https://github.com/docker/docker/issues/22053 but we don't use docker compose. \n\nI can also reproduce this issue by manually doing:\n\n```\ndocker restart talon-master-api\n```\n\nThe command then hangs for a while and outputs with intermittent kernel messages:\n\n```\nMessage from syslogd@talon at Apr 21 13:59:30 ...\n kernel:[71227.458071] unregister_netdevice: waiting for lo to become free. Usage count = 1\nError response from daemon: Cannot restart container talon-master-api: rpc error: code = 2 desc = \"containerd: container did not start before the specified timeout\"\n```\n\nSo this issue might be triggered by / connected to the open issue  https://github.com/docker/docker/issues/5618\n\nI restarted the docker daemon with systemctl after adding the --debug flag in the systemd unit. This made the problem go away temporarly; I suspect this is because it cleans up the \"waiting for lo to become free\" bug.\n\nAfter around 2 minutes, the \"waiting for lo to become free\" problem reappears. \n\nWhile trying to reproduce this issue, I run into another problem (not sure if connected): docker start and ps see a different state of one of the containers:\n\n```\nroot@talon /home/talonone # docker start demo-telco-master\nError response from daemon: Container 8b8eac92b62c06297ec87a1f27ef7e3d26aabd8b571bc026f88edbf9538d1e2c is aleady active\nError: failed to start containers: demo-telco-master\nroot@talon /home/talonone # docker ps | grep demo-telco-master\nroot@talon /home/talonone # docker ps -a | grep demo-telco-master\n8b8eac92b62c        docker.talon.one/demo-telco/master:latest             \"/bin/sh -c 'ruby app\"   17 hours ago        Exited (128) 6 minutes ago                                             demo-telco-master\n```\n\nI will file a seperate issue for this.\n\nIn docker debug mode, i wasn't quickly able to reproduce the main issue (container did not start). I will submit daemon debug logs as soon as it reappears.\n\nI will gladly supply any needed debug info to help resolve the issue.\n\n**Additional information you deem important (e.g. issue happens only occasionally):**\n\nDockerfile:\n\n```\nFROM alpine:latest\n\nRUN apk --update upgrade && \\\n    apk add ca-certificates && \\\n    update-ca-certificates && \\\n    rm -rf /var/cache/apk/*\n\nCOPY talon /talon/talon\nWORKDIR /talon\nENV PATH=$PATH:/talon \nCMD [\"talon\"]\n\nEXPOSE 9000   \n```\n\nReference to discussion on twitter: https://twitter.com/mntmn/status/723108094620786688\n"},"comment":{"url":"https://api.github.com/repos/docker/docker/issues/comments/259744680","html_url":"https://github.com/docker/docker/issues/22226#issuecomment-259744680","issue_url":"https://api.github.com/repos/docker/docker/issues/22226","id":259744680,"user":{"login":"dirk-thomas","id":1335366,"avatar_url":"https://avatars.githubusercontent.com/u/1335366?v=3","gravatar_id":"","url":"https://api.github.com/users/dirk-thomas","html_url":"https://github.com/dirk-thomas","followers_url":"https://api.github.com/users/dirk-thomas/followers","following_url":"https://api.github.com/users/dirk-thomas/following{/other_user}","gists_url":"https://api.github.com/users/dirk-thomas/gists{/gist_id}","starred_url":"https://api.github.com/users/dirk-thomas/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dirk-thomas/subscriptions","organizations_url":"https://api.github.com/users/dirk-thomas/orgs","repos_url":"https://api.github.com/users/dirk-thomas/repos","events_url":"https://api.github.com/users/dirk-thomas/events{/privacy}","received_events_url":"https://api.github.com/users/dirk-thomas/received_events","type":"User","site_admin":false},"created_at":"2016-11-10T16:55:49Z","updated_at":"2016-11-10T16:55:49Z","body":"The folder only contains a `config.json` as well as three pipelines `init-stderr`, `init-stdin`, and `init-stdout`."}}
{"actor":{"display_login":"dirk-thomas"}}