{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kops/issues/5047","repository_url":"https://api.github.com/repos/kubernetes/kops","labels_url":"https://api.github.com/repos/kubernetes/kops/issues/5047/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kops/issues/5047/comments","events_url":"https://api.github.com/repos/kubernetes/kops/issues/5047/events","html_url":"https://github.com/kubernetes/kops/issues/5047","id":316076794,"number":5047,"title":"Support docker ce 18.03","user":{"login":"chotiwat","id":767257,"avatar_url":"https://avatars3.githubusercontent.com/u/767257?v=4","gravatar_id":"","url":"https://api.github.com/users/chotiwat","html_url":"https://github.com/chotiwat","followers_url":"https://api.github.com/users/chotiwat/followers","following_url":"https://api.github.com/users/chotiwat/following{/other_user}","gists_url":"https://api.github.com/users/chotiwat/gists{/gist_id}","starred_url":"https://api.github.com/users/chotiwat/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chotiwat/subscriptions","organizations_url":"https://api.github.com/users/chotiwat/orgs","repos_url":"https://api.github.com/users/chotiwat/repos","events_url":"https://api.github.com/users/chotiwat/events{/privacy}","received_events_url":"https://api.github.com/users/chotiwat/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-04-19T23:38:42Z","updated_at":"2018-04-21T01:14:26Z","closed_at":null,"author_association":"NONE","body":"------------- FEATURE REQUEST TEMPLATE --------------------\r\n\r\n1. Describe IN DETAIL the feature/behavior/change you would like to see.\r\n\r\n    My team wanted to use docker's multi-stage build which is available on docker >= 17.05. We had tried docker 17.09 by specifying it in the kops cluster config but after the switch we started seeing pods stuck in terminating state because their container could not be killed. The behavior is similar to https://github.com/docker/for-linux/issues/1. The only solution was to restart the docker service on the node that is having the problem. It got worse after we created a kubernetes CronJob resource that runs every 5 minutes.\r\n\r\n    We have been testing docker 18.03, by using our modified nodeup executable based on kops 1.9.0, and also testing another cluster with docker 17.03. So far, we haven't seen the problem reappearing in any of the clusters but we will continue to monitor them. We understand that the validated docker version for k8s 1.9 is 17.03.x but we still want to support multi-stage build on our cluster. We'd be happy to open a pr with our nodeup change.\r\n\r\n    Versions:\r\n    * Kops: 1.9.0\r\n    * Kubernetes: 1.9.6\r\n    * Kubectl: 1.9.5\r\n    * Master/node OS: Ubuntu 16.04 xenial\r\n\r\n2. Feel free to provide a design supporting your feature request.\r\n\r\n    This is the cronjob spec that we use to reproduce the problem.\r\n```\r\napiVersion: batch/v1beta1\r\nkind: CronJob\r\nmetadata:\r\n  name: hello\r\n  namespace: default\r\nspec:\r\n  startingDeadlineSeconds: 30\r\n  concurrencyPolicy: Forbid\r\n  schedule: \"*/2 * * * *\"\r\n  jobTemplate:\r\n    spec:\r\n      parallelism: 1\r\n      completions: 1\r\n      backoffLimit: 1\r\n      activeDeadlineSeconds: 60\r\n      template:\r\n        spec:\r\n          containers:\r\n          - name: hello\r\n            image: hello-world\r\n          restartPolicy: Never\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/kubernetes/kops/issues/comments/383256672","html_url":"https://github.com/kubernetes/kops/issues/5047#issuecomment-383256672","issue_url":"https://api.github.com/repos/kubernetes/kops/issues/5047","id":383256672,"user":{"login":"chotiwat","id":767257,"avatar_url":"https://avatars3.githubusercontent.com/u/767257?v=4","gravatar_id":"","url":"https://api.github.com/users/chotiwat","html_url":"https://github.com/chotiwat","followers_url":"https://api.github.com/users/chotiwat/followers","following_url":"https://api.github.com/users/chotiwat/following{/other_user}","gists_url":"https://api.github.com/users/chotiwat/gists{/gist_id}","starred_url":"https://api.github.com/users/chotiwat/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chotiwat/subscriptions","organizations_url":"https://api.github.com/users/chotiwat/orgs","repos_url":"https://api.github.com/users/chotiwat/repos","events_url":"https://api.github.com/users/chotiwat/events{/privacy}","received_events_url":"https://api.github.com/users/chotiwat/received_events","type":"User","site_admin":false},"created_at":"2018-04-21T01:14:26Z","updated_at":"2018-04-21T01:14:26Z","author_association":"NONE","body":"Update: I reproduced the issue again, this time with a debian image `kope.io/k8s-1.8-debian-jessie-amd64-hvm-ebs-2018-02-08`, after running the CronJob for 4 hours. I also enabled debug log for docker daemon and this is what I got from `grep`ing the log with the container id of the first zombie pod:\r\n```\r\nApr 20 23:36:08 ip-xxxxxx dockerd[2692]: time=\"2018-04-20T23:36:08.852603579Z\" level=debug msg=\"Calling POST /v1.31/containers/ec2f5f3d9816912ec02445a51da0c696010b8ba3d6dd8aa5929a20c22f1128d0/start\"\r\nApr 20 23:36:08 ip-xxxxxx dockerd[2692]: time=\"2018-04-20T23:36:08.921334436Z\" level=warning msg=\"Unknown healthcheck type 'NONE' (expected 'CMD') in container ec2f5f3d9816912ec02445a51da0c696010b8ba3d6dd8aa5929a20c22f1128d0\"\r\nApr 20 23:36:08 ip-xxxxxx dockerd[2692]: time=\"2018-04-20T23:36:08.927344196Z\" level=debug msg=\"Calling GET /v1.31/containers/ec2f5f3d9816912ec02445a51da0c696010b8ba3d6dd8aa5929a20c22f1128d0/json\"\r\nApr 20 23:37:08 ip-xxxxxx dockerd[2692]: time=\"2018-04-20T23:37:08.966488414Z\" level=debug msg=\"Calling GET /v1.31/containers/ec2f5f3d9816912ec02445a51da0c696010b8ba3d6dd8aa5929a20c22f1128d0/json\"\r\nApr 20 23:37:08 ip-xxxxxx dockerd[2692]: time=\"2018-04-20T23:37:08.969425649Z\" level=debug msg=\"Calling POST /v1.31/containers/ec2f5f3d9816912ec02445a51da0c696010b8ba3d6dd8aa5929a20c22f1128d0/stop?t=30\"\r\nApr 20 23:37:08 ip-xxxxxx dockerd[2692]: time=\"2018-04-20T23:37:08.969462609Z\" level=debug msg=\"Sending kill signal 15 to container ec2f5f3d9816912ec02445a51da0c696010b8ba3d6dd8aa5929a20c22f1128d0\"\r\nApr 20 23:37:08 ip-xxxxxx dockerd[2692]: time=\"2018-04-20T23:37:08.971371712Z\" level=warning msg=\"container kill failed because of 'container not found' or 'no such process': Cannot kill container ec2f5f3d9816912ec02445a51da0c696010b8ba3d6dd8aa5929a20c22f1128d0: rpc error: code = Unknown desc = containerd: container not found\"\r\nApr 20 23:37:38 ip-xxxxxx dockerd[2692]: time=\"2018-04-20T23:37:38.975168187Z\" level=info msg=\"Container ec2f5f3d9816912ec02445a51da0c696010b8ba3d6dd8aa5929a20c22f1128d0 failed to exit within 30 seconds of signal 15 - using the force\"\r\nApr 20 23:37:38 ip-xxxxxx dockerd[2692]: time=\"2018-04-20T23:37:38.975539946Z\" level=debug msg=\"Sending kill signal 9 to container ec2f5f3d9816912ec02445a51da0c696010b8ba3d6dd8aa5929a20c22f1128d0\"\r\nApr 20 23:37:38 ip-xxxxxx dockerd[2692]: time=\"2018-04-20T23:37:38.985650139Z\" level=warning msg=\"container kill failed because of 'container not found' or 'no such process': Cannot kill container ec2f5f3d9816912ec02445a51da0c696010b8ba3d6dd8aa5929a20c22f1128d0: rpc error: code = Unknown desc = containerd: container not found\"\r\nApr 20 23:38:10 ip-xxxxxx dockerd[2692]: time=\"2018-04-20T23:38:10.995119918Z\" level=debug msg=\"Calling GET /v1.31/containers/ec2f5f3d9816912ec02445a51da0c696010b8ba3d6dd8aa5929a20c22f1128d0/json\"\r\nApr 20 23:38:10 ip-xxxxxx dockerd[2692]: time=\"2018-04-20T23:38:10.997696415Z\" level=debug msg=\"Calling POST /v1.31/containers/ec2f5f3d9816912ec02445a51da0c696010b8ba3d6dd8aa5929a20c22f1128d0/stop?t=30\"\r\nApr 20 23:38:10 ip-xxxxxx dockerd[2692]: time=\"2018-04-20T23:38:10.997992274Z\" level=debug msg=\"Sending kill signal 15 to container ec2f5f3d9816912ec02445a51da0c696010b8ba3d6dd8aa5929a20c22f1128d0\"\r\nApr 20 23:38:10 ip-xxxxxx dockerd[2692]: time=\"2018-04-20T23:38:10.998585925Z\" level=warning msg=\"container kill failed because of 'container not found' or 'no such process': Cannot kill container ec2f5f3d9816912ec02445a51da0c696010b8ba3d6dd8aa5929a20c22f1128d0: rpc error: code = Unknown desc = containerd: container not found\"\r\nApr 20 23:38:40 ip-xxxxxx dockerd[2692]: time=\"2018-04-20T23:38:40.999039585Z\" level=info msg=\"Container ec2f5f3d9816912ec02445a51da0c696010b8ba3d6dd8aa5929a20c22f1128d0 failed to exit within 30 seconds of signal 15 - using the force\"\r\nApr 20 23:38:40 ip-xxxxxx dockerd[2692]: time=\"2018-04-20T23:38:40.999307052Z\" level=debug msg=\"Sending kill signal 9 to container ec2f5f3d9816912ec02445a51da0c696010b8ba3d6dd8aa5929a20c22f1128d0\"\r\nApr 20 23:38:41 ip-xxxxxx dockerd[2692]: time=\"2018-04-20T23:38:40.999881607Z\" level=warning msg=\"container kill failed because of 'container not found' or 'no such process': Cannot kill container ec2f5f3d9816912ec02445a51da0c696010b8ba3d6dd8aa5929a20c22f1128d0: rpc error: code = Unknown desc = containerd: container not found\"\r\nApr 20 23:39:12 ip-xxxxxx dockerd[2692]: time=\"2018-04-20T23:39:12.957926317Z\" level=debug msg=\"Calling GET /v1.31/containers/ec2f5f3d9816912ec02445a51da0c696010b8ba3d6dd8aa5929a20c22f1128d0/json\"\r\nApr 20 23:39:12 ip-xxxxxx dockerd[2692]: time=\"2018-04-20T23:39:12.972695460Z\" level=debug msg=\"Calling GET /v1.31/containers/ec2f5f3d9816912ec02445a51da0c696010b8ba3d6dd8aa5929a20c22f1128d0/json\"\r\nApr 20 23:39:12 ip-xxxxxx dockerd[2692]: time=\"2018-04-20T23:39:12.976839162Z\" level=debug msg=\"Calling POST /v1.31/containers/ec2f5f3d9816912ec02445a51da0c696010b8ba3d6dd8aa5929a20c22f1128d0/stop?t=30\"\r\nApr 20 23:39:12 ip-xxxxxx dockerd[2692]: time=\"2018-04-20T23:39:12.977147551Z\" level=debug msg=\"Sending kill signal 15 to container ec2f5f3d9816912ec02445a51da0c696010b8ba3d6dd8aa5929a20c22f1128d0\"\r\nApr 20 23:39:12 ip-xxxxxx dockerd[2692]: time=\"2018-04-20T23:39:12.977789931Z\" level=warning msg=\"container kill failed because of 'container not found' or 'no such process': Cannot kill container ec2f5f3d9816912ec02445a51da0c696010b8ba3d6dd8aa5929a20c22f1128d0: rpc error: code = Unknown desc = containerd: container not found\"\r\nApr 20 23:39:42 ip-xxxxxx dockerd[2692]: time=\"2018-04-20T23:39:42.979087621Z\" level=info msg=\"Container ec2f5f3d9816912ec02445a51da0c696010b8ba3d6dd8aa5929a20c22f1128d0 failed to exit within 30 seconds of signal 15 - using the force\"\r\n```\r\n\r\nI found that the handling of these error messages has been changed since docker 17.12, specifically in moby/moby#35809, which hopefully is what fixed the problem on my cluster with docker 18.03."}}
{"actor":{"display_login":"chotiwat"}}