{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kops/issues/4974","repository_url":"https://api.github.com/repos/kubernetes/kops","labels_url":"https://api.github.com/repos/kubernetes/kops/issues/4974/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kops/issues/4974/comments","events_url":"https://api.github.com/repos/kubernetes/kops/issues/4974/events","html_url":"https://github.com/kubernetes/kops/issues/4974","id":313539696,"node_id":"MDU6SXNzdWUzMTM1Mzk2OTY=","number":4974,"title":"Kubelet service breaks after reboot of GCE nodes (permission denied)","user":{"login":"jacobrandall","id":4229502,"node_id":"MDQ6VXNlcjQyMjk1MDI=","avatar_url":"https://avatars2.githubusercontent.com/u/4229502?v=4","gravatar_id":"","url":"https://api.github.com/users/jacobrandall","html_url":"https://github.com/jacobrandall","followers_url":"https://api.github.com/users/jacobrandall/followers","following_url":"https://api.github.com/users/jacobrandall/following{/other_user}","gists_url":"https://api.github.com/users/jacobrandall/gists{/gist_id}","starred_url":"https://api.github.com/users/jacobrandall/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jacobrandall/subscriptions","organizations_url":"https://api.github.com/users/jacobrandall/orgs","repos_url":"https://api.github.com/users/jacobrandall/repos","events_url":"https://api.github.com/users/jacobrandall/events{/privacy}","received_events_url":"https://api.github.com/users/jacobrandall/received_events","type":"User","site_admin":false},"labels":[{"id":778139307,"node_id":"MDU6TGFiZWw3NzgxMzkzMDc=","url":"https://api.github.com/repos/kubernetes/kops/labels/lifecycle/stale","name":"lifecycle/stale","color":"795548","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":9,"created_at":"2018-04-12T00:58:58Z","updated_at":"2018-10-22T04:14:15Z","closed_at":null,"author_association":"NONE","body":"1. What `kops` version are you running? The command `kops version`, will display\r\n this information.\r\n\r\n`Version 1.8.1`\r\n\r\n2. What Kubernetes version are you running? `kubectl version` will print the\r\n version if a cluster is running or provide the Kubernetes version specified as\r\n a `kops` flag.\r\n\r\n`1.8.7` and `1.8.8`\r\n\r\n3. What cloud provider are you using?\r\n\r\nGCE\r\n\r\n4. What commands did you run?  What is the simplest way to reproduce this issue?\r\n\r\nAfter the initial node boot, we're able to run `/home/kubernetes/bin/kubelet -h` and the `kubelet` service starts up fine, however if the node is rebooted (GCE maintenance or manually running `sudo shutdown -r now`) the kubelet service fails to start and we're unable to run it manually (even with `sudo`), getting permission denied.\r\n\r\n```\r\nsystemd[20259]: kubelet.service: Failed at step EXEC spawning /home/kubernetes/bin/kubelet: Permission denied\r\n```\r\n\r\nand\r\n\r\n```\r\n $ sudo /home/kubernetes/bin/kubelet \r\nsudo: unable to execute /home/kubernetes/bin/kubelet: Permission denied\r\n```\r\n\r\n5. What happened after the commands executed?\r\n\r\nGet permission denied.\r\n\r\n6. What did you expect to happen?\r\n\r\nThe kubelet service to start after the node is rebooted\r\n\r\n7. Please provide your cluster manifest. Execute\r\n  `kops get --name my.example.com -oyaml` to display your cluster manifest.\r\n  You may want to remove your cluster name and other sensitive information.\r\n\r\nn/a\r\n\r\n8. Please run the commands with most verbose logging by adding the `-v 10` flag.\r\n  Paste the logs into this report, or in a gist and provide the gist link here.\r\n\r\nn/a\r\n\r\n9. Anything else do we need to know?\r\n\r\nWe also have k8s clusters provisioned via kops in AWS and confirmed we are able to reboot nodes and afterwards they return to `Ready` state without issue.\r\n\r\nBefore rebooting the instance has the following mount listed:\r\n\r\n```\r\n$ mount | grep kubernetes/bin\r\n/dev/sda1 on /home/kubernetes/bin type ext4 (rw,nosuid,nodev,relatime,commit=30,data=ordered)\r\n```\r\n\r\nand it does not show up after the reboot. I'm trying to dig in to how Kops / Container Optimized OS manage these but assuming it's related to something along those lines. If the issue lies with Container Optimized OS, apologies for opening this ticket, but any help you can provide to work around or resolve it would still be much appreicated.\r\n\r\nI should mention the file is still \"there\", and the permissions show anyone _should_ be able to execute it:\r\n\r\n```\r\n$ ls -alh /home/kubernetes/bin/kubelet \r\n-rwxr-xr-x 1 root root 132M Apr 12 00:38 /home/kubernetes/bin/kubelet\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/kubernetes/kops/issues/comments/431737052","html_url":"https://github.com/kubernetes/kops/issues/4974#issuecomment-431737052","issue_url":"https://api.github.com/repos/kubernetes/kops/issues/4974","id":431737052,"node_id":"MDEyOklzc3VlQ29tbWVudDQzMTczNzA1Mg==","user":{"login":"fejta-bot","id":30488982,"node_id":"MDQ6VXNlcjMwNDg4OTgy","avatar_url":"https://avatars1.githubusercontent.com/u/30488982?v=4","gravatar_id":"","url":"https://api.github.com/users/fejta-bot","html_url":"https://github.com/fejta-bot","followers_url":"https://api.github.com/users/fejta-bot/followers","following_url":"https://api.github.com/users/fejta-bot/following{/other_user}","gists_url":"https://api.github.com/users/fejta-bot/gists{/gist_id}","starred_url":"https://api.github.com/users/fejta-bot/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fejta-bot/subscriptions","organizations_url":"https://api.github.com/users/fejta-bot/orgs","repos_url":"https://api.github.com/users/fejta-bot/repos","events_url":"https://api.github.com/users/fejta-bot/events{/privacy}","received_events_url":"https://api.github.com/users/fejta-bot/received_events","type":"User","site_admin":false},"created_at":"2018-10-22T04:14:15Z","updated_at":"2018-10-22T04:14:15Z","author_association":"NONE","body":"Issues go stale after 90d of inactivity.\nMark the issue as fresh with `/remove-lifecycle stale`.\nStale issues rot after an additional 30d of inactivity and eventually close.\n\nIf this issue is safe to close now please do so with `/close`.\n\nSend feedback to sig-testing, kubernetes/test-infra and/or [fejta](https://github.com/fejta).\n/lifecycle stale"}}
{"actor":{"display_login":"fejta-bot"}}