{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/36611","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/36611/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/36611/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/36611/events","html_url":"https://github.com/kubernetes/kubernetes/issues/36611","id":188635503,"number":36611,"title":"AzureDisk Mount causing Kubelet Crash","user":{"login":"jpoon","id":85374,"avatar_url":"https://avatars.githubusercontent.com/u/85374?v=3","gravatar_id":"","url":"https://api.github.com/users/jpoon","html_url":"https://github.com/jpoon","followers_url":"https://api.github.com/users/jpoon/followers","following_url":"https://api.github.com/users/jpoon/following{/other_user}","gists_url":"https://api.github.com/users/jpoon/gists{/gist_id}","starred_url":"https://api.github.com/users/jpoon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jpoon/subscriptions","organizations_url":"https://api.github.com/users/jpoon/orgs","repos_url":"https://api.github.com/users/jpoon/repos","events_url":"https://api.github.com/users/jpoon/events{/privacy}","received_events_url":"https://api.github.com/users/jpoon/received_events","type":"User","site_admin":false},"labels":[{"id":116719829,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/component/kubelet","name":"component/kubelet","color":"0052cc","default":false},{"id":173493827,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/cluster","name":"team/cluster","color":"d2b48c","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":16,"created_at":"2016-11-10T22:26:20Z","updated_at":"2016-11-22T05:44:46Z","closed_at":null,"body":"**Kubernetes version** (use `kubectl version`):\r\n```\r\nClient Version: version.Info{Major:\"1\", Minor:\"4\", GitVersion:\"v1.4.0\", GitCommit:\"a16c0a7f71a6f93c7e0f222d961f4675cd97a46b\", GitTreeState:\"clean\", BuildDate:\"2016-09-26T18:16:57Z\", GoVersion:\"go1.6.3\", Compiler:\"gc\", Platform:\"darwin/amd64\"}\r\nServer Version: version.Info{Major:\"1\", Minor:\"4\", GitVersion:\"v1.4.5\", GitCommit:\"5a0a696437ad35c133c0c8493f7e9d22b0f9b81b\", GitTreeState:\"clean\", BuildDate:\"2016-10-29T01:32:42Z\", GoVersion:\"go1.6.3\", Compiler:\"gc\", Platform:\"linux/amd64\"}\r\n```\r\n\r\n**Environment**:\r\n- **Cloud provider or hardware configuration**: azure (acs-engine)\r\n- **OS** (e.g. from /etc/os-release): Ubuntu 16.04 LTS\r\n- **Kernel** (e.g. `uname -a`): Linux k8s-master-5A10CE6E-0 4.4.0-47-generic #68-Ubuntu SMP Wed Oct 26 19:39:52 UTC 2016x86_64 x86_64 x86_64 GNU/Linux\r\n\r\n\r\n**What happened**:\r\n\r\nDeploying Pachyderm onto Kubernetes cluster on Azure. Pachyderm requires a persistent volume for their RethinkDB and we're seeing kubelet crash and eventual VM lock up.\r\n\r\n* Empty data disk is created, ext4 formatted, and uploaded to storage\r\n* The data-disk mount is successful\r\n ```\r\nI1110 18:46:17.084038   17216 operation_executor.go:766] MountVolume.MountDevice succeeded for volume \"kubernetes.io/azure-disk/pach-disk.vhd\" (spec.Name: \"rethink-volume\") pod \"03ad3308-a775-11e6-aa08-000d3a34f678\" (UID: \"03ad3308-a775-11e6-aa08-000d3a34f678\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/azure-disk/mounts/pach-disk.vhd\"\r\nI1110 18:46:17.118751   17216 operation_executor.go:803] MountVolume.SetUp succeeded for volume \"kubernetes.io/azure-disk/pach-disk.vhd\" (spec.Name: \"rethink-volume\") pod \"03ad3308-a775-11e6-aa08-000d3a34f678\" (UID: \"03ad3308-a775-11e6-aa08-000d3a34f678\")\r\n```\r\n* After about 4-5 minutes, kubelet crashes on the node\r\n ```\r\nNov 10 18:45:28 k8s-agent-5A10CE6E-1 docker[12793]: panic: runtime error: invalid memory address or nil pointer dereference\r\nNov 10 18:45:28 k8s-agent-5A10CE6E-1 docker[12793]: [signal 0xb code=0x1 addr=0x0 pc=0xe3a62c]\r\nNov 10 18:45:28 k8s-agent-5A10CE6E-1 docker[12793]: goroutine 283 [running]:\r\nNov 10 18:45:28 k8s-agent-5A10CE6E-1 docker[12793]: panic(0x44ce1a0, 0xc820012070)\r\nNov 10 18:45:28 k8s-agent-5A10CE6E-1 docker[12793]:         /usr/local/go/src/runtime/panic.go:481 +0x3e6\r\nNov 10 18:45:28 k8s-agent-5A10CE6E-1 docker[12793]: k8s.io/kubernetes/pkg/volume/azure_dd.(*azureDataDiskPlugin).newMounterInternal(0xc82066c060, 0xc821c86120, 0xc8218dca56, 0x24, 0x7ff0f97fc650, 0x762e900, 0x0, 0x0, 0x0, 0x0)\r\nNov 10 18:45:28 k8s-agent-5A10CE6E-1 docker[12793]:         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/volume/azure_dd/azure_dd.go:117 +0x61c\r\nNov 10 18:45:28 k8s-agent-5A10CE6E-1 docker[12793]: k8s.io/kubernetes/pkg/volume/azure_dd.(*azureDataDiskPlugin).NewMounter(0xc82066c060, 0xc821c86120, 0xc821840500, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\r\nNov 10 18:45:28 k8s-agent-5A10CE6E-1 docker[12793]:         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/volume/azure_dd/azure_dd.go:106 +0xc3\r\nNov 10 18:45:28 k8s-agent-5A10CE6E-1 docker[12793]: k8s.io/kubernetes/pkg/kubelet/volumemanager/reconciler.(*reconciler).reconstructVolume(0xc82088d8c0, 0xc8218dca56, 0x24, 0xc82187405c, 0xe, 0xc821874150, 0x6a, 0xc821b2b1e0, 0x18, 0x0, ...)\r\nNov 10 18:45:28 k8s-agent-5A10CE6E-1 docker[12793]:         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/volumemanager/reconciler/reconciler.go:517 +0x399\r\nNov 10 18:45:28 k8s-agent-5A10CE6E-1 docker[12793]: k8s.io/kubernetes/pkg/kubelet/volumemanager/reconciler.(*reconciler).syncStates(0xc82088d8c0, 0xc820072e20, 0x15)\r\nNov 10 18:45:28 k8s-agent-5A10CE6E-1 docker[12793]:         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/volumemanager/reconciler/reconciler.go:430 +0x354\r\nNov 10 18:45:28 k8s-agent-5A10CE6E-1 docker[12793]: k8s.io/kubernetes/pkg/kubelet/volumemanager/reconciler.(*reconciler).sync(0xc82088d8c0)\r\nNov 10 18:45:28 k8s-agent-5A10CE6E-1 docker[12793]:         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/volumemanager/reconciler/reconciler.go:384 +0x68\r\nNov 10 18:45:28 k8s-agent-5A10CE6E-1 docker[12793]: k8s.io/kubernetes/pkg/kubelet/volumemanager/reconciler.(*reconciler).reconciliationLoopFunc.func1()\r\nNov 10 18:45:28 k8s-agent-5A10CE6E-1 docker[12793]:         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/volumemanager/reconciler/reconciler.go:147 +0xf4\r\nNov 10 18:45:28 k8s-agent-5A10CE6E-1 docker[12793]: k8s.io/kubernetes/pkg/util/wait.JitterUntil.func1(0xc821172120)\r\nNov 10 18:45:28 k8s-agent-5A10CE6E-1 docker[12793]:         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:84 +0x19\r\nNov 10 18:45:28 k8s-agent-5A10CE6E-1 docker[12793]: k8s.io/kubernetes/pkg/util/wait.JitterUntil(0xc821172120, 0x5f5e100, 0x0, 0xc821172101, 0xc820050fc0)\r\nNov 10 18:45:28 k8s-agent-5A10CE6E-1 docker[12793]:         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:85 +0xb4\r\nNov 10 18:45:28 k8s-agent-5A10CE6E-1 docker[12793]: k8s.io/kubernetes/pkg/util/wait.Until(0xc821172120, 0x5f5e100, 0xc820050fc0)\r\nNov 10 18:45:28 k8s-agent-5A10CE6E-1 docker[12793]:         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:47 +0x43\r\nNov 10 18:45:28 k8s-agent-5A10CE6E-1 docker[12793]: k8s.io/kubernetes/pkg/kubelet/volumemanager/reconciler.(*reconciler).Run(0xc82088d8c0, 0x7ff0f97ff160, 0xc8201b32c0, 0xc820050fc0)\r\nNov 10 18:45:28 k8s-agent-5A10CE6E-1 docker[12793]:         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/volumemanager/reconciler/reconciler.go:133 +0x5b\r\nNov 10 18:45:28 k8s-agent-5A10CE6E-1 docker[12793]: created by k8s.io/kubernetes/pkg/kubelet/volumemanager.(*volumeManager).Run\r\nNov 10 18:45:28 k8s-agent-5A10CE6E-1 docker[12793]:         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/volumemanager/volume_manager.go:240 +0x160\r\nNov 10 18:45:28 k8s-agent-5A10CE6E-1 systemd[1]: kubelet.service: Main process exited, code=exited, status=2/INVALIDARGUMENT\r\n```\r\n* This repeats for about an hour until the VM locks up (e.g. unable to SSH). I needed to forcibly stop and start the VM from the portal (restarting did not work). Over that hour, the node’s memory usage steadily grows and the CPU spikes close to 100%\r\n\r\n<img width=\"533\" alt=\"screen shot 2016-11-10 at 12 12 32\" src=\"https://cloud.githubusercontent.com/assets/85374/20196865/839a22ac-a751-11e6-8ff5-af4ffdccc811.png\">\r\n\r\n(Pachyderm deployed at 10:40am, VM locks up at 11:30ish am)\r\n\r\n\r\n**What you expected to happen**:\r\n\r\nAzure Disk mounted, no crash. Rainbows and unicorns.\r\n\r\n**How to reproduce it** (as minimally and precisely as possible):\r\n\r\n* Deploy Kubernetes cluster via ACS through Azure Portal\r\n* Deploy Pachyderm (script for doing so: https://gist.github.com/jpoon/c4b781c5eeb395b9ea8452b42cb993a4)\r\n\r\n**Anything else do we need to know**:\r\n\r\n* I did not see this behavior (VM lock-up, memory/CPU spike) with a v1.4.0 Kubernetes cluster deployed via kubernetes-anywhere. \r\n* [kubelet log](https://github.com/kubernetes/kubernetes/files/584575/kubelet.txt)\r\n* [pachyderm_manifest.txt](https://github.com/kubernetes/kubernetes/files/584578/pachyderm_manifest.txt)\r\n\r\ncc @colemickens "},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/262153244","html_url":"https://github.com/kubernetes/kubernetes/issues/36611#issuecomment-262153244","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/36611","id":262153244,"user":{"login":"jgeraerts","id":115240,"avatar_url":"https://avatars.githubusercontent.com/u/115240?v=3","gravatar_id":"","url":"https://api.github.com/users/jgeraerts","html_url":"https://github.com/jgeraerts","followers_url":"https://api.github.com/users/jgeraerts/followers","following_url":"https://api.github.com/users/jgeraerts/following{/other_user}","gists_url":"https://api.github.com/users/jgeraerts/gists{/gist_id}","starred_url":"https://api.github.com/users/jgeraerts/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jgeraerts/subscriptions","organizations_url":"https://api.github.com/users/jgeraerts/orgs","repos_url":"https://api.github.com/users/jgeraerts/repos","events_url":"https://api.github.com/users/jgeraerts/events{/privacy}","received_events_url":"https://api.github.com/users/jgeraerts/received_events","type":"User","site_admin":false},"created_at":"2016-11-22T05:44:46Z","updated_at":"2016-11-22T05:44:46Z","body":"thank you @colemickens , didn't get into collecting all the information yet. "}}
{"actor":{"display_login":"jgeraerts"}}