{"action":"created","issue":{"url":"https://api.github.com/repos/Azure/AKS/issues/387","repository_url":"https://api.github.com/repos/Azure/AKS","labels_url":"https://api.github.com/repos/Azure/AKS/issues/387/labels{/name}","comments_url":"https://api.github.com/repos/Azure/AKS/issues/387/comments","events_url":"https://api.github.com/repos/Azure/AKS/issues/387/events","html_url":"https://github.com/Azure/AKS/issues/387","id":326170049,"node_id":"MDU6SXNzdWUzMjYxNzAwNDk=","number":387,"title":"New Cluster: Heapster stuck in CrashLoopBackOff","user":{"login":"lastcoolnameleft","id":686999,"node_id":"MDQ6VXNlcjY4Njk5OQ==","avatar_url":"https://avatars1.githubusercontent.com/u/686999?v=4","gravatar_id":"","url":"https://api.github.com/users/lastcoolnameleft","html_url":"https://github.com/lastcoolnameleft","followers_url":"https://api.github.com/users/lastcoolnameleft/followers","following_url":"https://api.github.com/users/lastcoolnameleft/following{/other_user}","gists_url":"https://api.github.com/users/lastcoolnameleft/gists{/gist_id}","starred_url":"https://api.github.com/users/lastcoolnameleft/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lastcoolnameleft/subscriptions","organizations_url":"https://api.github.com/users/lastcoolnameleft/orgs","repos_url":"https://api.github.com/users/lastcoolnameleft/repos","events_url":"https://api.github.com/users/lastcoolnameleft/events{/privacy}","received_events_url":"https://api.github.com/users/lastcoolnameleft/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":16,"created_at":"2018-05-24T15:17:46Z","updated_at":"2018-06-14T14:38:59Z","closed_at":null,"author_association":"MEMBER","body":"I created a brand new cluster in EastUS using B2 agent pool.\r\n\r\nHere is heapster in CrashLoopBackOff\r\n```\r\n➜  ~ kubectl get pods\r\nNAME                                                              READY     STATUS             RESTARTS   AGE\r\naddon-http-application-routing-default-http-backend-66c97f644rk   1/1       Running            0          27m\r\naddon-http-application-routing-external-dns-f9c5c6d5-pn22p        1/1       Running            0          27m\r\naddon-http-application-routing-nginx-ingress-controller-64n4k6b   1/1       Running            2          27m\r\nazureproxy-79c5db744-2vklk                                        1/1       Running            2          27m\r\nheapster-7cd8dd888b-dqs9h                                         1/2       CrashLoopBackOff   9          27m\r\nkube-dns-v20-7c556f89c5-6fbhl                                     3/3       Running            0          27m\r\nkube-dns-v20-7c556f89c5-h6lwm                                     3/3       Running            0          27m\r\nkube-proxy-9v4gj                                                  1/1       Running            0          27m\r\nkube-svc-redirect-f29d4                                           1/1       Running            0          27m\r\nkubernetes-dashboard-546f987686-tbc5x                             1/1       Running            4          27m\r\nomsagent-hwk84                                                    1/1       Running            0          27m\r\ntunnelfront-55c84579bf-ld6g4                                      1/1       Running            0          27m\r\n```\r\n\r\nDescription\r\n```\r\n➜  ~ kubectl describe pod heapster-7cd8dd888b-dqs9h\r\nName:           heapster-7cd8dd888b-dqs9h\r\nNamespace:      kube-system\r\nNode:           aks-agentpool-38881274-0/10.240.0.4\r\nStart Time:     Thu, 24 May 2018 09:48:14 -0500\r\nLabels:         k8s-app=heapster\r\n                pod-template-hash=3784884446\r\nAnnotations:    scheduler.alpha.kubernetes.io/critical-pod=\r\nStatus:         Running\r\nIP:             10.244.0.4\r\nControlled By:  ReplicaSet/heapster-7cd8dd888b\r\nContainers:\r\n  heapster:\r\n    Container ID:  docker://c409eac12e33ac04ccedf5d8d47777736313e0d78515244dcc8b9e07c6b07601\r\n    Image:         k8s-gcrio.azureedge.net/heapster-amd64:v1.5.1\r\n    Image ID:      docker-pullable://k8s-gcrio.azureedge.net/heapster-amd64@sha256:d4d10455d921802bdb004e7edfe423a2b2f88911319b48abf47e0af909f27f15\r\n    Port:          <none>\r\n    Host Port:     <none>\r\n    Command:\r\n      /heapster\r\n      --source=kubernetes.summary_api:\"\"\r\n    State:          Running\r\n      Started:      Thu, 24 May 2018 09:48:50 -0500\r\n    Ready:          True\r\n    Restart Count:  0\r\n    Limits:\r\n      cpu:     80m\r\n      memory:  140Mi\r\n    Requests:\r\n      cpu:        80m\r\n      memory:     140Mi\r\n    Environment:  <none>\r\n    Mounts:\r\n      /var/run/secrets/kubernetes.io/serviceaccount from heapster-token-k994b (ro)\r\n  heapster-nanny:\r\n    Container ID:  docker://2060b8cc6b09413573fcbcb8c937ca5ce5dba42ab376d5369c0a227ead5b4f6f\r\n    Image:         k8s-gcrio.azureedge.net/addon-resizer:1.8.1\r\n    Image ID:      docker-pullable://k8s-gcrio.azureedge.net/addon-resizer@sha256:507aa9845ecce1fdde4d61f530c802f4dc2974c700ce0db7730866e442db958d\r\n    Port:          <none>\r\n    Host Port:     <none>\r\n    Command:\r\n      /pod_nanny\r\n      --cpu=80m\r\n      --extra-cpu=0.5m\r\n      --memory=140Mi\r\n      --extra-memory=4Mi\r\n      --threshold=5\r\n      --deployment=heapster\r\n      --container=heapster\r\n      --poll-period=300000\r\n      --estimator=exponential\r\n    State:          Waiting\r\n      Reason:       CrashLoopBackOff\r\n    Last State:     Terminated\r\n      Reason:       Error\r\n      Exit Code:    2\r\n      Started:      Thu, 24 May 2018 10:12:17 -0500\r\n      Finished:     Thu, 24 May 2018 10:12:19 -0500\r\n    Ready:          False\r\n    Restart Count:  9\r\n    Limits:\r\n      cpu:     50m\r\n      memory:  90Mi\r\n    Requests:\r\n      cpu:     50m\r\n      memory:  90Mi\r\n    Environment:\r\n      MY_POD_NAME:       heapster-7cd8dd888b-dqs9h (v1:metadata.name)\r\n      MY_POD_NAMESPACE:  kube-system (v1:metadata.namespace)\r\n    Mounts:\r\n      /var/run/secrets/kubernetes.io/serviceaccount from heapster-token-k994b (ro)\r\nConditions:\r\n  Type           Status\r\n  Initialized    True\r\n  Ready          False\r\n  PodScheduled   True\r\nVolumes:\r\n  heapster-token-k994b:\r\n    Type:        Secret (a volume populated by a Secret)\r\n    SecretName:  heapster-token-k994b\r\n    Optional:    false\r\nQoS Class:       Guaranteed\r\nNode-Selectors:  beta.kubernetes.io/os=linux\r\nTolerations:     CriticalAddonsOnly\r\n                 node.kubernetes.io/not-ready:NoExecute for 300s\r\n                 node.kubernetes.io/unreachable:NoExecute for 300s\r\nEvents:\r\n  Type     Reason                 Age                From                               Message\r\n  ----     ------                 ----               ----                               -------\r\n  Warning  FailedScheduling       27m (x6 over 28m)  default-scheduler                  0/1 nodes are available: 1 NodeNotReady.\r\n  Normal   Scheduled              27m                default-scheduler                  Successfully assigned heapster-7cd8dd888b-dqs9h to aks-agentpool-38881274-0\r\n  Normal   SuccessfulMountVolume  27m                kubelet, aks-agentpool-38881274-0  MountVolume.SetUp succeeded for volume \"heapster-token-k994b\"\r\n  Normal   Pulling                27m                kubelet, aks-agentpool-38881274-0  pulling image \"k8s-gcrio.azureedge.net/heapster-amd64:v1.5.1\"\r\n  Normal   Pulled                 26m                kubelet, aks-agentpool-38881274-0  Successfully pulled image \"k8s-gcrio.azureedge.net/heapster-amd64:v1.5.1\"\r\n  Normal   Created                26m                kubelet, aks-agentpool-38881274-0  Created container\r\n  Normal   Pulling                26m                kubelet, aks-agentpool-38881274-0  pulling image \"k8s-gcrio.azureedge.net/addon-resizer:1.8.1\"\r\n  Normal   Started                26m                kubelet, aks-agentpool-38881274-0  Started container\r\n  Normal   Pulled                 24m                kubelet, aks-agentpool-38881274-0  Successfully pulled image \"k8s-gcrio.azureedge.net/addon-resizer:1.8.1\"\r\n  Normal   Created                23m (x4 over 24m)  kubelet, aks-agentpool-38881274-0  Created container\r\n  Normal   Started                23m (x4 over 24m)  kubelet, aks-agentpool-38881274-0  Started container\r\n  Normal   Pulled                 23m (x4 over 24m)  kubelet, aks-agentpool-38881274-0  Container image \"k8s-gcrio.azureedge.net/addon-resizer:1.8.1\" already present on machine\r\n  Warning  BackOff                2m (x97 over 24m)  kubelet, aks-agentpool-38881274-0  Back-off restarting failed container\r\n```\r\n\r\n*Heapster Logs*\r\n\r\n```\r\n➜  ~ kubectl logs heapster-7cd8dd888b-dqs9h heapster | head -20\r\nI0524 14:48:51.724146       1 heapster.go:78] /heapster --source=kubernetes.summary_api:\"\"\r\nI0524 14:48:51.724183       1 heapster.go:79] Heapster version v1.5.1\r\nI0524 14:48:51.724529       1 configs.go:61] Using Kubernetes client with master \"https://10.0.0.1:443\" and version v1\r\nI0524 14:48:51.724603       1 configs.go:62] Using kubelet port 10255\r\nI0524 14:48:51.725844       1 heapster.go:202] Starting with Metric Sink\r\nI0524 14:48:52.017454       1 heapster.go:112] Starting heapster on port 8082\r\nE0524 14:49:21.817709       1 reflector.go:190] k8s.io/heapster/metrics/util/util.go:30: Failed to list *v1.Node: Get https://10.0.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.0.0.1:443: i/o timeout\r\nE0524 14:49:21.818953       1 reflector.go:190] k8s.io/heapster/metrics/heapster.go:328: Failed to list *v1.Pod: Get https://10.0.0.1:443/api/v1/pods?resourceVersion=0: dial tcp 10.0.0.1:443: i/o timeout\r\nE0524 14:49:21.819104       1 reflector.go:190] k8s.io/heapster/metrics/util/util.go:30: Failed to list *v1.Node: Get https://10.0.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.0.0.1:443: i/o timeout\r\nE0524 14:49:21.820089       1 reflector.go:190] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:89: Failed to list *v1.Namespace: Get https://10.0.0.1:443/api/v1/namespaces?resourceVersion=0: dial tcp 10.0.0.1:443: i/o timeout\r\nE0524 14:49:21.821152       1 reflector.go:190] k8s.io/heapster/metrics/util/util.go:30: Failed to list *v1.Node: Get https://10.0.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.0.0.1:443: i/o timeout\r\nE0524 14:49:22.818368       1 reflector.go:190] k8s.io/heapster/metrics/util/util.go:30: Failed to list *v1.Node: Get https://10.0.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.0.0.1:443: getsockopt: connection refused\r\nE0524 14:49:22.919059       1 reflector.go:190] k8s.io/heapster/metrics/util/util.go:30: Failed to list *v1.Node: Get https://10.0.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.0.0.1:443: getsockopt: connection refused\r\nE0524 14:49:22.919367       1 reflector.go:190] k8s.io/heapster/metrics/util/util.go:30: Failed to list *v1.Node: Get https://10.0.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.0.0.1:443: getsockopt: connection refused\r\nE0524 14:49:22.919418       1 reflector.go:190] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:89: Failed to list *v1.Namespace: Get https://10.0.0.1:443/api/v1/namespaces?resourceVersion=0: dial tcp 10.0.0.1:443: getsockopt: connection refused\r\nE0524 14:49:22.919465       1 reflector.go:190] k8s.io/heapster/metrics/heapster.go:328: Failed to list *v1.Pod: Get https://10.0.0.1:443/api/v1/pods?resourceVersion=0: dial tcp 10.0.0.1:443: getsockopt: connection refused\r\nE0524 14:49:23.818928       1 reflector.go:190] k8s.io/heapster/metrics/util/util.go:30: Failed to list *v1.Node: Get https://10.0.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.0.0.1:443: getsockopt: connection refused\r\nE0524 14:49:23.919656       1 reflector.go:190] k8s.io/heapster/metrics/util/util.go:30: Failed to list *v1.Node: Get https://10.0.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.0.0.1:443: getsockopt: connection refused\r\nE0524 14:49:23.920552       1 reflector.go:190] k8s.io/heapster/metrics/util/util.go:30: Failed to list *v1.Node: Get https://10.0.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.0.0.1:443: getsockopt: connection refused\r\nE0524 14:49:24.017361       1 reflector.go:190] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:89: Failed to list *v1.Namespace: Get https://10.0.0.1:443/api/v1/namespaces?resourceVersion=0: dial tcp 10.0.0.1:443: getsockopt: connection refused\r\n➜  ~ kubectl logs heapster-7cd8dd888b-dqs9h heapster | tail -20\r\nW0524 15:04:47.517089       1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:89: watch of *v1.Namespace ended with: very short watch\r\nW0524 15:04:47.522124       1 reflector.go:323] k8s.io/heapster/metrics/heapster.go:328: watch of *v1.Pod ended with: very short watch\r\nW0524 15:05:48.630472       1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:89: watch of *v1.Namespace ended with: very short watch\r\nW0524 15:05:48.717416       1 reflector.go:323] k8s.io/heapster/metrics/heapster.go:328: watch of *v1.Pod ended with: very short watch\r\nW0524 15:06:49.751413       1 reflector.go:323] k8s.io/heapster/metrics/heapster.go:328: watch of *v1.Pod ended with: very short watch\r\nW0524 15:06:49.765769       1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:89: watch of *v1.Namespace ended with: very short watch\r\nW0524 15:07:50.899740       1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:89: watch of *v1.Namespace ended with: very short watch\r\nW0524 15:08:51.948755       1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:89: watch of *v1.Namespace ended with: very short watch\r\nW0524 15:09:24.878201       1 reflector.go:323] k8s.io/heapster/metrics/heapster.go:328: watch of *v1.Pod ended with: very short watch\r\nW0524 15:09:53.026347       1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:89: watch of *v1.Namespace ended with: very short watch\r\nW0524 15:10:25.943945       1 reflector.go:323] k8s.io/heapster/metrics/heapster.go:328: watch of *v1.Pod ended with: very short watch\r\nW0524 15:10:54.137800       1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:89: watch of *v1.Namespace ended with: very short watch\r\nW0524 15:11:27.040989       1 reflector.go:323] k8s.io/heapster/metrics/heapster.go:328: watch of *v1.Pod ended with: very short watch\r\nW0524 15:11:55.243451       1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:89: watch of *v1.Namespace ended with: very short watch\r\nW0524 15:12:56.437041       1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:89: watch of *v1.Namespace ended with: very short watch\r\nW0524 15:13:57.635665       1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:89: watch of *v1.Namespace ended with: very short watch\r\nW0524 15:14:34.917610       1 reflector.go:323] k8s.io/heapster/metrics/heapster.go:328: watch of *v1.Pod ended with: very short watch\r\nW0524 15:14:58.759901       1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:89: watch of *v1.Namespace ended with: very short watch\r\nW0524 15:15:36.042469       1 reflector.go:323] k8s.io/heapster/metrics/heapster.go:328: watch of *v1.Pod ended with: very short watch\r\nW0524 15:15:59.853125       1 reflector.go:323] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:89: watch of *v1.Namespace ended with: very short watch\r\n```\r\n\r\n*Heapster Nanny Logs*\r\n\r\n```\r\n➜  ~ kubectl logs heapster-7cd8dd888b-dqs9h heapster-nanny\r\nERROR: logging before flag.Parse: I0524 15:12:19.220717       1 pod_nanny.go:64] Invoked by [/pod_nanny --cpu=80m --extra-cpu=0.5m --memory=140Mi --extra-memory=4Mi --threshold=5 --deployment=heapster --container=heapster --poll-period=300000 --estimator=exponential]\r\nERROR: logging before flag.Parse: I0524 15:12:19.220797       1 pod_nanny.go:76] Watching namespace: kube-system, pod: heapster-7cd8dd888b-dqs9h, container: heapster.\r\nERROR: logging before flag.Parse: I0524 15:12:19.220804       1 pod_nanny.go:77] storage: MISSING, extra_storage: 0Gi\r\nERROR: logging before flag.Parse: I0524 15:12:19.318199       1 pod_nanny.go:162] Failed to read data from config file \"MISSING/NannyConfiguration\": open MISSING/NannyConfiguration: no such file or directory, using default parameters\r\nERROR: logging before flag.Parse: I0524 15:12:19.318347       1 pod_nanny.go:166]\r\nERROR: logging before flag.Parse: I0524 15:12:19.318443       1 pod_nanny.go:101] cpu: , extra_cpu: , memory: , extra_memory:\r\npanic: cannot parse '': quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'\r\n\r\ngoroutine 1 [running]:\r\nk8s.io/autoscaler/addon-resizer/vendor/k8s.io/apimachinery/pkg/api/resource.MustParse(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)\r\n\t/usr/local/google/home/kawych/go-tools/src/k8s.io/autoscaler/addon-resizer/vendor/k8s.io/apimachinery/pkg/api/resource/quantity.go:136 +0x253\r\nmain.main()\r\n\t/usr/local/google/home/kawych/go-tools/src/k8s.io/autoscaler/addon-resizer/nanny/main/pod_nanny.go:108 +0x6d5\r\n```"},"comment":{"url":"https://api.github.com/repos/Azure/AKS/issues/comments/397319506","html_url":"https://github.com/Azure/AKS/issues/387#issuecomment-397319506","issue_url":"https://api.github.com/repos/Azure/AKS/issues/387","id":397319506,"node_id":"MDEyOklzc3VlQ29tbWVudDM5NzMxOTUwNg==","user":{"login":"Mumrau","id":14805850,"node_id":"MDQ6VXNlcjE0ODA1ODUw","avatar_url":"https://avatars1.githubusercontent.com/u/14805850?v=4","gravatar_id":"","url":"https://api.github.com/users/Mumrau","html_url":"https://github.com/Mumrau","followers_url":"https://api.github.com/users/Mumrau/followers","following_url":"https://api.github.com/users/Mumrau/following{/other_user}","gists_url":"https://api.github.com/users/Mumrau/gists{/gist_id}","starred_url":"https://api.github.com/users/Mumrau/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Mumrau/subscriptions","organizations_url":"https://api.github.com/users/Mumrau/orgs","repos_url":"https://api.github.com/users/Mumrau/repos","events_url":"https://api.github.com/users/Mumrau/events{/privacy}","received_events_url":"https://api.github.com/users/Mumrau/received_events","type":"User","site_admin":false},"created_at":"2018-06-14T14:38:59Z","updated_at":"2018-06-14T14:38:59Z","author_association":"NONE","body":"I just wanted to point out that I did not perform deep testing over a fresh cluster, but it indeed looks fixed on their side.\r\n\r\nPretty sure this issue can be closed."}}
{"actor":{"display_login":"Mumrau"}}