{"action":"created","issue":{"url":"https://api.github.com/repos/projectcalico/calico/issues/2183","repository_url":"https://api.github.com/repos/projectcalico/calico","labels_url":"https://api.github.com/repos/projectcalico/calico/issues/2183/labels{/name}","comments_url":"https://api.github.com/repos/projectcalico/calico/issues/2183/comments","events_url":"https://api.github.com/repos/projectcalico/calico/issues/2183/events","html_url":"https://github.com/projectcalico/calico/issues/2183","id":358795791,"node_id":"MDU6SXNzdWUzNTg3OTU3OTE=","number":2183,"title":"calico-node v3.0.4 fails with CrashLoopBackOff","user":{"login":"fckbo","id":10832015,"node_id":"MDQ6VXNlcjEwODMyMDE1","avatar_url":"https://avatars2.githubusercontent.com/u/10832015?v=4","gravatar_id":"","url":"https://api.github.com/users/fckbo","html_url":"https://github.com/fckbo","followers_url":"https://api.github.com/users/fckbo/followers","following_url":"https://api.github.com/users/fckbo/following{/other_user}","gists_url":"https://api.github.com/users/fckbo/gists{/gist_id}","starred_url":"https://api.github.com/users/fckbo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fckbo/subscriptions","organizations_url":"https://api.github.com/users/fckbo/orgs","repos_url":"https://api.github.com/users/fckbo/repos","events_url":"https://api.github.com/users/fckbo/events{/privacy}","received_events_url":"https://api.github.com/users/fckbo/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-09-10T21:09:40Z","updated_at":"2018-09-11T11:34:27Z","closed_at":null,"author_association":"NONE","body":"<!--- Provide a general summary of the issue in the Title above -->\r\nHi it seems that after rebooting an running kubernetes single node install (IBM Cloud Private community edition), calico-node does not want to restart properly. \r\n\r\n## Expected Behavior\r\n<!--- If you're describing a bug, tell us what should happen -->\r\n<!--- If you're suggesting a change/improvement, tell us how it should work -->\r\ncalico-node pod running normally\r\n\r\n## Current Behavior\r\n<!--- If describing a bug, tell us what happens instead of the expected behavior -->\r\n<!--- If suggesting a change/improvement, explain the difference from current behavior -->\r\n```fbs-MacBook-Pro:cam-to-svn fb$ kubectl get pods -n kube-system\r\nNAME                                                          READY     STATUS             RESTARTS   AGE\r\nauth-apikeys-9tfvc                                            1/1       Running            5          40d\r\nauth-idp-b64tl                                                3/3       Running            12         40d\r\nauth-idp-platform-auth-cert-gen-tb7gs                         0/1       Completed          0          40d\r\nauth-pap-xbvdp                                                1/1       Running            5          40d\r\nauth-pdp-rmmmg                                                1/1       Running            4          40d\r\ncalico-kube-controllers-759f7fc556-s9z9s                      1/1       Running            4          40d\r\ncalico-node-j4t6l                                             1/2       CrashLoopBackOff   18         1h\r\ncatalog-ui-d476x                                              1/1       Running            4          40d\r\ndefault-backend-7c6d6df9d5-dwg9b                              1/1       Running            5          40d\r\nheapster-5649f84695-4bcs4                                     2/2       Running            9          40d\r\nhelm-api-688cf45dd9-k26bv                                     2/2       Running            7          40d\r\nhelm-api-helm-cert-gen-job-s724p                              0/1       Completed          0          40d\r\nhelm-api-helm-service-onboard-job-d8n2b                       0/1       Completed          0          40d\r\nhelm-repo-7ffc9544cd-6kzzh                                    1/1       Running            4          40d\r\nicp-management-ingress-bvtk2                                  1/1       Running            5          40d\r\nicp-mongodb-0                                                 1/1       Running            4          40d\r\nimage-manager-0                                               2/2       Running            10         40d\r\nk8s-etcd-192.168.61.51                                        1/1       Running            5          40d\r\nk8s-master-192.168.61.51                                      3/3       Running            16         40d\r\nk8s-proxy-192.168.61.51                                       1/1       Running            5          40d\r\nkube-dns-gtjr4                                                3/3       Running            12         40d\r\nlogging-elk-client-65745dcd68-xzkvj                           2/2       Running            10         40d\r\nlogging-elk-data-0                                            1/1       Running            4          40d\r\nlogging-elk-filebeat-ds-8cdlv                                 1/1       Running            4          40d\r\nlogging-elk-logstash-76c548744b-nt7hv                         1/1       Running            4          40d\r\nlogging-elk-master-686fbdd984-dvwlw                           1/1       Running            4          40d\r\nmariadb-0                                                     1/1       Running            5          40d\r\nmetering-dm-5b95578b8-gbwgv                                   1/1       Running            4          40d\r\nmetering-reader-px7b9                                         1/1       Running            4          40d\r\nmetering-server-566c88ff9f-9hpgf                              1/1       Running            4          40d\r\nmetering-ui-5988bc66f5-m5qcz                                  1/1       Running            4          40d\r\nmetrics-server-7f4fdb695f-bxnp9                               1/1       Running            5          40d\r\nmonitoring-exporter-6b88bcd65b-nw4k7                          1/1       Running            4          40d\r\nmonitoring-grafana-7b9f86786-htqps                            2/2       Running            10         40d\r\nmonitoring-grafana-ds-blhmj                                   0/1       Completed          2          40d\r\nmonitoring-prometheus-77d664c568-67fjr                        3/3       Running            15         40d\r\nmonitoring-prometheus-alertmanager-7fdbf6c5cd-qhff6           3/3       Running            12         40d\r\nmonitoring-prometheus-elasticsearchexporter-bc66cf47d-xlbqh   1/1       Running            5          40d\r\nmonitoring-prometheus-kubestatemetrics-855bcd8dcb-lw6fv       1/1       Running            4          40d\r\nmonitoring-prometheus-nodeexporter-m57rw                      1/1       Running            5          40d\r\nnginx-ingress-controller-78st6                                1/1       Running            4          40d\r\noidc-client-registration-s6mcb                                0/1       Completed          0          40d\r\nplatform-api-rvz4p                                            1/1       Running            4          40d\r\nplatform-deploy-w6vq2                                         1/1       Running            4          40d\r\nplatform-ui-f9wg5                                             1/1       Running            5          40d\r\nrescheduler-ffgwq                                             1/1       Running            5          40d\r\nsecurity-onboarding-llj2p                                     0/1       Completed          0          40d\r\nservice-catalog-apiserver-gp67l                               1/1       Running            5          40d\r\nservice-catalog-controller-manager-5b654dc8b8-4bc5p           1/1       Running            10         40d\r\ntiller-deploy-c59888d97-ljht5                                 1/1       Running            4          40d\r\nunified-router-q6mkn                                          1/1       Running            5          40d\r\nupdate-secrets-hggvz                                          1/1       Running            23         40d\r\n```\r\n\r\n**Using describe to get some details on the pod:**\r\n\r\n```fbs-MacBook-Pro:cam-to-svn fb$ kubectl describe pod/calico-node-j4t6l -n kube-system\r\nName:           calico-node-j4t6l\r\nNamespace:      kube-system\r\nNode:           192.168.61.51/192.168.61.51\r\nStart Time:     Mon, 10 Sep 2018 21:09:09 +0200\r\nLabels:         app=calico-node\r\n                controller-revision-hash=3780312153\r\n                pod-template-generation=1\r\nAnnotations:    kubernetes.io/psp=privileged\r\n                scheduler.alpha.kubernetes.io/critical-pod=\r\nStatus:         Running\r\nIP:             192.168.61.51\r\nControlled By:  DaemonSet/calico-node\r\nContainers:\r\n  calico-node:\r\n    Container ID:   docker://da40c79520529506185958246e024f254c28ddba2db6cda92ef7234b027e6afc\r\n    Image:          ibmcom/calico-node:v3.0.4\r\n    Image ID:       docker-pullable://ibmcom/calico-node@sha256:2a790801a535e29c6a8307e812c0b84443e2222a81ac73e627cc95e73e2a24d6\r\n    Port:           <none>\r\n    Host Port:      <none>\r\n    State:          Waiting\r\n      Reason:       CrashLoopBackOff\r\n    Last State:     Terminated\r\n      Reason:       Error\r\n      Exit Code:    1\r\n      Started:      Mon, 10 Sep 2018 22:57:21 +0200\r\n      Finished:     Mon, 10 Sep 2018 22:57:21 +0200\r\n    Ready:          False\r\n    Restart Count:  26\r\n    Liveness:       http-get http://:9099/liveness delay=10s timeout=1s period=10s #success=1 #failure=6\r\n    Readiness:      http-get http://:9099/readiness delay=0s timeout=1s period=10s #success=1 #failure=3\r\n    Environment:\r\n      ETCD_ENDPOINTS:                     <set to the key 'etcd_endpoints' of config map 'etcd-config'>    Optional: false\r\n      CALICO_NETWORKING_BACKEND:          <set to the key 'calico_backend' of config map 'calico-config'>  Optional: false\r\n      CLUSTER_TYPE:                       k8s,bgp\r\n      CALICO_DISABLE_FILE_LOGGING:        true\r\n      FELIX_DEFAULTENDPOINTTOHOSTACTION:  ACCEPT\r\n      CALICO_IPV4POOL_CIDR:               10.1.0.0/16\r\n      CALICO_IPV4POOL_IPIP:               Always\r\n      CALICO_K8S_NODE_REF:                 (v1:spec.nodeName)\r\n      FELIX_IPV6SUPPORT:                  false\r\n      FELIX_LOGSEVERITYSCREEN:            info\r\n      FELIX_IPINIPMTU:                    1430\r\n      ETCD_CA_CERT_FILE:                  <set to the key 'etcd_ca' of config map 'calico-config'>    Optional: false\r\n      ETCD_KEY_FILE:                      <set to the key 'etcd_key' of config map 'calico-config'>   Optional: false\r\n      ETCD_CERT_FILE:                     <set to the key 'etcd_cert' of config map 'calico-config'>  Optional: false\r\n      IP:                                 autodetect\r\n      FELIX_HEALTHENABLED:                true\r\n      IP_AUTODETECTION_METHOD:            can-reach=192.168.61.51\r\n    Mounts:\r\n      /calico-secrets from etcd-certs (rw)\r\n      /lib/modules from lib-modules (ro)\r\n      /var/run/calico from var-run-calico (rw)\r\n      /var/run/secrets/kubernetes.io/serviceaccount from default-token-qflkw (ro)\r\n  install-cni:\r\n    Container ID:  docker://c289c019eb662adf61cdb6a022c5389e26cd8453684554ee21bce83f293f34a6\r\n    Image:         ibmcom/calico-cni:v2.0.3\r\n    Image ID:      docker-pullable://ibmcom/calico-cni@sha256:5f21354a010b9b874dd230dafc27697061a1b27619c33a89bfac516f25bf68f7\r\n    Port:          <none>\r\n    Host Port:     <none>\r\n    Command:\r\n      /install-cni.sh\r\n    State:          Running\r\n      Started:      Mon, 10 Sep 2018 21:09:10 +0200\r\n    Ready:          True\r\n    Restart Count:  0\r\n    Environment:\r\n      CNI_CONF_NAME:       10-calico.conflist\r\n      ETCD_ENDPOINTS:      <set to the key 'etcd_endpoints' of config map 'etcd-config'>        Optional: false\r\n      CNI_NETWORK_CONFIG:  <set to the key 'cni_network_config' of config map 'calico-config'>  Optional: false\r\n    Mounts:\r\n      /calico-secrets from etcd-certs (rw)\r\n      /host/etc/cni/net.d from cni-net-dir (rw)\r\n      /host/opt/cni/bin from cni-bin-dir (rw)\r\n      /var/run/secrets/kubernetes.io/serviceaccount from default-token-qflkw (ro)\r\nConditions:\r\n  Type           Status\r\n  Initialized    True \r\n  Ready          False \r\n  PodScheduled   True \r\nVolumes:\r\n  lib-modules:\r\n    Type:          HostPath (bare host directory volume)\r\n    Path:          /lib/modules\r\n    HostPathType:  \r\n  var-run-calico:\r\n    Type:          HostPath (bare host directory volume)\r\n    Path:          /var/run/calico\r\n    HostPathType:  \r\n  cni-bin-dir:\r\n    Type:          HostPath (bare host directory volume)\r\n    Path:          /opt/cni/bin\r\n    HostPathType:  \r\n  cni-net-dir:\r\n    Type:          HostPath (bare host directory volume)\r\n    Path:          /etc/cni/net.d\r\n    HostPathType:  \r\n  etcd-certs:\r\n    Type:        Secret (a volume populated by a Secret)\r\n    SecretName:  etcd-secret\r\n    Optional:    false\r\n  default-token-qflkw:\r\n    Type:        Secret (a volume populated by a Secret)\r\n    SecretName:  default-token-qflkw\r\n    Optional:    false\r\nQoS Class:       BestEffort\r\nNode-Selectors:  <none>\r\nTolerations:     CriticalAddonsOnly\r\n                 dedicated:NoSchedule\r\n                 node.kubernetes.io/disk-pressure:NoSchedule\r\n                 node.kubernetes.io/memory-pressure:NoSchedule\r\n                 node.kubernetes.io/not-ready:NoExecute\r\n                 node.kubernetes.io/unreachable:NoExecute\r\nEvents:\r\n  Type     Reason   Age                From                    Message\r\n  ----     ------   ----               ----                    -------\r\n  Warning  BackOff  3m (x517 over 1h)  kubelet, 192.168.61.51  Back-off restarting failed container\r\n```\r\n\r\n\r\n## Possible Solution\r\n<!--- Not obligatory, but suggest a fix/reason for the bug, -->\r\n<!--- or ideas how to implement the addition or change -->\r\n1) I tried to remove the file /etc/cni/net.d/10-calico.conflist as suggested in another issue related to the same problem. After one minute the file was not re-created, I did remove the calico-node pod using kubectl delete pod command and the pod was redeployed automatically by kubernetes and the file re-created but this did not fixed the problem.\r\n\r\n2) I also installed calicoctl on the host machine on which kubernetes is running and connected it the underlying etcd. It seems to work but I do not know what to do to potentially try to understand what's wrong\r\n```root@localhost:/etc/calico# calicoctl get profile\r\nNAME              \r\nkns.default       \r\nkns.kube-public   \r\nkns.kube-system   \r\nkns.platform      \r\nkns.services\r\n```\r\n\r\n## Steps to Reproduce (for bugs)\r\n<!--- Provide a link to a live example, or an unambiguous set of steps to -->\r\n<!--- reproduce this bug. Include code to reproduce, if relevant -->\r\nDon't know\r\n\r\n## Context\r\n<!--- How has this issue affected you? What are you trying to accomplish? -->\r\n<!--- Providing context helps us come up with a solution that is most useful in the real world -->\r\nAn application running on the cluster is having issue to connect to an external system (it used to connect without any problem before this problem occur)\r\n\r\n## Your Environment\r\n<!--- Include as many relevant details about the environment you experienced the bug in -->\r\n* Calico version 3.0.4 \r\n* Orchestrator version (e.g. kubernetes, mesos, rkt): kubernetes\r\n```fbs-MacBook-Pro:cam-to-svn fb$ kubectl version\r\nClient Version: version.Info{Major:\"1\", Minor:\"10\", GitVersion:\"v1.10.0\", GitCommit:\"fc32d2f3698e36b93322a3465f63a14e9f0eaead\", GitTreeState:\"clean\", BuildDate:\"2018-03-27T00:13:02Z\", GoVersion:\"go1.9.4\", Compiler:\"gc\", Platform:\"darwin/amd64\"}\r\nServer Version: version.Info{Major:\"1\", Minor:\"10\", GitVersion:\"v1.10.0+icp\", GitCommit:\"1c6988c2361d678dab8ea5ddfab82cb52680b20b\", GitTreeState:\"clean\", BuildDate:\"2018-04-27T05:47:17Z\", GoVersion:\"go1.9.3\", Compiler:\"gc\", Platform:\"linux/amd64\"}\r\nfbs-MacBook-Pro:cam-to-svn fb$ kubetctl get nodes\r\n-bash: kubetctl: command not found\r\nfbs-MacBook-Pro:cam-to-svn fb$ kubectl get nodes\r\nNAME            STATUS    ROLES     AGE       VERSION\r\n192.168.61.51   Ready     <none>    40d       v1.10.0+icp\r\n```\r\n\r\n* Operating System and version: ubuntu 16.04 LTS\r\n* Link to your project (optional):\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/projectcalico/calico/issues/comments/420242152","html_url":"https://github.com/projectcalico/calico/issues/2183#issuecomment-420242152","issue_url":"https://api.github.com/repos/projectcalico/calico/issues/2183","id":420242152,"node_id":"MDEyOklzc3VlQ29tbWVudDQyMDI0MjE1Mg==","user":{"login":"fckbo","id":10832015,"node_id":"MDQ6VXNlcjEwODMyMDE1","avatar_url":"https://avatars2.githubusercontent.com/u/10832015?v=4","gravatar_id":"","url":"https://api.github.com/users/fckbo","html_url":"https://github.com/fckbo","followers_url":"https://api.github.com/users/fckbo/followers","following_url":"https://api.github.com/users/fckbo/following{/other_user}","gists_url":"https://api.github.com/users/fckbo/gists{/gist_id}","starred_url":"https://api.github.com/users/fckbo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fckbo/subscriptions","organizations_url":"https://api.github.com/users/fckbo/orgs","repos_url":"https://api.github.com/users/fckbo/repos","events_url":"https://api.github.com/users/fckbo/events{/privacy}","received_events_url":"https://api.github.com/users/fckbo/received_events","type":"User","site_admin":false},"created_at":"2018-09-11T11:34:27Z","updated_at":"2018-09-11T11:34:27Z","author_association":"NONE","body":"**@caseydavenport** \r\nthx a lot for your question, as it often happens, it generate more thinking & investigation and reading through the logs,  finally the pb was damn simple, the hostname of my machine was not persisted in  /etc/hostname file. What I can't understand is how it did work for so long...."}}
{"actor":{"display_login":"fckbo"}}