{"action":"created","issue":{"url":"https://api.github.com/repos/openfaas/faas/issues/553","repository_url":"https://api.github.com/repos/openfaas/faas","labels_url":"https://api.github.com/repos/openfaas/faas/issues/553/labels{/name}","comments_url":"https://api.github.com/repos/openfaas/faas/issues/553/comments","events_url":"https://api.github.com/repos/openfaas/faas/issues/553/events","html_url":"https://github.com/openfaas/faas/issues/553","id":301982111,"number":553,"title":"Troubleshooting question for faas-netes","user":{"login":"saad749","id":7619612,"avatar_url":"https://avatars0.githubusercontent.com/u/7619612?v=4","gravatar_id":"","url":"https://api.github.com/users/saad749","html_url":"https://github.com/saad749","followers_url":"https://api.github.com/users/saad749/followers","following_url":"https://api.github.com/users/saad749/following{/other_user}","gists_url":"https://api.github.com/users/saad749/gists{/gist_id}","starred_url":"https://api.github.com/users/saad749/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/saad749/subscriptions","organizations_url":"https://api.github.com/users/saad749/orgs","repos_url":"https://api.github.com/users/saad749/repos","events_url":"https://api.github.com/users/saad749/events{/privacy}","received_events_url":"https://api.github.com/users/saad749/received_events","type":"User","site_admin":false},"labels":[{"id":505489879,"url":"https://api.github.com/repos/openfaas/faas/labels/question","name":"question","color":"cc317c","default":true},{"id":850118088,"url":"https://api.github.com/repos/openfaas/faas/labels/support","name":"support","color":"b85ffc","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":7,"created_at":"2018-03-03T08:59:30Z","updated_at":"2018-03-03T12:31:04Z","closed_at":null,"author_association":"NONE","body":"## Title\r\n\r\nQueue worker and gateway pods in CrashLoopBackOff \r\n\r\n## Expected Behaviour\r\nThe Queue worker and gateway should be running.\r\n\r\n## Current Behaviour\r\nGateway and Queue worker are restarting and in CrashLoopBackOff status.\r\n\r\n\r\n## Steps to Reproduce (for bugs)\r\n\r\n1. Configure a 2 Node Kubernetes cluster with Weave\r\n2. curl -sL https://cli.openfaas.com | sudo sh\r\n3. git clone https://github.com/openfaas/faas-netes\r\n4. cd faas-netes && kubectl apply -f ./namespaces.yml,./yaml\r\n5. kubectl get pod --all-namespaces -o wide\r\n\r\n```\r\nkube-master@ubuntu:~/faas-netes$ kubectl get pod --all-namespaces -o wide\r\nNAMESPACE     NAME                                    READY     STATUS             RESTARTS   AGE       IP               NODE\r\nkube-system   etcd-ubuntu                             1/1       Running            0          11h       192.168.100.17   ubuntu\r\nkube-system   kube-apiserver-ubuntu                   1/1       Running            0          11h       192.168.100.17   ubuntu\r\nkube-system   kube-controller-manager-ubuntu          1/1       Running            0          11h       192.168.100.17   ubuntu\r\nkube-system   kube-dns-6f4fd4bdf-xmncp                3/3       Running            0          11h       10.32.0.9        ubuntu\r\nkube-system   kube-proxy-dlbgt                        1/1       Running            1          11h       192.168.100.12   kminion\r\nkube-system   kube-proxy-lv7r7                        1/1       Running            0          11h       192.168.100.17   ubuntu\r\nkube-system   kube-scheduler-ubuntu                   1/1       Running            0          11h       192.168.100.17   ubuntu\r\nkube-system   kubernetes-dashboard-5bd6f767c7-vssxs   1/1       Running            0          9h        10.36.0.3        kminion\r\nkube-system   weave-net-fhpdr                         2/2       Running            0          11h       192.168.100.17   ubuntu\r\nkube-system   weave-net-x7qv5                         2/2       Running            3          11h       192.168.100.12   kminion\r\nopenfaas      alertmanager-76c4595457-44l55           1/1       Running            0          1m        10.36.0.4        kminion\r\nopenfaas      faas-netesd-7d4fdd494d-9lxzc            1/1       Running            0          1m        10.36.0.8        kminion\r\nopenfaas      gateway-b85589656-57749                 0/1       CrashLoopBackOff   1          1m        10.36.0.5        kminion\r\nopenfaas      nats-6c4f7df-pg8sl                      1/1       Running            0          1m        10.36.0.6        kminion\r\nopenfaas      prometheus-65b9b5c86d-t7chh             1/1       Running            0          1m        10.32.0.10       ubuntu\r\nopenfaas      queue-worker-676ccd9fb9-4rfsk           0/1       CrashLoopBackOff   1          1m        10.36.0.7        kminion\r\n```\r\n\r\n\r\n## Context\r\nTrying to use OpenFaaS on kubernetes cluster.\r\n\r\n## Your Environment\r\n* Docker version `docker version` (e.g. Docker 17.0.05 ):\r\n```\r\nClient:\r\n Version:      1.13.1\r\n API version:  1.26\r\n Go version:   go1.8.3\r\n Git commit:   092cba3\r\n Built:        Thu Oct 12 22:34:44 2017\r\n OS/Arch:      linux/amd64\r\n\r\nServer:\r\n Version:      1.13.1\r\n API version:  1.26 (minimum version 1.12)\r\n Go version:   go1.8.3\r\n Git commit:   092cba3\r\n Built:        Thu Oct 12 22:34:44 2017\r\n OS/Arch:      linux/amd64\r\n Experimental: false\r\n```\r\n\r\n* Are you using Docker Swarm or Kubernetes (FaaS-netes)? **Kubernetes (FaaS-netes)**\r\n\r\n* Operating System and version (e.g. Linux, Windows, MacOS): **Ubuntu 17.10**\r\n\r\nI have a 2 node (2 cores, 6 GB ram, 50GB HDD) Ubuntu 17.10 cluster on VMware with bridged network.\r\nubuntu - 192.168.100.17 is master\r\nkminion - 192.168.100.12 is minion\r\n\r\nAdditional Logs:\r\n\r\n```\r\nkube-master@ubuntu:~/faas-netes$ kubectl describe pod queue-worker-676ccd9fb9-4rfsk --namespace=openfaas\r\nName:           queue-worker-676ccd9fb9-4rfsk\r\nNamespace:      openfaas\r\nNode:           kminion/192.168.100.12\r\nStart Time:     Fri, 02 Mar 2018 23:45:43 -0800\r\nLabels:         app=queue-worker\r\n                pod-template-hash=2327785965\r\nAnnotations:    <none>\r\nStatus:         Running\r\nIP:             10.36.0.7\r\nControlled By:  ReplicaSet/queue-worker-676ccd9fb9\r\nContainers:\r\n  queue-worker:\r\n    Container ID:   docker://797df68166ea78156849389fd984256f4c88d7ccabe68a555f1b9757d3b0654b\r\n    Image:          functions/queue-worker:0.4\r\n    Image ID:       docker-pullable://functions/queue-worker@sha256:c74aca2dfcc2ad3a2a0f5b9bb20eaae3656850b8e92236cac96e09ded4f615bc\r\n    Port:           <none>\r\n    State:          Running\r\n      Started:      Sat, 03 Mar 2018 00:29:59 -0800\r\n    Last State:     Terminated\r\n      Reason:       Error\r\n      Exit Code:    1\r\n      Started:      Sat, 03 Mar 2018 00:24:42 -0800\r\n      Finished:     Sat, 03 Mar 2018 00:24:44 -0800\r\n    Ready:          True\r\n    Restart Count:  13\r\n    Environment:\r\n      max_inflight:          1\r\n      ack_wait:              30s\r\n      faas_function_suffix:  .openfaas-fn\r\n    Mounts:\r\n      /var/run/secrets/kubernetes.io/serviceaccount from default-token-glvcx (ro)\r\nConditions:\r\n  Type           Status\r\n  Initialized    True\r\n  Ready          True\r\n  PodScheduled   True\r\nVolumes:\r\n  default-token-glvcx:\r\n    Type:        Secret (a volume populated by a Secret)\r\n    SecretName:  default-token-glvcx\r\n    Optional:    false\r\nQoS Class:       BestEffort\r\nNode-Selectors:  <none>\r\nTolerations:     node.kubernetes.io/not-ready:NoExecute for 300s\r\n                 node.kubernetes.io/unreachable:NoExecute for 300s\r\nEvents:\r\n  Type     Reason                 Age                 From               Message\r\n  ----     ------                 ----                ----               -------\r\n  Normal   Scheduled              44m                 default-scheduler  Successfully assigned queue-worker-676ccd9fb9-4rfsk to kminion\r\n  Normal   SuccessfulMountVolume  44m                 kubelet, kminion   MountVolume.SetUp succeeded for volume \"default-token-glvcx\"\r\n  Normal   Pulled                 41m (x4 over 43m)   kubelet, kminion   Successfully pulled image \"functions/queue-worker:0.4\"\r\n  Normal   Created                41m (x4 over 43m)   kubelet, kminion   Created container\r\n  Normal   Started                41m (x4 over 43m)   kubelet, kminion   Started container\r\n  Normal   Pulling                41m (x5 over 44m)   kubelet, kminion   pulling image \"functions/queue-worker:0.4\"\r\n  Warning  BackOff                4m (x167 over 42m)  kubelet, kminion   Back-off restarting failed container\r\n```\r\n\r\n```\r\nkube-master@ubuntu:~/faas-netes$ kubectl describe pod  gateway-b85589656-57749 --namespace=openfaas\r\nName:           gateway-b85589656-57749\r\nNamespace:      openfaas\r\nNode:           kminion/192.168.100.12\r\nStart Time:     Fri, 02 Mar 2018 23:45:43 -0800\r\nLabels:         app=gateway\r\n                pod-template-hash=641145212\r\nAnnotations:    <none>\r\nStatus:         Running\r\nIP:             10.36.0.5\r\nControlled By:  ReplicaSet/gateway-b85589656\r\nContainers:\r\n  gateway:\r\n    Container ID:   docker://7a2d441ff228e97d3f9efab3d27ea31a89abcf9ee061ce3c89390f88ac0f8c5e\r\n    Image:          functions/gateway:0.7.3\r\n    Image ID:       docker-pullable://functions/gateway@sha256:3e09c909e90d77f31bd26c7bf33b4c803f80dd37159f16c3ea25be64d5bd02fc\r\n    Port:           8080/TCP\r\n    State:          Waiting\r\n      Reason:       CrashLoopBackOff\r\n    Last State:     Terminated\r\n      Reason:       Error\r\n      Exit Code:    1\r\n      Started:      Sat, 03 Mar 2018 00:50:25 -0800\r\n      Finished:     Sat, 03 Mar 2018 00:50:27 -0800\r\n    Ready:          False\r\n    Restart Count:  17\r\n    Limits:\r\n      memory:  100Mi\r\n    Requests:\r\n      memory:  100Mi\r\n    Environment:\r\n      functions_provider_url:  http://faas-netesd.openfaas.svc.cluster.local:8080/\r\n      faas_nats_address:       nats.openfaas\r\n      faas_nats_port:          4222\r\n      read_timeout:            20s\r\n      write_timeout:           20s\r\n    Mounts:\r\n      /var/run/secrets/kubernetes.io/serviceaccount from default-token-glvcx (ro)\r\nConditions:\r\n  Type           Status\r\n  Initialized    True\r\n  Ready          False\r\n  PodScheduled   True\r\nVolumes:\r\n  default-token-glvcx:\r\n    Type:        Secret (a volume populated by a Secret)\r\n    SecretName:  default-token-glvcx\r\n    Optional:    false\r\nQoS Class:       Burstable\r\nNode-Selectors:  <none>\r\nTolerations:     node.kubernetes.io/not-ready:NoExecute for 300s\r\n                 node.kubernetes.io/unreachable:NoExecute for 300s\r\nEvents:\r\n  Type     Reason   Age                From              Message\r\n  ----     ------   ----               ----              -------\r\n  Warning  BackOff  3m (x282 over 1h)  kubelet, kminion  Back-off restarting failed container\r\n```\r\n\r\n```\r\nkube-master@ubuntu:~/faas-netes$ kubectl logs gateway-b85589656-57749 -n openfaas\r\n2018/03/03 08:19:10 HTTP Read Timeout: 20s\r\n2018/03/03 08:19:10 HTTP Write Timeout: 20s\r\n2018/03/03 08:19:10 Binding to external function provider: http://faas-netesd.openfaas.svc.cluster.local:8080/\r\n2018/03/03 08:19:10 Async enabled: Using NATS Streaming.\r\n2018/03/03 08:19:10 Opening connection to nats://nats.openfaas:4222\r\n2018/03/03 08:19:12 nats: no servers available for connection\r\n```\r\n\r\n```\r\nkube-master@ubuntu:~/faas-netes$ kubectl logs queue-worker-676ccd9fb9-4rfsk -n openfaas\r\nCan't connect: nats: no servers available for connection\r\n```\r\n\r\n```\r\nifconfig -a\r\nhttps://gist.github.com/saad749/ee97be7cd5dee6af74707b76cf0d45b8\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/openfaas/faas/issues/comments/370144207","html_url":"https://github.com/openfaas/faas/issues/553#issuecomment-370144207","issue_url":"https://api.github.com/repos/openfaas/faas/issues/553","id":370144207,"user":{"login":"stefanprodan","id":3797675,"avatar_url":"https://avatars3.githubusercontent.com/u/3797675?v=4","gravatar_id":"","url":"https://api.github.com/users/stefanprodan","html_url":"https://github.com/stefanprodan","followers_url":"https://api.github.com/users/stefanprodan/followers","following_url":"https://api.github.com/users/stefanprodan/following{/other_user}","gists_url":"https://api.github.com/users/stefanprodan/gists{/gist_id}","starred_url":"https://api.github.com/users/stefanprodan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stefanprodan/subscriptions","organizations_url":"https://api.github.com/users/stefanprodan/orgs","repos_url":"https://api.github.com/users/stefanprodan/repos","events_url":"https://api.github.com/users/stefanprodan/events{/privacy}","received_events_url":"https://api.github.com/users/stefanprodan/received_events","type":"User","site_admin":false},"created_at":"2018-03-03T12:31:04Z","updated_at":"2018-03-03T12:31:04Z","author_association":"MEMBER","body":"I'm guessing is something with the CNI setup, I see in your ifconfig flannel and weave, those don't work together well. Before running OpenFaaS on your cluster try to validate that networking is being properly setup. "}}
{"actor":{"display_login":"stefanprodan"}}