{"action":"created","issue":{"url":"https://api.github.com/repos/tigera/canal/issues/24","repository_url":"https://api.github.com/repos/tigera/canal","labels_url":"https://api.github.com/repos/tigera/canal/issues/24/labels{/name}","comments_url":"https://api.github.com/repos/tigera/canal/issues/24/comments","events_url":"https://api.github.com/repos/tigera/canal/issues/24/events","html_url":"https://github.com/tigera/canal/issues/24","id":190425588,"number":24,"title":"Canal self-hosted failing - not properly authenticating with Kubernetes","user":{"login":"micahhausler","id":791000,"avatar_url":"https://avatars.githubusercontent.com/u/791000?v=3","gravatar_id":"","url":"https://api.github.com/users/micahhausler","html_url":"https://github.com/micahhausler","followers_url":"https://api.github.com/users/micahhausler/followers","following_url":"https://api.github.com/users/micahhausler/following{/other_user}","gists_url":"https://api.github.com/users/micahhausler/gists{/gist_id}","starred_url":"https://api.github.com/users/micahhausler/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/micahhausler/subscriptions","organizations_url":"https://api.github.com/users/micahhausler/orgs","repos_url":"https://api.github.com/users/micahhausler/repos","events_url":"https://api.github.com/users/micahhausler/events{/privacy}","received_events_url":"https://api.github.com/users/micahhausler/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-11-18T21:26:07Z","updated_at":"2016-11-21T16:51:01Z","closed_at":null,"body":"I am unable to get self-hosted (or manual install) of Canal to authenticate with Kubernetes.\r\n\r\n**Cloud provider:** AWS (VPC)\r\n**OS:** Coreos 1185.3.0\r\n**Kernel:** ```Linux ip-172-31-7-173.us-west-2.compute.internal 4.7.3-coreos-r2 #1 SMP Tue Nov 1 01:38:43 UTC 2016 x86_64 Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz GenuineIntel GNU/Linux```\r\n**Kubernetes version:** v1.4.6\r\n**Docker version:** 1.11.2\r\n\r\nMy master and node images boot with basically just the required Kubernetes components (kubelet, kube-proxy, and the control-panel portions on the masters)\r\n\r\nI've also set:\r\n```\r\n            --network-plugin-dir=/etc/cni/net.d \\\r\n            --network-plugin=cni \r\n``` \r\non my kubelet.\r\n\r\nI'm first tried the files provided for [`canal.yaml`](https://github.com/tigera/canal/blob/3ddffe051ecf1622c5462ec0747540b73e969790/k8s-install/canal.yaml) and [`config.yaml`](https://github.com/tigera/canal/blob/89bd4fd8bbf2b2c33653e262b2c820a3b84a4cb8/k8s-install/config.yaml) without modification, without any success.\r\n\r\nI've also tried it with the following modifications:\r\n- Flannel image: `quay.io/coreos/flannel:v0.6.2`\r\n- calico-node image: `quay.io/calico/node:v1.0.0-beta-rc5`\r\n- install-calico-cni image: `calico/cni:v1.5.1`\r\n- kube-policy-controller image: `calico/kube-policy-controller:v0.5.0`\r\n\r\nI turned the `delegate.log_level` key to `DEBUG` in `cni_network_config` and set the `FELIX_LOGSEVERITYSCREEN=DEBUG` environment variable for calico-node\r\n\r\nHere is the output from the kubelet log when trying to create a simple nginx pod\r\n```\r\nNov 18 21:13:20 ip-172-31-7-173.us-west-2.compute.internal kubelet[4005]: E1118 21:13:20.795558    4005 docker_manager.go:746] Logging security options: {key:seccomp value:unconfined msg:}\r\nNov 18 21:13:20 ip-172-31-7-173.us-west-2.compute.internal kubelet[4005]: I1118 21:13:20.976330   24146 utils.go:224] libcalico glog logging configured\r\nNov 18 21:13:20 ip-172-31-7-173.us-west-2.compute.internal kubelet[4005]: time=\"2016-11-18T21:13:20Z\" level=info msg=\"Extracted identifiers\" Node=ip-172-31-7-173.us-west-2.compute.internal Orchestrator=k8s Workload=default.nginx-2226242539-1z7xp\r\nNov 18 21:13:20 ip-172-31-7-173.us-west-2.compute.internal kubelet[4005]: time=\"2016-11-18T21:13:20Z\" level=info msg=\"Loaded CNI NetConf\" NetConfg={canal calico { host-local 192.168.127.0/24 <nil> <nil>} 8951    http://127.0.0.1:2379 DEBUG {k8s https://10.100.0.1:443    } { /etc/cni/net.d/calico-kubeconfig } {{{ {[]}}}}    } Workload=default.nginx-2226242539-1z7xp\r\nNov 18 21:13:20 ip-172-31-7-173.us-west-2.compute.internal kubelet[4005]: time=\"2016-11-18T21:13:20Z\" level=info msg=\"Configured environment: [PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin JOURNAL_STREAM=8:23556 CNI_COMMAND=ADD CNI_CONTAINERID=8843fa036492e449ad2948d2b4fe6e3d87cc13e920122e7eaa8b27870ef4b92c CNI_NETNS=/proc/24112/ns/net CNI_ARGS=IgnoreUnknown=1;IgnoreUnknown=1;K8S_POD_NAMESPACE=default;K8S_POD_NAME=nginx-2226242539-1z7xp;K8S_POD_INFRA_CONTAINER_ID=8843fa036492e449ad2948d2b4fe6e3d87cc13e920122e7eaa8b27870ef4b92c CNI_IFNAME=eth0 CNI_PATH=/opt/cni/bin:/opt/flannel/bin ETCD_ENDPOINTS=http://127.0.0.1:2379 KUBECONFIG=/etc/cni/net.d/calico-kubeconfig]\"\r\nNov 18 21:13:20 ip-172-31-7-173.us-west-2.compute.internal kubelet[4005]: time=\"2016-11-18T21:13:20Z\" level=info msg=\"No config file specified, loading config from environment\"\r\nNov 18 21:13:20 ip-172-31-7-173.us-west-2.compute.internal kubelet[4005]: time=\"2016-11-18T21:13:20Z\" level=info msg=\"Datastore type: etcdv2\"\r\nNov 18 21:13:20 ip-172-31-7-173.us-west-2.compute.internal kubelet[4005]: time=\"2016-11-18T21:13:20Z\" level=debug msg=\"Using datastore type 'etcdv2'\"\r\nNov 18 21:13:20 ip-172-31-7-173.us-west-2.compute.internal kubelet[4005]: time=\"2016-11-18T21:13:20Z\" level=debug msg=\"List Key: /calico/v1/host/ip-172-31-7-173.us-west-2.compute.internal/workload/k8s/default.nginx-2226242539-1z7xp/endpoint\"\r\nNov 18 21:13:20 ip-172-31-7-173.us-west-2.compute.internal kubelet[4005]: time=\"2016-11-18T21:13:20Z\" level=debug msg=\"Key not found error\"\r\nNov 18 21:13:20 ip-172-31-7-173.us-west-2.compute.internal kubelet[4005]: time=\"2016-11-18T21:13:20Z\" level=debug msg=\"Retrieved endpoints: &{{workloadEndpointList v1} {} []}\" Workload=default.nginx-2226242539-1z7xp\r\nNov 18 21:13:20 ip-172-31-7-173.us-west-2.compute.internal kubelet[4005]: Calico CNI checking for existing endpoint: <nil>\r\nNov 18 21:13:20 ip-172-31-7-173.us-west-2.compute.internal kubelet[4005]: I1118 21:13:20.984454   24146 utils.go:224] libcalico glog logging configured\r\nNov 18 21:13:20 ip-172-31-7-173.us-west-2.compute.internal kubelet[4005]: time=\"2016-11-18T21:13:20Z\" level=info msg=\"Extracted identifiers for CmdAddK8s\" Node=ip-172-31-7-173.us-west-2.compute.internal Orchestrator=k8s Workload=default.nginx-2226242539-1z7xp\r\nNov 18 21:13:20 ip-172-31-7-173.us-west-2.compute.internal kubelet[4005]: I1118 21:13:20.985343   24146 loader.go:329] Config loaded from file /etc/cni/net.d/calico-kubeconfig\r\nNov 18 21:13:20 ip-172-31-7-173.us-west-2.compute.internal kubelet[4005]: time=\"2016-11-18T21:13:20Z\" level=debug msg=\"Kubernetes config &{https://10.100.0.1:443   {  <nil> <nil>}     <nil> <nil> {   [] [] []} true  <nil> <nil> 0 0 <nil>}\" Workload=default.nginx-2226242539-1z7xp\r\nNov 18 21:13:20 ip-172-31-7-173.us-west-2.compute.internal kubelet[4005]: time=\"2016-11-18T21:13:20Z\" level=debug msg=\"Created Kubernetes client\" Workload=default.nginx-2226242539-1z7xp client=&{0xc420386fa0 0xc420022198 0xc4200221a0 0xc4200221a8 0xc4200221b0 0xc4200221b8 0xc4200221c0}\r\nNov 18 21:13:20 ip-172-31-7-173.us-west-2.compute.internal kubelet[4005]: time=\"2016-11-18T21:13:20Z\" level=debug msg=\"Calling IPAM plugin host-local\" Workload=default.nginx-2226242539-1z7xp\r\nNov 18 21:13:20 ip-172-31-7-173.us-west-2.compute.internal kubelet[4005]: time=\"2016-11-18T21:13:20Z\" level=debug msg=\"IPAM plugin returned: IP4:{IP:{IP:192.168.127.2 Mask:ffffff00} Gateway:192.168.127.1 Routes:[{Dst:{IP:192.168.0.0 Mask:ffff0000} GW:<nil>}]}, DNS:{Nameservers:[] Domain: Search:[] Options:[]}\" Workload=default.nginx-2226242539-1z7xp\r\nNov 18 21:13:20 ip-172-31-7-173.us-west-2.compute.internal kubelet[4005]: time=\"2016-11-18T21:13:20Z\" level=info msg=\"Populated endpoint\" Workload=default.nginx-2226242539-1z7xp endpoint=&{{workloadEndpoint v1} {{} eth0 default.nginx-2226242539-1z7xp k8s ip-172-31-7-173.us-west-2.compute.internal map[]} {[192.168.127.2/32] [] <nil> <nil> [k8s_ns.default]  <nil>}}\r\nNov 18 21:13:20 ip-172-31-7-173.us-west-2.compute.internal kubelet[4005]: I1118 21:13:20.990913   24146 round_trippers.go:299] curl -k -v -XGET  -H \"Accept: application/json, */*\" -H \"User-Agent: calico/v0.0.0 (linux/amd64) kubernetes/$Format\" https://10.100.0.1:443/api/v1/namespaces/default/pods/nginx-2226242539-1z7xp\r\nNov 18 21:13:21 ip-172-31-7-173.us-west-2.compute.internal kubelet[4005]: I1118 21:13:21.063241   24146 round_trippers.go:318] GET https://10.100.0.1:443/api/v1/namespaces/default/pods/nginx-2226242539-1z7xp 401 Unauthorized in 72 milliseconds\r\nNov 18 21:13:21 ip-172-31-7-173.us-west-2.compute.internal kubelet[4005]: I1118 21:13:21.063259   24146 round_trippers.go:324] Response Headers:\r\nNov 18 21:13:21 ip-172-31-7-173.us-west-2.compute.internal kubelet[4005]: I1118 21:13:21.063266   24146 round_trippers.go:327]     Content-Type: text/plain; charset=utf-8\r\nNov 18 21:13:21 ip-172-31-7-173.us-west-2.compute.internal kubelet[4005]: I1118 21:13:21.063272   24146 round_trippers.go:327]     X-Content-Type-Options: nosniff\r\nNov 18 21:13:21 ip-172-31-7-173.us-west-2.compute.internal kubelet[4005]: I1118 21:13:21.063277   24146 round_trippers.go:327]     Date: Fri, 18 Nov 2016 21:13:21 GMT\r\nNov 18 21:13:21 ip-172-31-7-173.us-west-2.compute.internal kubelet[4005]: I1118 21:13:21.063282   24146 round_trippers.go:327]     Content-Length: 13\r\nNov 18 21:13:21 ip-172-31-7-173.us-west-2.compute.internal kubelet[4005]: I1118 21:13:21.065677   24146 request.go:908] Response Body: Unauthorized\r\nNov 18 21:13:21 ip-172-31-7-173.us-west-2.compute.internal kubelet[4005]: I1118 21:13:21.065699   24146 request.go:998] Response Body: \"Unauthorized\\n\"\r\nNov 18 21:13:21 ip-172-31-7-173.us-west-2.compute.internal kubelet[4005]: time=\"2016-11-18T21:13:21Z\" level=info msg=\"Cleaning up IP allocations for failed ADD\" Workload=default.nginx-2226242539-1z7xp\r\nNov 18 21:13:21 ip-172-31-7-173.us-west-2.compute.internal kubelet[4005]: E1118 21:13:21.071867    4005 cni.go:255] Error adding network: the server has asked for the client to provide credentials (get pods nginx-2226242539-1z7xp)\r\nNov 18 21:13:21 ip-172-31-7-173.us-west-2.compute.internal kubelet[4005]: E1118 21:13:21.072209    4005 cni.go:209] Error while adding to cni network: the server has asked for the client to provide credentials (get pods nginx-2226242539-1z7xp)\r\nNov 18 21:13:21 ip-172-31-7-173.us-west-2.compute.internal kubelet[4005]: E1118 21:13:21.072508    4005 docker_manager.go:2127] Failed to setup network for pod \"nginx-2226242539-1z7xp_default(ebf28f96-adc9-11e6-bb8b-0a672f8dd7e7)\" using network plugins \"cni\": the server has asked for the client to provide credentials (get pods nginx-2226242539-1z7xp); Skipping pod\r\nNov 18 21:13:21 ip-172-31-7-173.us-west-2.compute.internal kubelet[4005]: E1118 21:13:21.126666    4005 pod_workers.go:184] Error syncing pod ebf28f96-adc9-11e6-bb8b-0a672f8dd7e7, skipping: failed to \"SetupNetwork\" for \"nginx-2226242539-1z7xp_default\" with SetupNetworkError: \"Failed to setup network for pod \\\"nginx-2226242539-1z7xp_default(ebf28f96-adc9-11e6-bb8b-0a672f8dd7e7)\\\" using network plugins \\\"cni\\\": the server has asked for the client to provide credentials (get pods nginx-2226242539-1z7xp); Skipping pod\"\r\nNov 18 21:13:21 ip-172-31-7-173.us-west-2.compute.internal kubelet[4005]: E1118 21:13:21.127577    4005 kubelet_getters.go:249] Could not read directory /var/lib/kubelet/pods/4af5f5c1-adcb-11e6-bb8b-0a672f8dd7e7/volumes: open /var/lib/kubelet/pods/4af5f5c1-adcb-11e6-bb8b-0a672f8dd7e7/volumes: no such file or directory\r\nNov 18 21:13:21 ip-172-31-7-173.us-west-2.compute.internal kubelet[4005]: E1118 21:13:21.127592    4005 kubelet_volumes.go:159] Orphaned pod \"4af5f5c1-adcb-11e6-bb8b-0a672f8dd7e7\" found, but error open /var/lib/kubelet/pods/4af5f5c1-adcb-11e6-bb8b-0a672f8dd7e7/volumes: no such file or directory occured during reading volume dir from disk\r\n```\r\n\r\nHere is the yaml for an example pod \r\n\r\n```yaml\r\napiVersion: v1\r\nkind: Service\r\nmetadata:\r\n  labels:\r\n    app: nginx\r\n  name: nginx\r\n  namespace: default\r\nspec:\r\n  selector:\r\n    app: nginx-default\r\n  ports:\r\n  - name: https\r\n    port: 80\r\n    protocol: TCP\r\n    targetPort: 80\r\n  type: NodePort\r\n---\r\napiVersion: extensions/v1beta1\r\nkind: Deployment\r\nmetadata:\r\n  name: nginx\r\n  namespace: default\r\nspec:\r\n  replicas: 3\r\n  selector:\r\n    matchLabels:\r\n      app: nginx-default\r\n  template:\r\n    metadata:\r\n      labels:\r\n        app: nginx-default\r\n      name: nginx\r\n    spec:\r\n      containers:\r\n      - image: \"nginx:alpine\"\r\n        name: nginx\r\n        ports:\r\n        - containerPort: 80\r\n          protocol: TCP\r\n```\r\n\r\nThe pod is stuck in the `ContainerCreating`, and the events output from `kubectl describe pod` shows:\r\n```\r\n  1h\t1s\t1576\t{kubelet ip-172-31-7-173.us-west-2.compute.internal}\t\tWarning\tFailedSync\tError syncing pod, skipping: failed to \"SetupNetwork\" for \"nginx-2226242539-1z7xp_default\" with SetupNetworkError: \"Failed to setup network for pod \\\"nginx-2226242539-1z7xp_default(ebf28f96-adc9-11e6-bb8b-0a672f8dd7e7)\\\" using network plugins \\\"cni\\\": the server has asked for the client to provide credentials (get pods nginx-2226242539-1z7xp); Skipping pod\"\r\n```\r\n\r\nIt looks like an IP address is being allocated, but when querying kubernetes, the authorization fails. The file `/etc/cni/net.d/10-canal.conf` is getting properly populated with a valid k8s token.\r\n\r\nI ran \r\n```bash\r\ncurl \\\r\n    -H \"Authorization: Bearer $(cat /etc/cni/net.d/10-canal.conf |jq -r .delegate.policy.k8s_api_token)\" \\\r\n     https://10.100.0.1:443/api/v1/namespaces/default/pods/nginx-2226242539-1z7xp\r\n``` \r\nand got back a response. "},"comment":{"url":"https://api.github.com/repos/tigera/canal/issues/comments/261996046","html_url":"https://github.com/tigera/canal/issues/24#issuecomment-261996046","issue_url":"https://api.github.com/repos/tigera/canal/issues/24","id":261996046,"user":{"login":"micahhausler","id":791000,"avatar_url":"https://avatars.githubusercontent.com/u/791000?v=3","gravatar_id":"","url":"https://api.github.com/users/micahhausler","html_url":"https://github.com/micahhausler","followers_url":"https://api.github.com/users/micahhausler/followers","following_url":"https://api.github.com/users/micahhausler/following{/other_user}","gists_url":"https://api.github.com/users/micahhausler/gists{/gist_id}","starred_url":"https://api.github.com/users/micahhausler/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/micahhausler/subscriptions","organizations_url":"https://api.github.com/users/micahhausler/orgs","repos_url":"https://api.github.com/users/micahhausler/repos","events_url":"https://api.github.com/users/micahhausler/events{/privacy}","received_events_url":"https://api.github.com/users/micahhausler/received_events","type":"User","site_admin":false},"created_at":"2016-11-21T16:51:01Z","updated_at":"2016-11-21T16:51:01Z","body":"Also, as part of my kubelet boostrap, I'm installing v0.3.0 of https://github.com/containernetworking/cni/ into `/opt/cni/bin/`"}}
{"actor":{"display_login":"micahhausler"}}