{"action":"opened","issue":{"url":"https://api.github.com/repos/cilium/cilium/issues/5744","repository_url":"https://api.github.com/repos/cilium/cilium","labels_url":"https://api.github.com/repos/cilium/cilium/issues/5744/labels{/name}","comments_url":"https://api.github.com/repos/cilium/cilium/issues/5744/comments","events_url":"https://api.github.com/repos/cilium/cilium/issues/5744/events","html_url":"https://github.com/cilium/cilium/issues/5744","id":365557930,"node_id":"MDU6SXNzdWUzNjU1NTc5MzA=","number":5744,"title":"Connectivity issues during upgrades ","user":{"login":"eloycoto","id":868013,"node_id":"MDQ6VXNlcjg2ODAxMw==","avatar_url":"https://avatars1.githubusercontent.com/u/868013?v=4","gravatar_id":"","url":"https://api.github.com/users/eloycoto","html_url":"https://github.com/eloycoto","followers_url":"https://api.github.com/users/eloycoto/followers","following_url":"https://api.github.com/users/eloycoto/following{/other_user}","gists_url":"https://api.github.com/users/eloycoto/gists{/gist_id}","starred_url":"https://api.github.com/users/eloycoto/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/eloycoto/subscriptions","organizations_url":"https://api.github.com/users/eloycoto/orgs","repos_url":"https://api.github.com/users/eloycoto/repos","events_url":"https://api.github.com/users/eloycoto/events{/privacy}","received_events_url":"https://api.github.com/users/eloycoto/received_events","type":"User","site_admin":false},"labels":[{"id":721366506,"node_id":"MDU6TGFiZWw3MjEzNjY1MDY=","url":"https://api.github.com/repos/cilium/cilium/labels/area/daemon","name":"area/daemon","color":"089dd8","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-10-01T17:13:29Z","updated_at":"2018-10-01T17:13:29Z","closed_at":null,"author_association":"MEMBER","body":"Hi,\r\n\r\nUpdate connectivity issues:\r\n\r\n**To replicate:**\r\n\r\n\r\n```\r\ncd test/\r\nvagrant destroy -f\r\nvagrant up k8s1-1.11\r\nvagrant ssh k8s1-1.11\r\nkubectl apply -f https://gist.githubusercontent.com/eloycoto/cccbfcb5137f52cd4cd65e58934ca0de/raw/d8c7eee1207bf7bd4d243a57b81cafd8c530e3e5/cilium1.2.yaml\r\n```\r\n\r\nWaiting until cilium is ready. And install sample containers + Policy.\r\n```\r\nkubectl apply -f https://gist.githubusercontent.com/eloycoto/cccbfcb5137f52cd4cd65e58934ca0de/raw/d8c7eee1207bf7bd4d243a57b81cafd8c530e3e5/demo_extended.yaml\r\nkubectl apply -f https://gist.githubusercontent.com/eloycoto/cccbfcb5137f52cd4cd65e58934ca0de/raw/d8c7eee1207bf7bd4d243a57b81cafd8c530e3e5/l3-l4-policy-extended.yaml\r\n```\r\n\r\nDownload script to make background connectivity test.\r\n```\r\ncd /tmp/\r\nwget https://gist.githubusercontent.com/eloycoto/cccbfcb5137f52cd4cd65e58934ca0de/raw/d8c7eee1207bf7bd4d243a57b81cafd8c530e3e5/run_script.sh\r\nchmod 777 run_script.sh\r\n```\r\n\r\nRun two new splits, with the following parameters:\r\n\r\n```\r\n/tmp/run_script.sh $APP2 $APP4_IP\r\n/tmp/run_script.sh $APP2 $APP4_SERVICE_IP\r\n```\r\n\r\nIf curl can't connect wil return a error message, the normal thing is that no\r\nissues here.\r\n\r\nUpdate cilium to master:\r\n\r\n```\r\nkubectl apply -f https://gist.githubusercontent.com/eloycoto/cccbfcb5137f52cd4cd65e58934ca0de/raw/d8c7eee1207bf7bd4d243a57b81cafd8c530e3e5/cilium.yaml\r\n```\r\n\r\nYou will see that some http request will fail to connect to app4 services.\r\n\r\n*Bonus point*\r\n\r\nYou can also try from app3 to \"-k https://192.168.36.11:6443/api/\" it fails more\r\nthan app4.\r\n\r\nPoints that I already debug:\r\n\r\n- Connectivity issues from app2 to app4 service only happens when upgrade,\r\n  restart works ok.  `kubectl delete -n kube-system -l k8s-app=cilium` during the run_script.sh\r\n- Services are updated correctly, no issues on that\r\n```\r\nlevel=debug msg=\"service from BPF maps is managed by K8s; will not delete it from BPF maps\" l3n4Addr=\"{IP:10.96.0.10 L4Addr:{Protocol:TCP Port:53}}\" serviceID=0 subsys=daemon\r\nlevel=debug msg=\"service from BPF maps is managed by K8s; will not delete it from BPF maps\" l3n4Addr=\"{IP:10.110.218.253 L4Addr:{Protocol:TCP Port:80}}\" serviceID=3 subsys=daemon\r\nlevel=debug msg=\"service from BPF maps is managed by K8s; will not delete it from BPF maps\" l3n4Addr=\"{IP:10.96.0.1 L4Addr:{Protocol:TCP Port:443}}\" serviceID=0 subsys=daemon\r\nlevel=debug msg=\"service from BPF maps is managed by K8s; will not delete it from BPF maps\" l3n4Addr=\"{IP:10.110.218.253 L4Addr:{Protocol:TCP Port:80}}\" serviceID=3 subsys=daemon\r\nlevel=debug msg=\"service from BPF maps is managed by K8s; will not delete it from BPF maps\" l3n4Addr=\"{IP:10.96.0.1 L4Addr:{Protocol:TCP Port:443}}\" serviceID=0 subsys=daemon\r\nlevel=debug msg=\"service from BPF maps is managed by K8s; will not delete it from BPF maps\" l3n4Addr=\"{IP:10.96.0.10 L4Addr:{Protocol:TCP Port:53}}\" serviceID=0 subsys=daemon\r\nlevel=debug msg=\"service from BPF maps is managed by K8s; will not delete it from BPF maps\" l3n4Addr=\"{IP:10.111.215.168 L4Addr:{Protocol:TCP Port:80}}\" serviceID=2 subsys=daemon\r\nlevel=debug msg=\"service from BPF maps is managed by K8s; will not delete it from BPF maps\" l3n4Addr=\"{IP:10.111.215.168 L4Addr:{Protocol:TCP Port:80}}\" serviceID=2 subsys=daemon\r\nlevel=debug msg=\"service from BPF maps is managed by K8s; will not delete it from BPF maps\" l3n4Addr=\"{IP:10.111.215.168 L4Addr:{Protocol:TCP Port:80}}\" serviceID=2 subsys=daemon\r\n```\r\n    - Full log: https://gist.github.com/eloycoto/84b0ad7a92817da01d0bd422e782f7b1\r\n- No init policy related, endpoints are updating correctly.\r\n    -  I reviewed `func (d *Daemon) regenerateRestoredEndpoints(state *endpointRestoreState) {` and it's ok, all LGTM.\r\n\r\nWant to trigger using ginkgo test? Please checkout PR https://github.com/cilium/cilium/pull/5668\r\n"}}
{"actor":{"display_login":"eloycoto"}}