{"action":"created","issue":{"url":"https://api.github.com/repos/cilium/cilium/issues/5133","repository_url":"https://api.github.com/repos/cilium/cilium","labels_url":"https://api.github.com/repos/cilium/cilium/issues/5133/labels{/name}","comments_url":"https://api.github.com/repos/cilium/cilium/issues/5133/comments","events_url":"https://api.github.com/repos/cilium/cilium/issues/5133/events","html_url":"https://github.com/cilium/cilium/pull/5133","id":348799979,"node_id":"MDExOlB1bGxSZXF1ZXN0MjA3MDY1NTEw","number":5133,"title":"test: make K8s DNS checks more robust","user":{"login":"ianvernon","id":3858271,"node_id":"MDQ6VXNlcjM4NTgyNzE=","avatar_url":"https://avatars1.githubusercontent.com/u/3858271?v=4","gravatar_id":"","url":"https://api.github.com/users/ianvernon","html_url":"https://github.com/ianvernon","followers_url":"https://api.github.com/users/ianvernon/followers","following_url":"https://api.github.com/users/ianvernon/following{/other_user}","gists_url":"https://api.github.com/users/ianvernon/gists{/gist_id}","starred_url":"https://api.github.com/users/ianvernon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ianvernon/subscriptions","organizations_url":"https://api.github.com/users/ianvernon/orgs","repos_url":"https://api.github.com/users/ianvernon/repos","events_url":"https://api.github.com/users/ianvernon/events{/privacy}","received_events_url":"https://api.github.com/users/ianvernon/received_events","type":"User","site_admin":false},"labels":[{"id":709779088,"node_id":"MDU6TGFiZWw3MDk3NzkwODg=","url":"https://api.github.com/repos/cilium/cilium/labels/area/CI","name":"area/CI","color":"bfd4f2","default":false},{"id":677243782,"node_id":"MDU6TGFiZWw2NzcyNDM3ODI=","url":"https://api.github.com/repos/cilium/cilium/labels/kind/bug/CI","name":"kind/bug/CI","color":"fc2929","default":false},{"id":403149902,"node_id":"MDU6TGFiZWw0MDMxNDk5MDI=","url":"https://api.github.com/repos/cilium/cilium/labels/pending-review","name":"pending-review","color":"fbca04","default":false},{"id":627462913,"node_id":"MDU6TGFiZWw2Mjc0NjI5MTM=","url":"https://api.github.com/repos/cilium/cilium/labels/priority/high","name":"priority/high","color":"b60205","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2018-08-08T16:26:36Z","updated_at":"2018-08-08T19:15:50Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/cilium/cilium/pulls/5133","html_url":"https://github.com/cilium/cilium/pull/5133","diff_url":"https://github.com/cilium/cilium/pull/5133.diff","patch_url":"https://github.com/cilium/cilium/pull/5133.patch"},"body":"This PR addresses flakiness in the CI that results from CoreDNS caching hostname entries for a long time. It addresses the following problem:\r\n\r\nWe have multiple tests which use the same YAML (`test/k8sT/manifests/demo_ds.yaml`) for creating resources in the CI:\r\n\r\n* `test/k8sT/Tunnels.go`\r\n* `test/k8sT/Chaos.go`\r\n* `test/k8sT/Services.go`\r\n\r\nThese create services, which are referenced via hostnames in the CI. When CoreDNS is used in the CI (in Kubernetes >= 1.11 versions), hostname --> IP mappings are cached for some time, even after the service with the same name has been created in one test, and then re-created in another test shortly thereafter. This means that if any of the above tests run right after the other, there is a possibility that the IP returned by looking up a service name which was also used in the prior test which added a service with the same name would return the old IP for the service, which is not plumbed into the Cilium datapath, while the new service IP is. As a result, this means that tests which try to access services via hostname in the CI would fail because the DNS lookup would return the old IP for the service. This is odd behavior, considering that the `ConfigMap` for CoreDNS in the CI has the setting for cache (see https://coredns.io/plugins/cache/ ) to have a TTL of 3 seconds:\r\n\r\nhttps://github.com/cilium/cilium/blob/1f7a845c49df4717bb1c62904abbaf5525a98cfb/test/provision/manifest/coredns_deployment.yaml#L88-L103\r\n\r\nTo alleviate the flakiness that results in the CI, this is addressed by the addition of functionality to wait for DNS entries to be ready by not only checking that `dig` returns an IP for the given service name, but that the IP that is returned by `dig` matches the `ClusterIP` for the service returned by Kubernetes. \r\n\r\nAs part of debugging this, I also added some additional output / commands to make debugging such issues easier if they should arise in the future. The `WaitForKubeDNSEntry` helper function also runs `dig` for the given service name to provide additional information for debugging in the logs, and additional narration of the test in `test/k8sT/Chaos.go` to make it easier to determine how far the test has progressed while it runs. Finally, it also preforms a datapath check to not only the service name, but also the ClusterIP for the given service, to make it easier to see whether there is an issue with DNS resolution, or with the Cilium datapath itself in the test `test/k8sT/Chaos.go`.\r\n\r\nIdeally, we should not see errors like the following in the CI any more after this is merged:\r\n\r\n```\r\nCannot curl from \"testclient-24crq\" to testds-service\r\nExpected command: kubectl exec -n default testclient-24crq -- curl -s -D /dev/stderr --fail --connect-timeout 3 --max-time 3 http://testds-service.default.svc.cluster.local:80/ \r\nTo succeed, but it failed:\r\nExitcode: 28 \r\nStdout:\r\n \t \r\nStderr:\r\n \t command terminated with exit code 28\r\n```\r\n\r\nSigned-off by: Ian Vernon <ian@cilium.io>\r\n\r\nFixes: #5122 \r\n\r\n<!-- Reviewable:start -->\r\n---\r\nThis change is [<img src=\"https://reviewable.io/review_button.svg\" height=\"34\" align=\"absmiddle\" alt=\"Reviewable\"/>](https://reviewable.io/reviews/cilium/cilium/5133)\r\n<!-- Reviewable:end -->\r\n"},"comment":{"url":"https://api.github.com/repos/cilium/cilium/issues/comments/411520491","html_url":"https://github.com/cilium/cilium/pull/5133#issuecomment-411520491","issue_url":"https://api.github.com/repos/cilium/cilium/issues/5133","id":411520491,"node_id":"MDEyOklzc3VlQ29tbWVudDQxMTUyMDQ5MQ==","user":{"login":"ianvernon","id":3858271,"node_id":"MDQ6VXNlcjM4NTgyNzE=","avatar_url":"https://avatars1.githubusercontent.com/u/3858271?v=4","gravatar_id":"","url":"https://api.github.com/users/ianvernon","html_url":"https://github.com/ianvernon","followers_url":"https://api.github.com/users/ianvernon/followers","following_url":"https://api.github.com/users/ianvernon/following{/other_user}","gists_url":"https://api.github.com/users/ianvernon/gists{/gist_id}","starred_url":"https://api.github.com/users/ianvernon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ianvernon/subscriptions","organizations_url":"https://api.github.com/users/ianvernon/orgs","repos_url":"https://api.github.com/users/ianvernon/repos","events_url":"https://api.github.com/users/ianvernon/events{/privacy}","received_events_url":"https://api.github.com/users/ianvernon/received_events","type":"User","site_admin":false},"created_at":"2018-08-08T19:15:50Z","updated_at":"2018-08-08T19:15:50Z","author_association":"CONTRIBUTOR","body":"Cilium unit tests panicked - this is no way related to my changes:\r\n```\r\n19:03:37 ... Panic: runtime error: invalid memory address or nil pointer dereference (PC=0xF515CA)\r\n19:03:37 \r\n19:03:37 /usr/local/go/src/runtime/panic.go:502\r\n19:03:37   in gopanic\r\n19:03:37 /usr/local/go/src/runtime/panic.go:63\r\n19:03:37   in panicmem\r\n19:03:37 /usr/local/go/src/runtime/signal_unix.go:388\r\n19:03:37   in sigpanic\r\n19:03:37 /go/src/github.com/cilium/cilium/pkg/kvstore/store/store.go:275\r\n19:03:37   in SharedStore.UpdateLocalKey\r\n19:03:37 /go/src/github.com/cilium/cilium/pkg/kvstore/store/store.go:284\r\n19:03:37   in SharedStore.UpdateLocalKeySync\r\n19:03:37 /go/src/github.com/cilium/cilium/pkg/node/local_node.go:76\r\n19:03:37   in NotifyLocalNodeUpdated\r\n19:03:37 daemon.go:1367\r\n19:03:37   in NewDaemon\r\n19:03:37 daemon_test.go:105\r\n19:03:37   in DaemonSuite.SetUpTest\r\n19:03:37 daemon_test.go:159\r\n19:03:37   in DaemonEtcdSuite.SetUpTest\r\n19:03:37 /usr/local/go/src/reflect/value.go:308\r\n19:03:37   in Value.Call\r\n19:03:37 /usr/local/go/src/runtime/asm_amd64.s:2361\r\n19:03:37   in goexit\r\n19:03:37 PANIC: daemon_test.go:157: DaemonEtcdSuite.SetUpTest\r\n19:03:37 \r\n```"}}
{"actor":{"display_login":"ianvernon"}}