{"action":"created","issue":{"url":"https://api.github.com/repos/Azure/AKS/issues/676","repository_url":"https://api.github.com/repos/Azure/AKS","labels_url":"https://api.github.com/repos/Azure/AKS/issues/676/labels{/name}","comments_url":"https://api.github.com/repos/Azure/AKS/issues/676/comments","events_url":"https://api.github.com/repos/Azure/AKS/issues/676/events","html_url":"https://github.com/Azure/AKS/issues/676","id":366430459,"node_id":"MDU6SXNzdWUzNjY0MzA0NTk=","number":676,"title":"streamwatcher.go:109] Unable to decode an event from the watch stream: stream error: stream ID 1; INTERNAL_ERROR","user":{"login":"juan-lee","id":32657980,"node_id":"MDQ6VXNlcjMyNjU3OTgw","avatar_url":"https://avatars2.githubusercontent.com/u/32657980?v=4","gravatar_id":"","url":"https://api.github.com/users/juan-lee","html_url":"https://github.com/juan-lee","followers_url":"https://api.github.com/users/juan-lee/followers","following_url":"https://api.github.com/users/juan-lee/following{/other_user}","gists_url":"https://api.github.com/users/juan-lee/gists{/gist_id}","starred_url":"https://api.github.com/users/juan-lee/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/juan-lee/subscriptions","organizations_url":"https://api.github.com/users/juan-lee/orgs","repos_url":"https://api.github.com/users/juan-lee/repos","events_url":"https://api.github.com/users/juan-lee/events{/privacy}","received_events_url":"https://api.github.com/users/juan-lee/received_events","type":"User","site_admin":false},"labels":[{"id":730885656,"node_id":"MDU6TGFiZWw3MzA4ODU2NTY=","url":"https://api.github.com/repos/Azure/AKS/labels/known-issue","name":"known-issue","color":"efdb83","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":59,"created_at":"2018-10-03T16:45:29Z","updated_at":"2018-11-13T18:03:23Z","closed_at":null,"author_association":"COLLABORATOR","body":"**Symptoms**\r\nPods using the in cluster config to perform a watch on a resource will see intermittent timeouts and the following error in the pod log.\r\n\r\n`streamwatcher.go:109] Unable to decode an event from the watch stream: stream error: stream ID 1; INTERNAL_ERROR`\r\n\r\nIf the client performing the watch isn't handling errors gracefully, applications can get into an inconsistent state. Impacted applications include, but are not limited to, nginx-ingress and tiller (helm).\r\n\r\nA specific manifestation of this bug is the following error when attempting a helm deployment.\r\n\r\n`Error: watch closed before Until timeout`\r\n\r\n**Root Cause**\r\n1. The configuration of azureproxy results in unexpected timeouts for outbound watches targeting kubernetes.svc.cluster.local.\r\n2. On the client side, the client-go implementation of watch.Until does not handle intermittent network failures gracefully. See [watch closed before Until timeout](https://github.com/helm/helm/issues/2918), [Fix broken watches](https://github.com/kubernetes/kubernetes/pull/50102), and [Fix waiting in kubectl rollout status](https://github.com/kubernetes/kubernetes/pull/67817) for more details.\r\n\r\n**Workaround**\r\nFor the pods/containers that see the `INTERNAL_ERROR` in their logs add the following environment variables to the container spec. Be sure to replace `<your-fqdn-prefix>` and `<region>` so the aks kube-apiserver FQDN is correct.\r\n\r\n``` yaml\r\nspec:\r\n  containers:\r\n    env:\r\n    - name: KUBERNETES_PORT_443_TCP_ADDR\r\n      value: <your-fqdn-prefix>.hcp.<region>.azmk8s.io\r\n    - name: KUBERNETES_PORT\r\n      value: tcp://<your-fqdn-prefix>.hcp.<region>.azmk8s.io:443\r\n    - name: KUBERNETES_PORT_443_TCP\r\n      value: tcp://<your-fqdn-prefix>.hcp.<region>.azmk8s.io:443\r\n    - name: KUBERNETES_SERVICE_HOST\r\n      value: <your-fqdn-prefix>.hcp.<region>.azmk8s.io\r\n```"},"comment":{"url":"https://api.github.com/repos/Azure/AKS/issues/comments/438375271","html_url":"https://github.com/Azure/AKS/issues/676#issuecomment-438375271","issue_url":"https://api.github.com/repos/Azure/AKS/issues/676","id":438375271,"node_id":"MDEyOklzc3VlQ29tbWVudDQzODM3NTI3MQ==","user":{"login":"adinunzio84","id":34455143,"node_id":"MDQ6VXNlcjM0NDU1MTQz","avatar_url":"https://avatars1.githubusercontent.com/u/34455143?v=4","gravatar_id":"","url":"https://api.github.com/users/adinunzio84","html_url":"https://github.com/adinunzio84","followers_url":"https://api.github.com/users/adinunzio84/followers","following_url":"https://api.github.com/users/adinunzio84/following{/other_user}","gists_url":"https://api.github.com/users/adinunzio84/gists{/gist_id}","starred_url":"https://api.github.com/users/adinunzio84/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/adinunzio84/subscriptions","organizations_url":"https://api.github.com/users/adinunzio84/orgs","repos_url":"https://api.github.com/users/adinunzio84/repos","events_url":"https://api.github.com/users/adinunzio84/events{/privacy}","received_events_url":"https://api.github.com/users/adinunzio84/received_events","type":"User","site_admin":false},"created_at":"2018-11-13T18:03:23Z","updated_at":"2018-11-13T18:03:23Z","author_association":"NONE","body":"@mmosttler Haven't fully tested this yet, but I imagine it will be something like this:\r\n```\r\napiVersion: networking.istio.io/v1alpha3\r\nkind: ServiceEntry\r\nmetadata:\r\n  name: azmk8s-se\r\nspec:\r\n  hosts:\r\n  - \"*.azmk8s.io.net\"\r\n  ports:\r\n  - number: 443\r\n    name: https\r\n    protocol: HTTPS\r\n```\r\n\r\nI used a wildcard, but you can put your apiserver's FQDN there instead to be more specific... Let me know if you come up with something better though."}}
{"actor":{"display_login":"adinunzio84"}}