{"action":"created","issue":{"url":"https://api.github.com/repos/Azure/AKS/issues/704","repository_url":"https://api.github.com/repos/Azure/AKS","labels_url":"https://api.github.com/repos/Azure/AKS/issues/704/labels{/name}","comments_url":"https://api.github.com/repos/Azure/AKS/issues/704/comments","events_url":"https://api.github.com/repos/Azure/AKS/issues/704/events","html_url":"https://github.com/Azure/AKS/issues/704","id":374300140,"node_id":"MDU6SXNzdWUzNzQzMDAxNDA=","number":704,"title":"streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=839, ErrCode=NO_ERROR, debug=\"\"","user":{"login":"andig","id":184815,"node_id":"MDQ6VXNlcjE4NDgxNQ==","avatar_url":"https://avatars1.githubusercontent.com/u/184815?v=4","gravatar_id":"","url":"https://api.github.com/users/andig","html_url":"https://github.com/andig","followers_url":"https://api.github.com/users/andig/followers","following_url":"https://api.github.com/users/andig/following{/other_user}","gists_url":"https://api.github.com/users/andig/gists{/gist_id}","starred_url":"https://api.github.com/users/andig/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/andig/subscriptions","organizations_url":"https://api.github.com/users/andig/orgs","repos_url":"https://api.github.com/users/andig/repos","events_url":"https://api.github.com/users/andig/events{/privacy}","received_events_url":"https://api.github.com/users/andig/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2018-10-26T09:28:05Z","updated_at":"2018-10-30T15:18:13Z","closed_at":null,"author_association":"NONE","body":"I'm reproducibly seeing above error in the cluster autoscaler log files:\r\n\r\n\tI1026 09:20:18.456765       1 static_autoscaler.go:258] No unschedulable pods\r\n\tI1026 09:20:18.576563       1 scale_down.go:643] No candidates for scale down\r\n\tE1026 09:20:25.423590       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=3147, ErrCode=NO_ERROR, debug=\"\"\r\n\tE1026 09:20:25.424396       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=3147, ErrCode=NO_ERROR, debug=\"\"\r\n\tE1026 09:20:25.424613       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=3147, ErrCode=NO_ERROR, debug=\"\"\r\n\tE1026 09:20:25.425720       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=3147, ErrCode=NO_ERROR, debug=\"\"\r\n\tW1026 09:20:25.425852       1 reflector.go:341] k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:212: watch of *v1.Node ended with: very short watch: k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:212: Unexpected watch close - watch lasted less than a second and no items received\r\n\tE1026 09:20:25.425922       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=3147, ErrCode=NO_ERROR, debug=\"\"\r\n\tE1026 09:20:25.426055       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=3147, ErrCode=NO_ERROR, debug=\"\"\r\n\tE1026 09:20:25.426171       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=3147, ErrCode=NO_ERROR, debug=\"\"\r\n\tE1026 09:20:25.426273       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=3147, ErrCode=NO_ERROR, debug=\"\"\r\n\tE1026 09:20:25.436682       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=3147, ErrCode=NO_ERROR, debug=\"\"\r\n\tE1026 09:20:25.437003       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=3147, ErrCode=NO_ERROR, debug=\"\"\r\n\tE1026 09:20:25.437134       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=3147, ErrCode=NO_ERROR, debug=\"\"\r\n\tE1026 09:20:25.437351       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=3147, ErrCode=NO_ERROR, debug=\"\"\r\n\tE1026 09:20:25.437428       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=3147, ErrCode=NO_ERROR, debug=\"\"\r\n\tE1026 09:20:25.437571       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=3147, ErrCode=NO_ERROR, debug=\"\"\r\n\tE1026 09:20:25.437670       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=3147, ErrCode=NO_ERROR, debug=\"\"\r\n\tE1026 09:20:25.437902       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=3147, ErrCode=NO_ERROR, debug=\"\"\r\n\tW1026 09:20:25.457845       1 reflector.go:341] k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:130: watch of *v1.Service ended with: very short watch: k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:130: Unexpected watch close - watch lasted less than a second and no items received\r\n\tW1026 09:20:25.457899       1 reflector.go:341] k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:293: watch of *v1beta1.DaemonSet ended with: very short watch: k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:293: Unexpected watch close - watch lasted less than a second and no items received\r\n\tW1026 09:20:25.457958       1 reflector.go:341] k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:174: watch of *v1.Pod ended with: very short watch: k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:174: Unexpected watch close - watch lasted less than a second and no items received\r\n\tW1026 09:20:25.458722       1 reflector.go:341] k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:130: watch of *v1.StorageClass ended with: very short watch: k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:130: Unexpected watch close - watch lasted less than a second and no items received\r\n\tW1026 09:20:25.459685       1 reflector.go:341] k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:130: watch of *v1.Pod ended with: very short watch: k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:130: Unexpected watch close - watch lasted less than a second and no items received\r\n\tW1026 09:20:25.652833       1 reflector.go:341] k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:130: watch of *v1.ReplicationController ended with: too old resource version: 281755 (284135)\r\n\tW1026 09:20:25.655559       1 reflector.go:341] k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:266: watch of *v1beta1.PodDisruptionBudget ended with: too old resource version: 281765 (284138)\r\n\tW1026 09:20:25.656280       1 reflector.go:341] k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:130: watch of *v1.PersistentVolume ended with: too old resource version: 281748 (284133)\r\n\tW1026 09:20:25.671053       1 reflector.go:341] k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:130: watch of *v1beta1.StatefulSet ended with: too old resource version: 281769 (284139)\r\n\tW1026 09:20:25.671167       1 reflector.go:341] k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:130: watch of *v1beta1.PodDisruptionBudget ended with: too old resource version: 281765 (284138)\r\n\tW1026 09:20:25.693355       1 reflector.go:341] k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:130: watch of *v1.PersistentVolumeClaim ended with: too old resource version: 281748 (284134)\r\n\tI1026 09:20:26.455502       1 reflector.go:240] Listing and watching *v1.Node from k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:212\r\n\tI1026 09:20:26.458890       1 reflector.go:240] Listing and watching *v1beta1.DaemonSet from k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:293\r\n\tI1026 09:20:26.459199       1 reflector.go:240] Listing and watching *v1.Service from k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:130\r\n\tI1026 09:20:26.459266       1 reflector.go:240] Listing and watching *v1.StorageClass from k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:130\r\n\tI1026 09:20:26.459574       1 reflector.go:240] Listing and watching *v1.Pod from k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:174\r\n\tI1026 09:20:26.459803       1 reflector.go:240] Listing and watching *v1.Pod from k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:130\r\n\tI1026 09:20:26.653271       1 reflector.go:240] Listing and watching *v1.ReplicationController from k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:130\r\n\tI1026 09:20:26.655681       1 reflector.go:240] Listing and watching *v1beta1.PodDisruptionBudget from k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:266\r\n\tI1026 09:20:26.656535       1 reflector.go:240] Listing and watching *v1.PersistentVolume from k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:130\r\n\tI1026 09:20:26.671326       1 reflector.go:240] Listing and watching *v1beta1.PodDisruptionBudget from k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:130\r\n\tI1026 09:20:26.671472       1 reflector.go:240] Listing and watching *v1beta1.StatefulSet from k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:130\r\n\tI1026 09:20:26.693762       1 reflector.go:240] Listing and watching *v1.PersistentVolumeClaim from k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:130\r\n\tI1026 09:20:28.736725       1 utils.go:503] No pod using affinity / antiaffinity found in cluster, disabling affinity predicate for this loop\r\n\tI1026 09:20:28.736875       1 static_autoscaler.go:258] No unschedulable pods\r\n\tI1026 09:20:28.865323       1 scale_down.go:643] No candidates for scale down\r\n\tI1026 09:20:38.955677       1 azure_manager.go:261] Refreshed ASG list, next refresh after 2018-10-26 09:21:38.955668636 +0000 UTC m=+2863.900159080\r\n\tI1026 09:20:39.131065       1 utils.go:503] No pod using affinity / antiaffinity found in cluster, disabling affinity predicate for this loop\r\n\tI1026 09:20:39.131178       1 static_autoscaler.go:258] No unschedulable pods\r\n\tI1026 09:20:39.265120       1 scale_down.go:643] No candidates for scale down\r\n\tI1026 09:20:49.484940       1 static_autoscaler.go:160] 2 unregistered nodes present\r\n\r\nThe error happens whenever cluster scaling operation happens- independently if its triggered by the auto scaler or manually using azure cli. This includes scale *out* scaling especially which should not lead to resource shortage.\r\n\r\nWorkaround for https://github.com/Azure/AKS/issues/676 has already been applied and has taken care of that class of errors. \r\n"},"comment":{"url":"https://api.github.com/repos/Azure/AKS/issues/comments/434343089","html_url":"https://github.com/Azure/AKS/issues/704#issuecomment-434343089","issue_url":"https://api.github.com/repos/Azure/AKS/issues/704","id":434343089,"node_id":"MDEyOklzc3VlQ29tbWVudDQzNDM0MzA4OQ==","user":{"login":"weinong","id":4204090,"node_id":"MDQ6VXNlcjQyMDQwOTA=","avatar_url":"https://avatars2.githubusercontent.com/u/4204090?v=4","gravatar_id":"","url":"https://api.github.com/users/weinong","html_url":"https://github.com/weinong","followers_url":"https://api.github.com/users/weinong/followers","following_url":"https://api.github.com/users/weinong/following{/other_user}","gists_url":"https://api.github.com/users/weinong/gists{/gist_id}","starred_url":"https://api.github.com/users/weinong/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/weinong/subscriptions","organizations_url":"https://api.github.com/users/weinong/orgs","repos_url":"https://api.github.com/users/weinong/repos","events_url":"https://api.github.com/users/weinong/events{/privacy}","received_events_url":"https://api.github.com/users/weinong/received_events","type":"User","site_admin":false},"created_at":"2018-10-30T15:18:13Z","updated_at":"2018-10-30T15:18:13Z","author_association":"CONTRIBUTOR","body":"Unnecessary api server restart causes it. I’ll update when we have a fix.\n\nGet Outlook for iOS<https://aka.ms/o0ukef>\n\n________________________________\nFrom: andig <notifications@github.com>\nSent: Tuesday, October 30, 2018 2:21 AM\nTo: Azure/AKS\nCc: Weinong Wang; Mention\nSubject: Re: [Azure/AKS] streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=839, ErrCode=NO_ERROR, debug=\"\" (#704)\n\n\n@weinong<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fweinong&data=02%7C01%7Cweinongw%40microsoft.com%7Cdfe151fb9fcc4e64a16208d63e490c17%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636764880851939173&sdata=tRXK8XdhSHtC673a4J6URAA2zL4J%2BpLROYb9ScC68Cw%3D&reserved=0> any more insights whats going on on this one?\n\n—\nYou are receiving this because you were mentioned.\nReply to this email directly, view it on GitHub<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAzure%2FAKS%2Fissues%2F704%23issuecomment-434227279&data=02%7C01%7Cweinongw%40microsoft.com%7Cdfe151fb9fcc4e64a16208d63e490c17%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636764880851939173&sdata=kWtP%2FpSWq91jsFP5MHLygrrKBvqwdGABav5lSgxjrK0%3D&reserved=0>, or mute the thread<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAEAmOllC6FxNnwbFSG906C5GU3NEWk6pks5uqBoNgaJpZM4X73Fa&data=02%7C01%7Cweinongw%40microsoft.com%7Cdfe151fb9fcc4e64a16208d63e490c17%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636764880851949182&sdata=xtyh24O6yM%2B7NQIs3r1CQQMMWHw%2BQXd3mY%2FGbqJN18A%3D&reserved=0>.\n"}}
{"actor":{"display_login":"weinong"}}