{"action":"created","issue":{"url":"https://api.github.com/repos/istio/istio/issues/7665","repository_url":"https://api.github.com/repos/istio/istio","labels_url":"https://api.github.com/repos/istio/istio/issues/7665/labels{/name}","comments_url":"https://api.github.com/repos/istio/istio/issues/7665/comments","events_url":"https://api.github.com/repos/istio/istio/issues/7665/events","html_url":"https://github.com/istio/istio/issues/7665","id":347951905,"node_id":"MDU6SXNzdWUzNDc5NTE5MDU=","number":7665,"title":"503 errors when scaling down, or rolling out a new application version","user":{"login":"Stono","id":1486729,"node_id":"MDQ6VXNlcjE0ODY3Mjk=","avatar_url":"https://avatars2.githubusercontent.com/u/1486729?v=4","gravatar_id":"","url":"https://api.github.com/users/Stono","html_url":"https://github.com/Stono","followers_url":"https://api.github.com/users/Stono/followers","following_url":"https://api.github.com/users/Stono/following{/other_user}","gists_url":"https://api.github.com/users/Stono/gists{/gist_id}","starred_url":"https://api.github.com/users/Stono/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Stono/subscriptions","organizations_url":"https://api.github.com/users/Stono/orgs","repos_url":"https://api.github.com/users/Stono/repos","events_url":"https://api.github.com/users/Stono/events{/privacy}","received_events_url":"https://api.github.com/users/Stono/received_events","type":"User","site_admin":false},"labels":[{"id":735781806,"node_id":"MDU6TGFiZWw3MzU3ODE4MDY=","url":"https://api.github.com/repos/istio/istio/labels/area/networking","name":"area/networking","color":"0e8a16","default":false},{"id":822212191,"node_id":"MDU6TGFiZWw4MjIyMTIxOTE=","url":"https://api.github.com/repos/istio/istio/labels/kind/customer%20issue","name":"kind/customer issue","color":"fbca04","default":false}],"state":"open","locked":false,"assignee":{"login":"PiotrSikora","id":190297,"node_id":"MDQ6VXNlcjE5MDI5Nw==","avatar_url":"https://avatars3.githubusercontent.com/u/190297?v=4","gravatar_id":"","url":"https://api.github.com/users/PiotrSikora","html_url":"https://github.com/PiotrSikora","followers_url":"https://api.github.com/users/PiotrSikora/followers","following_url":"https://api.github.com/users/PiotrSikora/following{/other_user}","gists_url":"https://api.github.com/users/PiotrSikora/gists{/gist_id}","starred_url":"https://api.github.com/users/PiotrSikora/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/PiotrSikora/subscriptions","organizations_url":"https://api.github.com/users/PiotrSikora/orgs","repos_url":"https://api.github.com/users/PiotrSikora/repos","events_url":"https://api.github.com/users/PiotrSikora/events{/privacy}","received_events_url":"https://api.github.com/users/PiotrSikora/received_events","type":"User","site_admin":false},"assignees":[{"login":"PiotrSikora","id":190297,"node_id":"MDQ6VXNlcjE5MDI5Nw==","avatar_url":"https://avatars3.githubusercontent.com/u/190297?v=4","gravatar_id":"","url":"https://api.github.com/users/PiotrSikora","html_url":"https://github.com/PiotrSikora","followers_url":"https://api.github.com/users/PiotrSikora/followers","following_url":"https://api.github.com/users/PiotrSikora/following{/other_user}","gists_url":"https://api.github.com/users/PiotrSikora/gists{/gist_id}","starred_url":"https://api.github.com/users/PiotrSikora/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/PiotrSikora/subscriptions","organizations_url":"https://api.github.com/users/PiotrSikora/orgs","repos_url":"https://api.github.com/users/PiotrSikora/repos","events_url":"https://api.github.com/users/PiotrSikora/events{/privacy}","received_events_url":"https://api.github.com/users/PiotrSikora/received_events","type":"User","site_admin":false},{"login":"mandarjog","id":800375,"node_id":"MDQ6VXNlcjgwMDM3NQ==","avatar_url":"https://avatars2.githubusercontent.com/u/800375?v=4","gravatar_id":"","url":"https://api.github.com/users/mandarjog","html_url":"https://github.com/mandarjog","followers_url":"https://api.github.com/users/mandarjog/followers","following_url":"https://api.github.com/users/mandarjog/following{/other_user}","gists_url":"https://api.github.com/users/mandarjog/gists{/gist_id}","starred_url":"https://api.github.com/users/mandarjog/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mandarjog/subscriptions","organizations_url":"https://api.github.com/users/mandarjog/orgs","repos_url":"https://api.github.com/users/mandarjog/repos","events_url":"https://api.github.com/users/mandarjog/events{/privacy}","received_events_url":"https://api.github.com/users/mandarjog/received_events","type":"User","site_admin":false},{"login":"nmittler","id":6493296,"node_id":"MDQ6VXNlcjY0OTMyOTY=","avatar_url":"https://avatars3.githubusercontent.com/u/6493296?v=4","gravatar_id":"","url":"https://api.github.com/users/nmittler","html_url":"https://github.com/nmittler","followers_url":"https://api.github.com/users/nmittler/followers","following_url":"https://api.github.com/users/nmittler/following{/other_user}","gists_url":"https://api.github.com/users/nmittler/gists{/gist_id}","starred_url":"https://api.github.com/users/nmittler/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nmittler/subscriptions","organizations_url":"https://api.github.com/users/nmittler/orgs","repos_url":"https://api.github.com/users/nmittler/repos","events_url":"https://api.github.com/users/nmittler/events{/privacy}","received_events_url":"https://api.github.com/users/nmittler/received_events","type":"User","site_admin":false}],"milestone":null,"comments":12,"created_at":"2018-08-06T14:39:13Z","updated_at":"2018-08-30T06:08:39Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"**Describe the bug**\r\nHey, \r\nWe are noticing blips in services under load during kubernetes rollouts.  We observe a handful of 503 errors from istio-proxy on the pod being removed (either because of a rollout, or a scale down).  This screenshot is from three separate \"scale downs\":\r\n\r\n<img width=\"341\" alt=\"screen shot 2018-08-06 at 15 29 13\" src=\"https://user-images.githubusercontent.com/1486729/43722744-e7cdcf5a-998d-11e8-9503-c52b5e76e833.png\">\r\n\r\nWhen scaling down, this is the sequence of events we observe:\r\n\r\n  - pod goes into TERMINATING state and is removed from kubernetes endpoints\r\n  - A handful of the last requests to the pod are reported by istio-proxy as 503\r\n  - those requests are also logged in the upstream calling service as 503\r\n  - application exits\r\n  - istio-proxy exits\r\n\r\nAs you can see here:\r\n\r\n<img width=\"340\" alt=\"screen shot 2018-08-06 at 15 28 59\" src=\"https://user-images.githubusercontent.com/1486729/43722976-7835ffae-998e-11e8-9294-ffa91e6223b9.png\">\r\n\r\nAt the moment, our only saving grace is that we have configured a retry policy which means our end users experience a bit of a slow request, but not a failure - however relying on a retry mechanism in this scenario doesn't feel right.\r\n\r\n**Expected behavior**\r\nThe isito-proxy on the application being scaled down should not receive any requests after it has entered a TERMINATING state.\r\n\r\n**Steps to reproduce the bug**\r\nAs above, but I can get on hangouts and show you this in detail.\r\n\r\nThe application itself gracefully handles sigterms and drains and have confirmed this with load tests without istio-proxy in play.  I have also added a preStop hook to the application with istio, to ensure the app doesn't receive a SIGTERM until well after istio-proxy shuts down.\r\n\r\n**Version**\r\ngke 1.10.5, istio 1.0\r\n\r\n**Is Istio Auth enabled or not?**\r\nYes\r\n\r\n**Environment**\r\nGKE"},"comment":{"url":"https://api.github.com/repos/istio/istio/issues/comments/417201020","html_url":"https://github.com/istio/istio/issues/7665#issuecomment-417201020","issue_url":"https://api.github.com/repos/istio/istio/issues/7665","id":417201020,"node_id":"MDEyOklzc3VlQ29tbWVudDQxNzIwMTAyMA==","user":{"login":"ramaraochavali","id":17204904,"node_id":"MDQ6VXNlcjE3MjA0OTA0","avatar_url":"https://avatars0.githubusercontent.com/u/17204904?v=4","gravatar_id":"","url":"https://api.github.com/users/ramaraochavali","html_url":"https://github.com/ramaraochavali","followers_url":"https://api.github.com/users/ramaraochavali/followers","following_url":"https://api.github.com/users/ramaraochavali/following{/other_user}","gists_url":"https://api.github.com/users/ramaraochavali/gists{/gist_id}","starred_url":"https://api.github.com/users/ramaraochavali/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ramaraochavali/subscriptions","organizations_url":"https://api.github.com/users/ramaraochavali/orgs","repos_url":"https://api.github.com/users/ramaraochavali/repos","events_url":"https://api.github.com/users/ramaraochavali/events{/privacy}","received_events_url":"https://api.github.com/users/ramaraochavali/received_events","type":"User","site_admin":false},"created_at":"2018-08-30T06:08:39Z","updated_at":"2018-08-30T06:08:39Z","author_association":"NONE","body":"@rshriram just a question related to this - does istio proxy use draining during rolling updates? If it uses, is it via `/healthcheck/fail` endpoint supported by envoy? If it does not, how does it take care of inflight requests that might require egress calls? Sorry if it has already been explained and I missed that. "}}
{"actor":{"display_login":"ramaraochavali"}}