{"action":"created","issue":{"url":"https://api.github.com/repos/apache/incubator-druid/issues/6172","repository_url":"https://api.github.com/repos/apache/incubator-druid","labels_url":"https://api.github.com/repos/apache/incubator-druid/issues/6172/labels{/name}","comments_url":"https://api.github.com/repos/apache/incubator-druid/issues/6172/comments","events_url":"https://api.github.com/repos/apache/incubator-druid/issues/6172/events","html_url":"https://github.com/apache/incubator-druid/issues/6172","id":350413776,"node_id":"MDU6SXNzdWUzNTA0MTM3NzY=","number":6172,"title":"nodes should allow draining","user":{"login":"pdeva","id":3190176,"node_id":"MDQ6VXNlcjMxOTAxNzY=","avatar_url":"https://avatars3.githubusercontent.com/u/3190176?v=4","gravatar_id":"","url":"https://api.github.com/users/pdeva","html_url":"https://github.com/pdeva","followers_url":"https://api.github.com/users/pdeva/followers","following_url":"https://api.github.com/users/pdeva/following{/other_user}","gists_url":"https://api.github.com/users/pdeva/gists{/gist_id}","starred_url":"https://api.github.com/users/pdeva/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pdeva/subscriptions","organizations_url":"https://api.github.com/users/pdeva/orgs","repos_url":"https://api.github.com/users/pdeva/repos","events_url":"https://api.github.com/users/pdeva/events{/privacy}","received_events_url":"https://api.github.com/users/pdeva/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-08-14T12:50:02Z","updated_at":"2018-08-22T09:47:48Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"currently individual nodes have no concept of draining.\r\nthis means when updating the cluster, as you take down nodes, the queries in progress will fail.\r\nsimilarly, if you have broker nodes behind a load balancer, there is no way to tell the load balancer to stop sending new connection to the node you are about to update, which can result in many seconds of requests being sent to nodes down depending on health check interval.\r\n\r\ni suggest adding a couple endpoints to broker nodes:\r\n\r\n1. `/health/ping` returns 200 when the broker is ready to serve queries\r\n2. `/health/startDrain` sets a flag that makes `/health/ping` throw 500. this will make load balancer health checks fail while not dropping existing connections, resulting in zero downtime updates.\r\n\r\nSimilar endpoints can be put on MM and Historical nodes, with coordinator performing the health check. when the health check returns non-200 values, coordinator can instruct broker not to send any new queries.\r\n\r\nThis will result in 100% zero downtime rolling updates."},"comment":{"url":"https://api.github.com/repos/apache/incubator-druid/issues/comments/414976760","html_url":"https://github.com/apache/incubator-druid/issues/6172#issuecomment-414976760","issue_url":"https://api.github.com/repos/apache/incubator-druid/issues/6172","id":414976760,"node_id":"MDEyOklzc3VlQ29tbWVudDQxNDk3Njc2MA==","user":{"login":"lyman","id":205238,"node_id":"MDQ6VXNlcjIwNTIzOA==","avatar_url":"https://avatars0.githubusercontent.com/u/205238?v=4","gravatar_id":"","url":"https://api.github.com/users/lyman","html_url":"https://github.com/lyman","followers_url":"https://api.github.com/users/lyman/followers","following_url":"https://api.github.com/users/lyman/following{/other_user}","gists_url":"https://api.github.com/users/lyman/gists{/gist_id}","starred_url":"https://api.github.com/users/lyman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lyman/subscriptions","organizations_url":"https://api.github.com/users/lyman/orgs","repos_url":"https://api.github.com/users/lyman/repos","events_url":"https://api.github.com/users/lyman/events{/privacy}","received_events_url":"https://api.github.com/users/lyman/received_events","type":"User","site_admin":false},"created_at":"2018-08-22T09:47:48Z","updated_at":"2018-08-22T09:47:48Z","author_association":"NONE","body":"The problem of checking /druid/broker/v1/loadstatus is that it returns 200 even if the segment view is not initialized. This requires the load balancer to be able to check not only http code but also response content to keep response correct."}}
{"actor":{"display_login":"lyman"}}