{"action":"created","issue":{"url":"https://api.github.com/repos/openshift/origin-aggregated-logging/issues/1467","repository_url":"https://api.github.com/repos/openshift/origin-aggregated-logging","labels_url":"https://api.github.com/repos/openshift/origin-aggregated-logging/issues/1467/labels{/name}","comments_url":"https://api.github.com/repos/openshift/origin-aggregated-logging/issues/1467/comments","events_url":"https://api.github.com/repos/openshift/origin-aggregated-logging/issues/1467/events","html_url":"https://github.com/openshift/origin-aggregated-logging/issues/1467","id":385261929,"node_id":"MDU6SXNzdWUzODUyNjE5Mjk=","number":1467,"title":"Modifying Delayed Allocation Setting","user":{"login":"jcantrill","id":4548408,"node_id":"MDQ6VXNlcjQ1NDg0MDg=","avatar_url":"https://avatars2.githubusercontent.com/u/4548408?v=4","gravatar_id":"","url":"https://api.github.com/users/jcantrill","html_url":"https://github.com/jcantrill","followers_url":"https://api.github.com/users/jcantrill/followers","following_url":"https://api.github.com/users/jcantrill/following{/other_user}","gists_url":"https://api.github.com/users/jcantrill/gists{/gist_id}","starred_url":"https://api.github.com/users/jcantrill/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jcantrill/subscriptions","organizations_url":"https://api.github.com/users/jcantrill/orgs","repos_url":"https://api.github.com/users/jcantrill/repos","events_url":"https://api.github.com/users/jcantrill/events{/privacy}","received_events_url":"https://api.github.com/users/jcantrill/received_events","type":"User","site_admin":false},"labels":[{"id":260296583,"node_id":"MDU6TGFiZWwyNjAyOTY1ODM=","url":"https://api.github.com/repos/openshift/origin-aggregated-logging/labels/kind/question","name":"kind/question","color":"cc317c","default":false}],"state":"open","locked":false,"assignee":{"login":"lukas-vlcek","id":205174,"node_id":"MDQ6VXNlcjIwNTE3NA==","avatar_url":"https://avatars2.githubusercontent.com/u/205174?v=4","gravatar_id":"","url":"https://api.github.com/users/lukas-vlcek","html_url":"https://github.com/lukas-vlcek","followers_url":"https://api.github.com/users/lukas-vlcek/followers","following_url":"https://api.github.com/users/lukas-vlcek/following{/other_user}","gists_url":"https://api.github.com/users/lukas-vlcek/gists{/gist_id}","starred_url":"https://api.github.com/users/lukas-vlcek/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lukas-vlcek/subscriptions","organizations_url":"https://api.github.com/users/lukas-vlcek/orgs","repos_url":"https://api.github.com/users/lukas-vlcek/repos","events_url":"https://api.github.com/users/lukas-vlcek/events{/privacy}","received_events_url":"https://api.github.com/users/lukas-vlcek/received_events","type":"User","site_admin":false},"assignees":[{"login":"lukas-vlcek","id":205174,"node_id":"MDQ6VXNlcjIwNTE3NA==","avatar_url":"https://avatars2.githubusercontent.com/u/205174?v=4","gravatar_id":"","url":"https://api.github.com/users/lukas-vlcek","html_url":"https://github.com/lukas-vlcek","followers_url":"https://api.github.com/users/lukas-vlcek/followers","following_url":"https://api.github.com/users/lukas-vlcek/following{/other_user}","gists_url":"https://api.github.com/users/lukas-vlcek/gists{/gist_id}","starred_url":"https://api.github.com/users/lukas-vlcek/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lukas-vlcek/subscriptions","organizations_url":"https://api.github.com/users/lukas-vlcek/orgs","repos_url":"https://api.github.com/users/lukas-vlcek/repos","events_url":"https://api.github.com/users/lukas-vlcek/events{/privacy}","received_events_url":"https://api.github.com/users/lukas-vlcek/received_events","type":"User","site_admin":false},{"login":"portante","id":949097,"node_id":"MDQ6VXNlcjk0OTA5Nw==","avatar_url":"https://avatars2.githubusercontent.com/u/949097?v=4","gravatar_id":"","url":"https://api.github.com/users/portante","html_url":"https://github.com/portante","followers_url":"https://api.github.com/users/portante/followers","following_url":"https://api.github.com/users/portante/following{/other_user}","gists_url":"https://api.github.com/users/portante/gists{/gist_id}","starred_url":"https://api.github.com/users/portante/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/portante/subscriptions","organizations_url":"https://api.github.com/users/portante/orgs","repos_url":"https://api.github.com/users/portante/repos","events_url":"https://api.github.com/users/portante/events{/privacy}","received_events_url":"https://api.github.com/users/portante/received_events","type":"User","site_admin":false}],"milestone":null,"comments":11,"created_at":"2018-11-28T13:13:08Z","updated_at":"2018-12-12T13:52:23Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"While monitoring and maintaining the operations clusters, we routinely need to perform rolling restarts and run into issues with unassigned shards which are reported like:\r\n```\r\nproject.foo.abc-123.2018.11.1              0     r  UNASSIGNED    NODE_LEFT\r\n```\r\nMany times after the nodes return I see the cluster stuck in yellow state because the unassigned shards never become allocated.  The [reroute](https://www.elastic.co/guide/en/elasticsearch/reference/5.6/cluster-reroute.html#_retry_failed_shards) documentation alludes to an allocation retry count which makes me wonder if the cluster gives up.\r\n\r\nThere additionally is documentation about [delayed allocation](https://www.elastic.co/guide/en/elasticsearch/reference/5.6/delayed-allocation.html) which I think we should consider bumping the default (i.e. 1m) to at least something like 5 minutes. It's not uncommon for a pod to take that long to terminate and scale back up.  What concerns might we have or drawbacks to modifying this setting?\r\n\r\nAlso found [this](https://github.com/elastic/elasticsearch/pull/15912/files#diff-5f7818b093901b6114a63793bc045f03R93) which provides some additional info but still is a little lacking IMO."},"comment":{"url":"https://api.github.com/repos/openshift/origin-aggregated-logging/issues/comments/446594699","html_url":"https://github.com/openshift/origin-aggregated-logging/issues/1467#issuecomment-446594699","issue_url":"https://api.github.com/repos/openshift/origin-aggregated-logging/issues/1467","id":446594699,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0NjU5NDY5OQ==","user":{"login":"jcantrill","id":4548408,"node_id":"MDQ6VXNlcjQ1NDg0MDg=","avatar_url":"https://avatars2.githubusercontent.com/u/4548408?v=4","gravatar_id":"","url":"https://api.github.com/users/jcantrill","html_url":"https://github.com/jcantrill","followers_url":"https://api.github.com/users/jcantrill/followers","following_url":"https://api.github.com/users/jcantrill/following{/other_user}","gists_url":"https://api.github.com/users/jcantrill/gists{/gist_id}","starred_url":"https://api.github.com/users/jcantrill/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jcantrill/subscriptions","organizations_url":"https://api.github.com/users/jcantrill/orgs","repos_url":"https://api.github.com/users/jcantrill/repos","events_url":"https://api.github.com/users/jcantrill/events{/privacy}","received_events_url":"https://api.github.com/users/jcantrill/received_events","type":"User","site_admin":false},"created_at":"2018-12-12T13:52:23Z","updated_at":"2018-12-12T13:52:23Z","author_association":"CONTRIBUTOR","body":"> @jcantrill @portante @wshearn If we follow [official rolling upgrade](https://www.elastic.co/guide/en/elasticsearch/reference/5.6/rolling-upgrades.html) process then what we should see is the following:\r\n> \r\n> 1. **Disable allocation**\r\n>    Notice this explicitly disables [delayed allocation](https://www.elastic.co/guide/en/elasticsearch/reference/5.6/delayed-allocation.html) timeout which also means there should be no giving up shard allocation due to [retry failed](https://www.elastic.co/guide/en/elasticsearch/reference/5.6/cluster-reroute.html#_retry_failed_shards), are we on the same page here?\r\n> 2. Stop indexing (are we doing this by scaling fluentd or by other means?)\r\n\r\nThis step we do not do, we simply allow fluent to 'stop' by back pressure.\r\n\r\n> 3. Perform a single node upgrade (In our case we upgrade the image)\r\n> 4. Upgrade all plugins (We already do this as part of step 3. as everything is in the same image)\r\n> 5. Start the node and confirm it joins the cluster.\r\n>    So **we really wait** until the node joins, right? Notice the allocation is still disabled.\r\n> 6. **Reenable shard allocation**\r\n>    If I am not mistaken only from this point the delayed allocation is in effect again (including the retry failed).\r\n> 7. We **wait again** for the cluster to become green or simply \"healthy enough\"\r\n> \r\n> If we are talking about extensive wait period in step 5 then I do not see how any changes to delayed allocation timeout should have any effect.\r\n\r\n"}}
{"actor":{"display_login":"jcantrill"}}