{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/17986","repository_url":"https://api.github.com/repos/elastic/elasticsearch","labels_url":"https://api.github.com/repos/elastic/elasticsearch/issues/17986/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/issues/17986/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch/issues/17986/events","html_url":"https://github.com/elastic/elasticsearch/pull/17986","id":151121794,"number":17986,"title":"block_until_refresh: wait for changes to be visible by search","user":{"login":"nik9000","id":215970,"avatar_url":"https://avatars.githubusercontent.com/u/215970?v=3","gravatar_id":"","url":"https://api.github.com/users/nik9000","html_url":"https://github.com/nik9000","followers_url":"https://api.github.com/users/nik9000/followers","following_url":"https://api.github.com/users/nik9000/following{/other_user}","gists_url":"https://api.github.com/users/nik9000/gists{/gist_id}","starred_url":"https://api.github.com/users/nik9000/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nik9000/subscriptions","organizations_url":"https://api.github.com/users/nik9000/orgs","repos_url":"https://api.github.com/users/nik9000/repos","events_url":"https://api.github.com/users/nik9000/events{/privacy}","received_events_url":"https://api.github.com/users/nik9000/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/:CRUD","name":":CRUD","color":"009800"},{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/breaking","name":"breaking","color":"ff0000"},{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/enhancement","name":"enhancement","color":"4a4ea8"},{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/review","name":"review","color":"0052cc"},{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/v5.0.0","name":"v5.0.0","color":"dddddd"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":33,"created_at":"2016-04-26T12:34:15Z","updated_at":"2016-06-03T08:00:22Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/elastic/elasticsearch/pulls/17986","html_url":"https://github.com/elastic/elasticsearch/pull/17986","diff_url":"https://github.com/elastic/elasticsearch/pull/17986.diff","patch_url":"https://github.com/elastic/elasticsearch/pull/17986.patch"},"body":"This adds support for setting the `refresh` request parameter to `wait_for` in the `index`, `delete`, `update`, and `bulk` APIs. When `refresh=wait_for` is set those APIs will not return until their results have been made visible to search by a refresh.\r\n\r\nAlso it adds a `forced_refresh` field to the response of `index`, `delete`, `update`, and to each item in a `bulk` response. This will be `true` for requests with `?refresh` or `?refresh=true` and will be `true` for some requests (see below) with `refresh=wait_for` but ought to otherwise always be `false`.\r\n\r\n`refresh=wait_for` is implemented as a list of `Tuple<Translog.Location, Consumer<Boolean>>`s in the new `RefreshListeners` class that is managed by `IndexShard`. The dynamic, index scoped `index.max_refresh_listeners` setting controls a maximum number of listeners allowed in any shard. If more than that many listeners accumulate in the engine then a refresh will be forced, the thread that adds the listener will be blocked until the refresh completes, and then the listener will be called with a `forcedRefresh` flag so it knows that it was the \"straw that broke the camel's back\". These listeners are only used by `refresh=wait_for` and that flag manifests itself as `forced_refresh` being `true` in the response.\r\n\r\nAbout half of this change comes from piping async-ness down to the appropriate layer in a way that is compatible with the ongoing with with sequence ids.\r\n\r\n# Edit\r\nRewritten to reflect changes made as part of the discussion.\r\n## Original\r\nThis adds the `block_until_refresh` request parameter to the `index`, `delete`, `update`, and `bulk` APIs. When set those APIs will not return until their results have been made visible to search by a `refresh`.\r\n\r\nAlso it adds a `forced_refresh` element to the response of `index`, `delete`, `update`, and to each item in a `bulk` response. This will be `true` for requests marked as `refresh` and will be `true` for some requests (see below) with `block_until_refresh` but ought to otherwise always be false.\r\n\r\n`block_until_refresh` is implemented as a `LinkedTransferQueue` of `RefreshListener`s in `InternalEngine`. The dynamic, index scoped `index.max_refresh_listeners` setting controls a maximum number of listeners allowed in any shard. If more than that many listeners accumulate in the engine then a refresh will be force, the thread that adds the listener will be blocked until the refresh completes, and then the listener will be called with a `forcedRefresh` flag so it knows that it was the \"straw that broke the camel's back\". These listeners are only used by `block_until_refresh` and that flag manifests itself as `forced_refresh` being `true` in the response.\r\n\r\nSince `LinkedTransferQueue`'s `size` method is linear time `InternalEngine` uses a `volatile int` field to try and guess the size. There are threading issues with this but that shouldn't matter because undercounting the number of listeners or overcounting them seems like it is unlikely to cause too much trouble.\r\n\r\nCloses #1063"},"comment":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/comments/223515672","html_url":"https://github.com/elastic/elasticsearch/pull/17986#issuecomment-223515672","issue_url":"https://api.github.com/repos/elastic/elasticsearch/issues/17986","id":223515672,"user":{"login":"clintongormley","id":56599,"avatar_url":"https://avatars.githubusercontent.com/u/56599?v=3","gravatar_id":"","url":"https://api.github.com/users/clintongormley","html_url":"https://github.com/clintongormley","followers_url":"https://api.github.com/users/clintongormley/followers","following_url":"https://api.github.com/users/clintongormley/following{/other_user}","gists_url":"https://api.github.com/users/clintongormley/gists{/gist_id}","starred_url":"https://api.github.com/users/clintongormley/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/clintongormley/subscriptions","organizations_url":"https://api.github.com/users/clintongormley/orgs","repos_url":"https://api.github.com/users/clintongormley/repos","events_url":"https://api.github.com/users/clintongormley/events{/privacy}","received_events_url":"https://api.github.com/users/clintongormley/received_events","type":"User","site_admin":false},"created_at":"2016-06-03T08:00:21Z","updated_at":"2016-06-03T08:00:21Z","body":"> Note: this PR is still breaking because it drops support for ?refresh=1. We only support ?refresh=true and ?refresh to trigger a refresh\r\n\r\n/cc @elastic/es-clients see https://github.com/elastic/elasticsearch/pull/17986/files#diff-02599d6739820daa4eb7295784714ca8L24"}}
{"actor":{"display_login":"clintongormley"}}