{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/20859","repository_url":"https://api.github.com/repos/elastic/elasticsearch","labels_url":"https://api.github.com/repos/elastic/elasticsearch/issues/20859/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/issues/20859/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch/issues/20859/events","html_url":"https://github.com/elastic/elasticsearch/issues/20859","id":182263311,"number":20859,"title":"Introduce a new `_last_modified` metafield","user":{"login":"s1monw","id":973334,"avatar_url":"https://avatars.githubusercontent.com/u/973334?v=3","gravatar_id":"","url":"https://api.github.com/users/s1monw","html_url":"https://github.com/s1monw","followers_url":"https://api.github.com/users/s1monw/followers","following_url":"https://api.github.com/users/s1monw/following{/other_user}","gists_url":"https://api.github.com/users/s1monw/gists{/gist_id}","starred_url":"https://api.github.com/users/s1monw/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/s1monw/subscriptions","organizations_url":"https://api.github.com/users/s1monw/orgs","repos_url":"https://api.github.com/users/s1monw/repos","events_url":"https://api.github.com/users/s1monw/events{/privacy}","received_events_url":"https://api.github.com/users/s1monw/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/:Core","name":":Core","color":"009800"},{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/enhancement","name":"enhancement","color":"4a4ea8"},{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/v5.1.0","name":"v5.1.0","color":"dddddd"},{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/v6.0.0","name":"v6.0.0","color":"dddddd"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-10-11T13:26:34Z","updated_at":"2016-10-11T16:02:35Z","closed_at":null,"body":"I had some discussions with @niemyjski and @ejsmith on #17895 about `_timestamp`, it's guarantees / semantics and how it was used vs. it's purpose. Yet, it became obvious to me that the semantics that are expected from the users perspective is somewhat a `last_modified` timestamp that is really updated every time the document is indexed on the primary shard. This implies, the value of this fields is updated:\r\n\r\n * when reindexed via `_reindex`\r\n * when updated (with script or not)\r\n * when indexed via the API\r\n\r\nthe field should be very clear in it's semantics such that it can NOT be provided or modified from the outside. Yet, what is still somewhat unclear to me is if it should be indexed, stored and / or have docvalues. I think when we add this we want the configuration options to be as small as possible maybe only `enabled: true|false`? For this to happen we need to know the usecases and I was hoping @niemyjski and @ejsmith could help with this to design this new field with clear semantics?"},"comment":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/comments/252962327","html_url":"https://github.com/elastic/elasticsearch/issues/20859#issuecomment-252962327","issue_url":"https://api.github.com/repos/elastic/elasticsearch/issues/20859","id":252962327,"user":{"login":"ejsmith","id":282584,"avatar_url":"https://avatars.githubusercontent.com/u/282584?v=3","gravatar_id":"","url":"https://api.github.com/users/ejsmith","html_url":"https://github.com/ejsmith","followers_url":"https://api.github.com/users/ejsmith/followers","following_url":"https://api.github.com/users/ejsmith/following{/other_user}","gists_url":"https://api.github.com/users/ejsmith/gists{/gist_id}","starred_url":"https://api.github.com/users/ejsmith/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ejsmith/subscriptions","organizations_url":"https://api.github.com/users/ejsmith/orgs","repos_url":"https://api.github.com/users/ejsmith/repos","events_url":"https://api.github.com/users/ejsmith/events{/privacy}","received_events_url":"https://api.github.com/users/ejsmith/received_events","type":"User","site_admin":false},"created_at":"2016-10-11T16:02:34Z","updated_at":"2016-10-11T16:02:34Z","body":"The primary use case is for doing things like online reindex and bulk operations. It's unrealistic for us to take our cluster online while we do a 100gb reindex. So we do this:\r\n\r\n- Alias that the app uses to read and write data\r\n- Reindex in the background to a new index\r\n- 2nd pass of reindex where the timestamp is greater than the date that we started the initial reindex. This gets us pretty close to being real-time\r\n- Swap the alias from the old index to the new one\r\n- Do one more reindex pass for any documents that may have been modified in the time it took us to swap aliases and do the 2nd pass reindex\r\n\r\nSo you can see that the timestamp field is really important to this strategy. Also, we use it for other bulk operations as well.\r\n\r\nTo me _timestamp is the correct name for this feature. \"_last_modified\" sounds like it's only set when the document is modified, but it should be set when new documents are written as well. I think the core issue with the current _timestamp is that you let it be mapped to the document and modified. I understand that you don't want to break existing people, but you are already breaking them by removing the _timestamp feature.  So couldn't we just modify that feature instead to do the right thing?"}}
{"actor":{"display_login":"ejsmith"}}