{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/10005","labels_url":"https://api.github.com/repos/elastic/elasticsearch/issues/10005/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/issues/10005/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch/issues/10005/events","html_url":"https://github.com/elastic/elasticsearch/issues/10005","id":60026809,"number":10005,"title":"Date type has not enough precision for the logging use case.","user":{"login":"jordansissel","id":131818,"avatar_url":"https://avatars.githubusercontent.com/u/131818?v=3","gravatar_id":"","url":"https://api.github.com/users/jordansissel","html_url":"https://github.com/jordansissel","followers_url":"https://api.github.com/users/jordansissel/followers","following_url":"https://api.github.com/users/jordansissel/following{/other_user}","gists_url":"https://api.github.com/users/jordansissel/gists{/gist_id}","starred_url":"https://api.github.com/users/jordansissel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jordansissel/subscriptions","organizations_url":"https://api.github.com/users/jordansissel/orgs","repos_url":"https://api.github.com/users/jordansissel/repos","events_url":"https://api.github.com/users/jordansissel/events{/privacy}","received_events_url":"https://api.github.com/users/jordansissel/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/:Dates","name":":Dates","color":"009800"},{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/discuss","name":"discuss","color":"fbca04"},{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/high%20hanging%20fruit","name":"high hanging fruit","color":"009800"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":26,"created_at":"2015-03-05T22:28:11Z","updated_at":"2016-01-18T21:02:01Z","closed_at":null,"body":"At present, the 'date' type is millisecond precision. For many log use cases, higher precision time is valuable - microsecond, nanosecond, etc.\r\n\r\nThe biggest impact of this is during sorting of search results. If you sort chronologically, newest-first, by a date field, documents with the same date will probably be sorted incorrectly (because they match). This is often reported by users seeing events \"out of order\" when they have the same timestamp. Specific example being sorting by date and seeing events in newest-first order, unless there is a tie, in which case oldest-first (or first-written?) appears. This causes a bit of confusion for the ELK use case.\r\n\r\nRelated: https://github.com/logstash-plugins/logstash-filter-date/pull/8\r\n\r\nI don't have any firm proposals, but I have two different implementation ideas:\r\n\r\n* Proposal 1, use a separate field: Store our own custom-precision time in a separate field as a _long_. This allows us to do correct sorting (because we have higher precision), but it makes any date-related functionality in Elasticsearch not usable (searching `now-1h` or doing date_histogram, etc)\r\n* Proposal 2, date type has tunable precision: Have the `date` type have configurable precision, with the default (backwards compatible) precision being milliseconds. This would let us choose, for example, nanosecond precision for the logging use case, and year precision for an archaeological use case (billions of years ago, or something). Benefit here is date histogram and other date-related features could still work. Further, having the precision configurable would allow us to keep the underlying data structure a 64bit long and users could choose their most appropriate precision."},"comment":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/comments/172653000","html_url":"https://github.com/elastic/elasticsearch/issues/10005#issuecomment-172653000","issue_url":"https://api.github.com/repos/elastic/elasticsearch/issues/10005","id":172653000,"user":{"login":"jcollie","id":740022,"avatar_url":"https://avatars.githubusercontent.com/u/740022?v=3","gravatar_id":"","url":"https://api.github.com/users/jcollie","html_url":"https://github.com/jcollie","followers_url":"https://api.github.com/users/jcollie/followers","following_url":"https://api.github.com/users/jcollie/following{/other_user}","gists_url":"https://api.github.com/users/jcollie/gists{/gist_id}","starred_url":"https://api.github.com/users/jcollie/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jcollie/subscriptions","organizations_url":"https://api.github.com/users/jcollie/orgs","repos_url":"https://api.github.com/users/jcollie/repos","events_url":"https://api.github.com/users/jcollie/events{/privacy}","received_events_url":"https://api.github.com/users/jcollie/received_events","type":"User","site_admin":false},"created_at":"2016-01-18T21:02:01Z","updated_at":"2016-01-18T21:02:01Z","body":"@dtr2 that's a lot of \"ifs\" to be relying on ElasticSearch's autogenerated IDs, nearly all of which are violated in my cluster:\r\n\r\n1) Some messages supply their own ID if the source has a unique ID already associated with it (systemd journal messages in my case).\r\n2) All of my data runs through a RabbitMQ server (sometimes passing through multiple queues depending on the amount of processing that needs to be done) with multiple consumers per queue so there's no way that I can expect documents to be indexed in any specific order, much less by the same ElasticSearch node.\r\n\r\nIn any case, ElasticSearch does not guarantee the behavior of autogenerated IDs.  The docs only guarantee that the IDs are unique:\r\n\r\nhttps://www.elastic.co/guide/en/elasticsearch/guide/current/index-doc.html\r\n\r\nSo I can hope that you see that trying to impose an order based upon the ID cannot be relied upon in the general case.  Yes, there may be certain specific instances where that would work, but you'd be relying on undocumented implementation details."}}