{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/beats/issues/1872","repository_url":"https://api.github.com/repos/elastic/beats","labels_url":"https://api.github.com/repos/elastic/beats/issues/1872/labels{/name}","comments_url":"https://api.github.com/repos/elastic/beats/issues/1872/comments","events_url":"https://api.github.com/repos/elastic/beats/issues/1872/events","html_url":"https://github.com/elastic/beats/pull/1872","id":160522904,"number":1872,"title":"[Proposal] Add fingerprint filter for event deduplication","user":{"login":"andrewkroh","id":4565752,"avatar_url":"https://avatars.githubusercontent.com/u/4565752?v=3","gravatar_id":"","url":"https://api.github.com/users/andrewkroh","html_url":"https://github.com/andrewkroh","followers_url":"https://api.github.com/users/andrewkroh/followers","following_url":"https://api.github.com/users/andrewkroh/following{/other_user}","gists_url":"https://api.github.com/users/andrewkroh/gists{/gist_id}","starred_url":"https://api.github.com/users/andrewkroh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/andrewkroh/subscriptions","organizations_url":"https://api.github.com/users/andrewkroh/orgs","repos_url":"https://api.github.com/users/andrewkroh/repos","events_url":"https://api.github.com/users/andrewkroh/events{/privacy}","received_events_url":"https://api.github.com/users/andrewkroh/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/elastic/beats/labels/:Filtering","name":":Filtering","color":"ffffff"},{"url":"https://api.github.com/repos/elastic/beats/labels/discuss","name":"discuss","color":"ffa500"},{"url":"https://api.github.com/repos/elastic/beats/labels/enhancement","name":"enhancement","color":"fbca04"},{"url":"https://api.github.com/repos/elastic/beats/labels/libbeat","name":"libbeat","color":"008000"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2016-06-15T21:01:39Z","updated_at":"2016-06-27T14:40:51Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/elastic/beats/pulls/1872","html_url":"https://github.com/elastic/beats/pull/1872","diff_url":"https://github.com/elastic/beats/pull/1872.diff","patch_url":"https://github.com/elastic/beats/pull/1872.patch"},"body":"The fingerprint filter uses a cryptographic hash function to calculate a hash value from specified fields in the event. The resulting hex encoded hash value is stored in the id field. The id field is used as the _id field when the event is sent through the elasticsearch output. Since there cannot be duplicate _id values stored in an elasticsearch index, this prevents an event from being duplicated.\r\n\r\nThe fingerprint filter is not appropriate for all use cases. It should only be applied when the combination input fields are sufficiently unique.\r\n\r\nSome config examples:\r\n\r\n```\r\n# Filebeat\r\nfilters:\r\n  - fingerprint:\r\n      fields: [beat.hostname, source, message]\r\n\r\n# Winlogbeat\r\nfilters:\r\n  - fingerprint:\r\n      fields: [computer_name, log_name, record_number, message]\r\n```\r\n\r\n\r\nBenchmarks:\r\n\r\n```\r\nBenchmarkFingerprintFilterSHA1-4  \t 1000000\t      2218 ns/op\t     664 B/op\t      12 allocs/op\r\nBenchmarkFingerprintFilterSHA256-4\t  500000\t      3203 ns/op\t     696 B/op\t      12 allocs/op\r\nBenchmarkFingerprintFilterSHA512-4\t  500000\t      3332 ns/op\t    1016 B/op\t      13 allocs/op\r\nBenchmarkFingerprintFilterMD5-4   \t 1000000\t      1880 ns/op\t     552 B/op\t      11 allocs/op\r\n```"},"comment":{"url":"https://api.github.com/repos/elastic/beats/issues/comments/228765717","html_url":"https://github.com/elastic/beats/pull/1872#issuecomment-228765717","issue_url":"https://api.github.com/repos/elastic/beats/issues/1872","id":228765717,"user":{"login":"polyfractal","id":1224228,"avatar_url":"https://avatars.githubusercontent.com/u/1224228?v=3","gravatar_id":"","url":"https://api.github.com/users/polyfractal","html_url":"https://github.com/polyfractal","followers_url":"https://api.github.com/users/polyfractal/followers","following_url":"https://api.github.com/users/polyfractal/following{/other_user}","gists_url":"https://api.github.com/users/polyfractal/gists{/gist_id}","starred_url":"https://api.github.com/users/polyfractal/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/polyfractal/subscriptions","organizations_url":"https://api.github.com/users/polyfractal/orgs","repos_url":"https://api.github.com/users/polyfractal/repos","events_url":"https://api.github.com/users/polyfractal/events{/privacy}","received_events_url":"https://api.github.com/users/polyfractal/received_events","type":"User","site_admin":false},"created_at":"2016-06-27T14:40:51Z","updated_at":"2016-06-27T14:40:51Z","body":"For reference, here's the ingest issue that I opened a while ago:  https://github.com/elastic/elasticsearch/issues/16938\r\n\r\nIt mainly focused on \"fuzzy fingerprinting\" using things like minhash, simhash, etc. where you want to group similar documents under a single \"fingerprint\".  But it could easily be extended to include exact hash functions that act as a de-duplication field.\r\n\r\nAnd somewhat related, we recently merged the [Fingerprint Analyzer](https://www.elastic.co/guide/en/elasticsearch/reference/master/analysis-fingerprint-analyzer.html), which can be used for fingerprinting text fields.  Although this is definitely more for grouping/clustering/ML on text than de-duping.\r\n\r\nJust a thought:  if this is added to Beats but the user doesn't specify *all* the fields, it'd be easy for accidental \"collisions\" to clobber existing data.  E.g.\r\n\r\n1. an event arrives with the tuple `\"192.168.0.160-hostFoo-abcxyz\"` which hashes to _id: `12345`.  The doc also has an additional, non-hashed field: `\"bar\": \"baz\"`.  \r\n2. a second event arrives with the same tuple `\"192.168.0.160-hostFoo-abcxyz\"` but now `\"bar\": \"bizzbuzz\"`.  It'll clobber the existing \"bar\" field\r\n\r\nObviously it's up to the user to configure the set of hashed fields to prevent this from happening, but it seems like it could be very trappy and hard to detect problems?"}}
{"actor":{"display_login":"polyfractal"}}