{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/16938","repository_url":"https://api.github.com/repos/elastic/elasticsearch","labels_url":"https://api.github.com/repos/elastic/elasticsearch/issues/16938/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/issues/16938/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch/issues/16938/events","html_url":"https://github.com/elastic/elasticsearch/issues/16938","id":138265685,"number":16938,"title":"Fingerprinting Ingest Processor","user":{"login":"polyfractal","id":1224228,"avatar_url":"https://avatars1.githubusercontent.com/u/1224228?v=4","gravatar_id":"","url":"https://api.github.com/users/polyfractal","html_url":"https://github.com/polyfractal","followers_url":"https://api.github.com/users/polyfractal/followers","following_url":"https://api.github.com/users/polyfractal/following{/other_user}","gists_url":"https://api.github.com/users/polyfractal/gists{/gist_id}","starred_url":"https://api.github.com/users/polyfractal/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/polyfractal/subscriptions","organizations_url":"https://api.github.com/users/polyfractal/orgs","repos_url":"https://api.github.com/users/polyfractal/repos","events_url":"https://api.github.com/users/polyfractal/events{/privacy}","received_events_url":"https://api.github.com/users/polyfractal/received_events","type":"User","site_admin":false},"labels":[{"id":268963484,"url":"https://api.github.com/repos/elastic/elasticsearch/labels/:Core/Ingest","name":":Core/Ingest","color":"0e8a16","default":false},{"id":111416437,"url":"https://api.github.com/repos/elastic/elasticsearch/labels/discuss","name":"discuss","color":"fbca04","default":false}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":25,"created_at":"2016-03-03T18:52:43Z","updated_at":"2018-03-19T18:03:51Z","closed_at":"2018-03-13T16:17:31Z","author_association":"MEMBER","body":"A potentially useful processor is one that can generate one or more \"fingerprints\" from an incoming document.  This could aid in finding duplicates, detecting plagarism, or clustering similar documents together.\n\nI think there are two realms of fingerprinting: content fingerprinting and structural fingerprinting.\n### Content Fingerprinting\n\nHashes the content of fields to generate a fingerprint-per-field, and optionally, a fingerprint that represents all the fields.  Could use simple hashing, or perhaps something more sophisticated like MinHash, SimHash, [Winnowing](http://igm.univ-mlv.fr/~mac/ENS/DOC/sigmod03-1.pdf) or [Probabilistic Fingerprinting](http://www.cs.cmu.edu/~gveda/reports/IITK/cs497_DocFingerprinting.pdf).  \n\nThe API could look something like:\n\n``` json\n{\n  \"fingerprint\": {\n    \"type\": \"content\",\n    \"fields\": [\"foo\", \"bar\"],\n    \"hash\": \"minhash\",\n    \"hash_all\": true\n  }\n}\n```\n\nE.g. specify the type of fingerprinting we want to do (`content`), a list of fields to hash, the style of hashing and if we should also hash all the hashes together.  The output would then be the document + new fingerprint fields:\n\n``` json\n{\n  \"foo\": \"...\",\n  \"bar\": \"...\",\n  \"fingerprint_foo\": 1283891,\n  \"fingerprint_bar\": 8372038,\n  \"fingerprint_all\": 3817273\n}\n```\n### Structural Fingerprinting\n\nThe other mode of fingerprinting could be structural in nature (this is the one I'm more interested in, tbh).  Instead of fingerprinting the content of fields, we are actually fingerprinting the structure of the document itself.  Essentially, we would recursively parse the JSON and hash the keys at each level in the JSON tree.  These hashes then become a fingerprint for the structure of the document.\n\nImportantly, this type of fingerprinting ignores the leaf values...we just want to fingerprint the JSON keys themselves.\n\n``` json\n{\n  \"fingerprint\": {\n    \"type\": \"structure\",\n    \"root\": [\"foo\"],\n    \"recursive\": true,\n    \"hash\": \"murmur3\",\n    \"hash_all\": true\n  }\n}\n```\n- `root`: defines where to start recursing, in case you only care about a portion of the document.  Could be omitted or set to `\"*\"` to process the entire document\n- `recursive` if you want the processor to fingerprint all the layers.  False if you just want the top-level of keys hashed.\n- `hash`: murmur, minhash, etc\n- `hash_all`: if all the hashes should be hashed together to build a final fingerprint\n\nAnd the new document:\n\n``` json\n{\n  \"foo\": {\n    \"bar\": {\n      \"baz\": \"buzz\"\n    },\n    \"beep\": {\n      \"boop\": \"bop\"\n    }\n  },\n  \"fingerprint_level1\": 001734,\n  \"fingerprint_level2\": 992727,\n  \"fingerprint_level3\": 110293,\n  \"fingerprint_all\": 235240\n}\n```\n\nInstead of a fingerprint-per-field, we now have one per \"level\".  \n\nI can think of a number of objections to both of these, but this should at least kick off the discussion :)\n\n/cc @martijnvg \n"},"comment":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/comments/374309550","html_url":"https://github.com/elastic/elasticsearch/issues/16938#issuecomment-374309550","issue_url":"https://api.github.com/repos/elastic/elasticsearch/issues/16938","id":374309550,"user":{"login":"talevy","id":388837,"avatar_url":"https://avatars0.githubusercontent.com/u/388837?v=4","gravatar_id":"","url":"https://api.github.com/users/talevy","html_url":"https://github.com/talevy","followers_url":"https://api.github.com/users/talevy/followers","following_url":"https://api.github.com/users/talevy/following{/other_user}","gists_url":"https://api.github.com/users/talevy/gists{/gist_id}","starred_url":"https://api.github.com/users/talevy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/talevy/subscriptions","organizations_url":"https://api.github.com/users/talevy/orgs","repos_url":"https://api.github.com/users/talevy/repos","events_url":"https://api.github.com/users/talevy/events{/privacy}","received_events_url":"https://api.github.com/users/talevy/received_events","type":"User","site_admin":false},"created_at":"2018-03-19T18:03:51Z","updated_at":"2018-03-19T18:03:51Z","author_association":"CONTRIBUTOR","body":"> I'll leave it up to @talevy if he wants to reuse this issue, or open a new ticket, for his implementation. :)\r\n\r\n> @talevy do we have another opened issue or is it just a matter of days/hours to have a PR on it so we can wait for the description in the PR?\r\n\r\nI'll open a PR with a description of its features and link to this issue to preserve a breadcrumb to this discussion"}}
{"actor":{"display_login":"talevy"}}