{"action":"closed","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/18266","repository_url":"https://api.github.com/repos/elastic/elasticsearch","labels_url":"https://api.github.com/repos/elastic/elasticsearch/issues/18266/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/issues/18266/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch/issues/18266/events","html_url":"https://github.com/elastic/elasticsearch/issues/18266","id":154185417,"number":18266,"title":"Fingerprint analyzer asciifolds AFTER deduplication","user":{"login":"clintongormley","id":56599,"avatar_url":"https://avatars.githubusercontent.com/u/56599?v=3","gravatar_id":"","url":"https://api.github.com/users/clintongormley","html_url":"https://github.com/clintongormley","followers_url":"https://api.github.com/users/clintongormley/followers","following_url":"https://api.github.com/users/clintongormley/following{/other_user}","gists_url":"https://api.github.com/users/clintongormley/gists{/gist_id}","starred_url":"https://api.github.com/users/clintongormley/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/clintongormley/subscriptions","organizations_url":"https://api.github.com/users/clintongormley/orgs","repos_url":"https://api.github.com/users/clintongormley/repos","events_url":"https://api.github.com/users/clintongormley/events{/privacy}","received_events_url":"https://api.github.com/users/clintongormley/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/:Analysis","name":":Analysis","color":"009800"},{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/discuss","name":"discuss","color":"fbca04"},{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/enhancement","name":"enhancement","color":"4a4ea8"}],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2016-05-11T08:17:16Z","updated_at":"2016-05-12T13:34:15Z","closed_at":"2016-05-12T13:34:15Z","body":"As currently implemented, the `fingerprint` analyzer performs asciifolding after terms have been deduplicated.  In other words, the following:\r\n\r\n    POST my_index/_analyze\r\n    {\r\n      \"analyzer\": \"fingerprint\",\r\n      \"text\": \"Yes Godel yes, Gödel said this sentence is consistent and.\"\r\n    }\r\n\r\nproduces the token `and consistent godel godel is said sentence this yes`\r\n\r\nI realise that this is as described in [OpenRefine](https://github.com/OpenRefine/OpenRefine/blob/master/main/src/com/google/refine/clustering/binning/FingerprintKeyer.java#L68) but I'm wondering if it makes sense?\r\n\r\nWe have the `preserve_original` option to the parameter which results in two terms: one with extended characters and one without:\r\n\r\n    PUT my_index\r\n    {\r\n      \"settings\": {\r\n        \"analysis\": {\r\n          \"analyzer\": {\r\n            \"my_fingerprint_analyzer\": {\r\n              \"type\": \"fingerprint\",\r\n              \"preserve_original\": true\r\n            }\r\n          }\r\n        }\r\n      }\r\n    }\r\n\r\n    POST my_index/_analyze\r\n    {\r\n      \"analyzer\": \"my_fingerprint_analyzer\",\r\n      \"text\": \"Yes Godel yes, Gödel said this sentence is consistent and.\"\r\n    }\r\n\r\nThis currently produces: \r\n\r\n* `and consistent godel godel is said sentence this yes`\r\n* `and consistent godel gödel is said sentence this yes`\r\n\r\nI'm thinking it would make more sense to:\r\n\r\n* asciifold\r\n* dedupe\r\n* then concatenate\r\n\r\nIn which case the standard output would be: `and consistent godel is said sentence this yes` and with `preserve_original`:\r\n\r\n* `and consistent godel is said sentence this yes`\r\n* `and consistent godel gödel is said sentence this yes`\r\n\r\n@markharwood @polyfractal what do you think?"}}