{"action":"created","comment":{"url":"https://api.github.com/repos/elastic/elasticsearch/pulls/comments/63885132","id":63885132,"diff_hunk":"@@ -459,4 +472,83 @@ public Weight createNormalizedWeight(Query query, boolean needsScores) throws IO\n         }\n     }\n \n+    private static PercolateQuery.QueryStore createStore(PercolatorFieldMapper.PercolatorFieldType fieldType,\n+                                                         QueryShardContext context,\n+                                                         boolean mapUnmappedFieldsAsString) {\n+        return ctx -> {\n+            LeafReader leafReader = ctx.reader();\n+            BinaryDocValues binaryDocValues = leafReader.getBinaryDocValues(fieldType.getQueryBuilderFieldName());\n+            Bits bits = leafReader.getDocsWithField(fieldType.getQueryBuilderFieldName());\n+            return docId -> {\n+                if (bits.get(docId)) {\n+                    BytesRef qbSource = binaryDocValues.get(docId);\n+                    if (qbSource.length > 0) {\n+                        XContent xContent = PercolatorFieldMapper.QUERY_BUILDER_CONTENT_TYPE.xContent();\n+                        try (XContentParser sourceParser = xContent.createParser(qbSource.bytes, qbSource.offset, qbSource.length)) {\n+                            return parseQuery(context, mapUnmappedFieldsAsString, sourceParser);\n+                        }\n+                    } else {\n+                        return null;\n+                    }\n+                } else {\n+                    return null;\n+                }\n+            };\n+        };\n+    }\n+\n+    private static PercolateQuery.QueryStore createLegacyStore(QueryShardContext context, boolean mapUnmappedFieldsAsString) {\n+        return ctx -> {\n+            LeafReader leafReader = ctx.reader();\n+            return docId -> {\n+                LegacyQueryFieldVisitor visitor = new LegacyQueryFieldVisitor();\n+                leafReader.document(docId, visitor);\n+                try (XContentParser sourceParser = XContentHelper.createParser(visitor.source)) {","path":"core/src/main/java/org/elasticsearch/index/query/PercolateQueryBuilder.java","position":142,"original_position":142,"commit_id":"0eb6065ec4efb40ad31345d42d02498c5eaaa015","original_commit_id":"0eb6065ec4efb40ad31345d42d02498c5eaaa015","user":{"login":"martijnvg","id":580421,"avatar_url":"https://avatars.githubusercontent.com/u/580421?v=3","gravatar_id":"","url":"https://api.github.com/users/martijnvg","html_url":"https://github.com/martijnvg","followers_url":"https://api.github.com/users/martijnvg/followers","following_url":"https://api.github.com/users/martijnvg/following{/other_user}","gists_url":"https://api.github.com/users/martijnvg/gists{/gist_id}","starred_url":"https://api.github.com/users/martijnvg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/martijnvg/subscriptions","organizations_url":"https://api.github.com/users/martijnvg/orgs","repos_url":"https://api.github.com/users/martijnvg/repos","events_url":"https://api.github.com/users/martijnvg/events{/privacy}","received_events_url":"https://api.github.com/users/martijnvg/received_events","type":"User","site_admin":false},"body":"+1","created_at":"2016-05-19T14:11:55Z","updated_at":"2016-05-19T14:11:55Z","html_url":"https://github.com/elastic/elasticsearch/pull/18434#discussion_r63885132","pull_request_url":"https://api.github.com/repos/elastic/elasticsearch/pulls/18434","_links":{"self":{"href":"https://api.github.com/repos/elastic/elasticsearch/pulls/comments/63885132"},"html":{"href":"https://github.com/elastic/elasticsearch/pull/18434#discussion_r63885132"},"pull_request":{"href":"https://api.github.com/repos/elastic/elasticsearch/pulls/18434"}}},"pull_request":{"url":"https://api.github.com/repos/elastic/elasticsearch/pulls/18434","id":70491282,"html_url":"https://github.com/elastic/elasticsearch/pull/18434","diff_url":"https://github.com/elastic/elasticsearch/pull/18434.diff","patch_url":"https://github.com/elastic/elasticsearch/pull/18434.patch","issue_url":"https://api.github.com/repos/elastic/elasticsearch/issues/18434","number":18434,"state":"open","locked":false,"title":"Remove percolator cache","user":{"login":"martijnvg","id":580421,"avatar_url":"https://avatars.githubusercontent.com/u/580421?v=3","gravatar_id":"","url":"https://api.github.com/users/martijnvg","html_url":"https://github.com/martijnvg","followers_url":"https://api.github.com/users/martijnvg/followers","following_url":"https://api.github.com/users/martijnvg/following{/other_user}","gists_url":"https://api.github.com/users/martijnvg/gists{/gist_id}","starred_url":"https://api.github.com/users/martijnvg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/martijnvg/subscriptions","organizations_url":"https://api.github.com/users/martijnvg/orgs","repos_url":"https://api.github.com/users/martijnvg/repos","events_url":"https://api.github.com/users/martijnvg/events{/privacy}","received_events_url":"https://api.github.com/users/martijnvg/received_events","type":"User","site_admin":false},"body":"Before 5.0 it was required that the percolator queries were cached in jvm heap as Lucene queries for two reasons:\r\n* Performance. The percolator evaluated all percolator queries all the time. There was no pre-selecting queries optimization that selects candidate matches like we have today.\r\n* Updates made to percolator queries were visible in realtime, today these changes are visible in near realtime. So updating no longer requires the percolator to have the queries in jvm heap.\r\n    \r\nSo having the percolator queries in jvm heap via the percolator cache is now less attractive. Especially when there are many percolator queries then these queries can consume many GBs of jvm heap. Removing the percolator cache does make the percolate query slower compared to how the execution time in 5.0.0-alpha1 and alpha2, but it is still significantly faster compared to 2.x and before.\r\n\r\nI think removing the percolator cache is the right tradeoff. I've been running some tests with an index that contains ~35M queries. With this change this change the used heap space is ~250MB and without 24GB. The query time slowdown compared with this change depends on how many candidate queries need to be evaluated if they actually match. With some test queries I have seen a slow down up to 5 times. However if query time is compared with how things work in 2.x and before then query time is still significantly better with this change.\r\n\r\nAlso if this gets merged we can improve other things, so that the slowdown introduced by this change get smaller:\r\n* For percolator queries that are a disjunction query we know that they will match if they are are a candidate match. So there is no need to evaluate these queries by the MemoryIndex.\r\n* Add range query extract logic, so that we can intelligently pre-select percolator queries that contain range queries.\r\n* Right now we store the xcontent representation of the percolator queries. We can instead store the binary representation which improve query time (parsing logic takes less time) for this change at the cost of bwc for stored percolator queries. The binary format of query builders has bwc within a major version as apposed for across a major version for the xcontent format. So that isn't too bad and with the reindex api, updating the percolator queries is now much more pleasant. So this could be explored as well.\r\n\r\nNote the breaking part is the removal of percolator related statistics from the stats and cat APIs that are no longer needed now.","created_at":"2016-05-18T09:00:27Z","updated_at":"2016-05-19T14:11:55Z","closed_at":null,"merged_at":null,"merge_commit_sha":"9689a466d6c33acd1a52dd542bc66cc4f30b326d","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/elastic/elasticsearch/pulls/18434/commits","review_comments_url":"https://api.github.com/repos/elastic/elasticsearch/pulls/18434/comments","review_comment_url":"https://api.github.com/repos/elastic/elasticsearch/pulls/comments{/number}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/issues/18434/comments","statuses_url":"https://api.github.com/repos/elastic/elasticsearch/statuses/0eb6065ec4efb40ad31345d42d02498c5eaaa015","head":{"label":"martijnvg:remove_percolator_cache","ref":"remove_percolator_cache","sha":"0eb6065ec4efb40ad31345d42d02498c5eaaa015","user":{"login":"martijnvg","id":580421,"avatar_url":"https://avatars.githubusercontent.com/u/580421?v=3","gravatar_id":"","url":"https://api.github.com/users/martijnvg","html_url":"https://github.com/martijnvg","followers_url":"https://api.github.com/users/martijnvg/followers","following_url":"https://api.github.com/users/martijnvg/following{/other_user}","gists_url":"https://api.github.com/users/martijnvg/gists{/gist_id}","starred_url":"https://api.github.com/users/martijnvg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/martijnvg/subscriptions","organizations_url":"https://api.github.com/users/martijnvg/orgs","repos_url":"https://api.github.com/users/martijnvg/repos","events_url":"https://api.github.com/users/martijnvg/events{/privacy}","received_events_url":"https://api.github.com/users/martijnvg/received_events","type":"User","site_admin":false},"repo":{"id":3584739,"name":"elasticsearch","full_name":"martijnvg/elasticsearch","owner":{"login":"martijnvg","id":580421,"avatar_url":"https://avatars.githubusercontent.com/u/580421?v=3","gravatar_id":"","url":"https://api.github.com/users/martijnvg","html_url":"https://github.com/martijnvg","followers_url":"https://api.github.com/users/martijnvg/followers","following_url":"https://api.github.com/users/martijnvg/following{/other_user}","gists_url":"https://api.github.com/users/martijnvg/gists{/gist_id}","starred_url":"https://api.github.com/users/martijnvg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/martijnvg/subscriptions","organizations_url":"https://api.github.com/users/martijnvg/orgs","repos_url":"https://api.github.com/users/martijnvg/repos","events_url":"https://api.github.com/users/martijnvg/events{/privacy}","received_events_url":"https://api.github.com/users/martijnvg/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/martijnvg/elasticsearch","description":"Open Source, Distributed, RESTful Search Engine with grouping","fork":true,"url":"https://api.github.com/repos/martijnvg/elasticsearch","forks_url":"https://api.github.com/repos/martijnvg/elasticsearch/forks","keys_url":"https://api.github.com/repos/martijnvg/elasticsearch/keys{/key_id}","collaborators_url":"https://api.github.com/repos/martijnvg/elasticsearch/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/martijnvg/elasticsearch/teams","hooks_url":"https://api.github.com/repos/martijnvg/elasticsearch/hooks","issue_events_url":"https://api.github.com/repos/martijnvg/elasticsearch/issues/events{/number}","events_url":"https://api.github.com/repos/martijnvg/elasticsearch/events","assignees_url":"https://api.github.com/repos/martijnvg/elasticsearch/assignees{/user}","branches_url":"https://api.github.com/repos/martijnvg/elasticsearch/branches{/branch}","tags_url":"https://api.github.com/repos/martijnvg/elasticsearch/tags","blobs_url":"https://api.github.com/repos/martijnvg/elasticsearch/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/martijnvg/elasticsearch/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/martijnvg/elasticsearch/git/refs{/sha}","trees_url":"https://api.github.com/repos/martijnvg/elasticsearch/git/trees{/sha}","statuses_url":"https://api.github.com/repos/martijnvg/elasticsearch/statuses/{sha}","languages_url":"https://api.github.com/repos/martijnvg/elasticsearch/languages","stargazers_url":"https://api.github.com/repos/martijnvg/elasticsearch/stargazers","contributors_url":"https://api.github.com/repos/martijnvg/elasticsearch/contributors","subscribers_url":"https://api.github.com/repos/martijnvg/elasticsearch/subscribers","subscription_url":"https://api.github.com/repos/martijnvg/elasticsearch/subscription","commits_url":"https://api.github.com/repos/martijnvg/elasticsearch/commits{/sha}","git_commits_url":"https://api.github.com/repos/martijnvg/elasticsearch/git/commits{/sha}","comments_url":"https://api.github.com/repos/martijnvg/elasticsearch/comments{/number}","issue_comment_url":"https://api.github.com/repos/martijnvg/elasticsearch/issues/comments{/number}","contents_url":"https://api.github.com/repos/martijnvg/elasticsearch/contents/{+path}","compare_url":"https://api.github.com/repos/martijnvg/elasticsearch/compare/{base}...{head}","merges_url":"https://api.github.com/repos/martijnvg/elasticsearch/merges","archive_url":"https://api.github.com/repos/martijnvg/elasticsearch/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/martijnvg/elasticsearch/downloads","issues_url":"https://api.github.com/repos/martijnvg/elasticsearch/issues{/number}","pulls_url":"https://api.github.com/repos/martijnvg/elasticsearch/pulls{/number}","milestones_url":"https://api.github.com/repos/martijnvg/elasticsearch/milestones{/number}","notifications_url":"https://api.github.com/repos/martijnvg/elasticsearch/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/martijnvg/elasticsearch/labels{/name}","releases_url":"https://api.github.com/repos/martijnvg/elasticsearch/releases{/id}","deployments_url":"https://api.github.com/repos/martijnvg/elasticsearch/deployments","created_at":"2012-02-29T18:56:33Z","updated_at":"2014-08-09T22:19:31Z","pushed_at":"2016-05-19T09:06:09Z","git_url":"git://github.com/martijnvg/elasticsearch.git","ssh_url":"git@github.com:martijnvg/elasticsearch.git","clone_url":"https://github.com/martijnvg/elasticsearch.git","svn_url":"https://github.com/martijnvg/elasticsearch","homepage":"http://elasticsearch.org","size":202724,"stargazers_count":12,"watchers_count":12,"language":"Java","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":2,"mirror_url":null,"open_issues_count":0,"forks":2,"open_issues":0,"watchers":12,"default_branch":"master"}},"base":{"label":"elastic:master","ref":"master","sha":"6dacac49b394df9c96d20f3b4c74c8a8d7e22558","user":{"login":"elastic","id":6764390,"avatar_url":"https://avatars.githubusercontent.com/u/6764390?v=3","gravatar_id":"","url":"https://api.github.com/users/elastic","html_url":"https://github.com/elastic","followers_url":"https://api.github.com/users/elastic/followers","following_url":"https://api.github.com/users/elastic/following{/other_user}","gists_url":"https://api.github.com/users/elastic/gists{/gist_id}","starred_url":"https://api.github.com/users/elastic/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/elastic/subscriptions","organizations_url":"https://api.github.com/users/elastic/orgs","repos_url":"https://api.github.com/users/elastic/repos","events_url":"https://api.github.com/users/elastic/events{/privacy}","received_events_url":"https://api.github.com/users/elastic/received_events","type":"Organization","site_admin":false},"repo":{"id":507775,"name":"elasticsearch","full_name":"elastic/elasticsearch","owner":{"login":"elastic","id":6764390,"avatar_url":"https://avatars.githubusercontent.com/u/6764390?v=3","gravatar_id":"","url":"https://api.github.com/users/elastic","html_url":"https://github.com/elastic","followers_url":"https://api.github.com/users/elastic/followers","following_url":"https://api.github.com/users/elastic/following{/other_user}","gists_url":"https://api.github.com/users/elastic/gists{/gist_id}","starred_url":"https://api.github.com/users/elastic/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/elastic/subscriptions","organizations_url":"https://api.github.com/users/elastic/orgs","repos_url":"https://api.github.com/users/elastic/repos","events_url":"https://api.github.com/users/elastic/events{/privacy}","received_events_url":"https://api.github.com/users/elastic/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/elastic/elasticsearch","description":"Open Source, Distributed, RESTful Search Engine","fork":false,"url":"https://api.github.com/repos/elastic/elasticsearch","forks_url":"https://api.github.com/repos/elastic/elasticsearch/forks","keys_url":"https://api.github.com/repos/elastic/elasticsearch/keys{/key_id}","collaborators_url":"https://api.github.com/repos/elastic/elasticsearch/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/elastic/elasticsearch/teams","hooks_url":"https://api.github.com/repos/elastic/elasticsearch/hooks","issue_events_url":"https://api.github.com/repos/elastic/elasticsearch/issues/events{/number}","events_url":"https://api.github.com/repos/elastic/elasticsearch/events","assignees_url":"https://api.github.com/repos/elastic/elasticsearch/assignees{/user}","branches_url":"https://api.github.com/repos/elastic/elasticsearch/branches{/branch}","tags_url":"https://api.github.com/repos/elastic/elasticsearch/tags","blobs_url":"https://api.github.com/repos/elastic/elasticsearch/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/elastic/elasticsearch/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/elastic/elasticsearch/git/refs{/sha}","trees_url":"https://api.github.com/repos/elastic/elasticsearch/git/trees{/sha}","statuses_url":"https://api.github.com/repos/elastic/elasticsearch/statuses/{sha}","languages_url":"https://api.github.com/repos/elastic/elasticsearch/languages","stargazers_url":"https://api.github.com/repos/elastic/elasticsearch/stargazers","contributors_url":"https://api.github.com/repos/elastic/elasticsearch/contributors","subscribers_url":"https://api.github.com/repos/elastic/elasticsearch/subscribers","subscription_url":"https://api.github.com/repos/elastic/elasticsearch/subscription","commits_url":"https://api.github.com/repos/elastic/elasticsearch/commits{/sha}","git_commits_url":"https://api.github.com/repos/elastic/elasticsearch/git/commits{/sha}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/comments{/number}","issue_comment_url":"https://api.github.com/repos/elastic/elasticsearch/issues/comments{/number}","contents_url":"https://api.github.com/repos/elastic/elasticsearch/contents/{+path}","compare_url":"https://api.github.com/repos/elastic/elasticsearch/compare/{base}...{head}","merges_url":"https://api.github.com/repos/elastic/elasticsearch/merges","archive_url":"https://api.github.com/repos/elastic/elasticsearch/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/elastic/elasticsearch/downloads","issues_url":"https://api.github.com/repos/elastic/elasticsearch/issues{/number}","pulls_url":"https://api.github.com/repos/elastic/elasticsearch/pulls{/number}","milestones_url":"https://api.github.com/repos/elastic/elasticsearch/milestones{/number}","notifications_url":"https://api.github.com/repos/elastic/elasticsearch/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/elastic/elasticsearch/labels{/name}","releases_url":"https://api.github.com/repos/elastic/elasticsearch/releases{/id}","deployments_url":"https://api.github.com/repos/elastic/elasticsearch/deployments","created_at":"2010-02-08T13:20:56Z","updated_at":"2016-05-19T14:03:22Z","pushed_at":"2016-05-19T13:47:34Z","git_url":"git://github.com/elastic/elasticsearch.git","ssh_url":"git@github.com:elastic/elasticsearch.git","clone_url":"https://github.com/elastic/elasticsearch.git","svn_url":"https://github.com/elastic/elasticsearch","homepage":"https://www.elastic.co/products/elasticsearch","size":316724,"stargazers_count":16520,"watchers_count":16520,"language":"Java","has_issues":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":5590,"mirror_url":null,"open_issues_count":1097,"forks":5590,"open_issues":1097,"watchers":16520,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/elastic/elasticsearch/pulls/18434"},"html":{"href":"https://github.com/elastic/elasticsearch/pull/18434"},"issue":{"href":"https://api.github.com/repos/elastic/elasticsearch/issues/18434"},"comments":{"href":"https://api.github.com/repos/elastic/elasticsearch/issues/18434/comments"},"review_comments":{"href":"https://api.github.com/repos/elastic/elasticsearch/pulls/18434/comments"},"review_comment":{"href":"https://api.github.com/repos/elastic/elasticsearch/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/elastic/elasticsearch/pulls/18434/commits"},"statuses":{"href":"https://api.github.com/repos/elastic/elasticsearch/statuses/0eb6065ec4efb40ad31345d42d02498c5eaaa015"}}}}