{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/42899","repository_url":"https://api.github.com/repos/elastic/elasticsearch","labels_url":"https://api.github.com/repos/elastic/elasticsearch/issues/42899/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/issues/42899/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch/issues/42899/events","html_url":"https://github.com/elastic/elasticsearch/issues/42899","id":452568650,"node_id":"MDU6SXNzdWU0NTI1Njg2NTA=","number":42899,"title":"Elasticsearch 2.4.3 very high cpu usage","user":{"login":"fgdgtz183","id":19529758,"node_id":"MDQ6VXNlcjE5NTI5NzU4","avatar_url":"https://avatars3.githubusercontent.com/u/19529758?v=4","gravatar_id":"","url":"https://api.github.com/users/fgdgtz183","html_url":"https://github.com/fgdgtz183","followers_url":"https://api.github.com/users/fgdgtz183/followers","following_url":"https://api.github.com/users/fgdgtz183/following{/other_user}","gists_url":"https://api.github.com/users/fgdgtz183/gists{/gist_id}","starred_url":"https://api.github.com/users/fgdgtz183/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fgdgtz183/subscriptions","organizations_url":"https://api.github.com/users/fgdgtz183/orgs","repos_url":"https://api.github.com/users/fgdgtz183/repos","events_url":"https://api.github.com/users/fgdgtz183/events{/privacy}","received_events_url":"https://api.github.com/users/fgdgtz183/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2019-06-05T15:30:17Z","updated_at":"2019-06-05T20:58:20Z","closed_at":"2019-06-05T20:58:20Z","author_association":"NONE","body":"<!--\r\n\r\n** Please read the guidelines below. **\r\n\r\nIssues that do not follow these guidelines are likely to be closed.\r\n\r\n1.  GitHub is reserved for bug reports and feature requests. The best place to\r\n    ask a general question is at the Elastic [forums](https://discuss.elastic.co).\r\n    GitHub is not the place for general questions.\r\n\r\n2.  Is this bug report or feature request for a supported OS? If not, it\r\n    is likely to be closed.  See https://www.elastic.co/support/matrix#show_os\r\n\r\n3.  Please fill out EITHER the feature request block or the bug report block\r\n    below, and delete the other block.\r\n\r\n-->\r\n\r\n<!-- Feature request -->\r\n\r\n**Describe the feature**:\r\n\r\n<!-- Bug report -->\r\n\r\n**Elasticsearch version** (`bin/elasticsearch --version`):\r\nVersion: 2.4.3\r\n\r\n**Plugins installed**:\r\ndelete_by_query,head,ik\r\n\r\n**JVM version** (`java -version`):\r\nVersion:1.8.0_121\r\nJava HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)\r\n\r\n**OS version** (`uname -a` if on a Unix-like system):\r\nLinux version 4.4.0-93-generic (buildd@lcy01-28) (gcc version 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04.3) ) #116\r\n~14.04.1-Ubuntu SMP Mon Aug 14 16:07:05 UTC 2017\r\n\r\n**Description of the problem including expected versus actual behavior**:\r\n\r\nMy production ES nodes just only one,it is master node and also the data node ,it have 4 CPUs, and 8 GB RAM,ES heap size is 3900M，1 indices\r\n\r\nelasticsearch.yml:\r\nnode.name: node_master\r\nnode.master: true\r\nnode.data: true\r\nhttp.cors.enabled: true\r\nhttp.cors.allow-origin: \"*\"\r\nhttp.cors.allow-headers: Authorization\r\naction.auto_create_index: true\r\nscript.inline: on\r\nscript.indexed: on\r\n\r\nnohup ./elasticsearch  -Des.insecure.allow.root=true\r\n\r\nWe use logstash to synchronize MySQL data to ES and use the ES client api to batch delete expired data every day at 1am，online data is maintained at around 500 Mbp，it has been running normally for more than a year.\r\n\r\nlogstash jdbc.conf:\r\njdbc_driver_class => \"com.mysql.jdbc.Driver\"\r\njdbc_paging_enabled => \"true\"\r\njdbc_page_size => \"50000\"\r\njdbc_fetch_size => \"50000\"\r\nuse_column_value => \"true\"\r\nrecord_last_run => \"true\"\r\ntracking_column => \"updatetime\"\r\nlast_run_metadata_path => \"/usr/elasticsearch/logstash-2.4.1/run_metadata\"\r\nstatement_filepath => \"jdbc.sql\"\r\n\r\n\r\nIt suddenly had a problem last night, the cpu was very high, I used the top command to query, the result shows 376%.\r\n![image](https://user-images.githubusercontent.com/19529758/58966916-bf677600-87e5-11e9-942e-19a3ec66fcf3.png)\r\n\r\njstat -gc pid, I didn't see a frequent GC happening\r\n\r\nI tried to rebuild  indices and the problem still exists.\r\n\r\nThe strange thing is that I can use the ES of the test environment to run normally.It have 2 CPUS and 4G RAM,ES heap size is 1900M\r\n\r\n**Provide logs (if relevant)**:\r\nHot threads \r\n\r\n1:\r\n Hot threads at 2019-06-04T21:57:10.910Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:\r\n   \r\n   48.8% (243.7ms out of 500ms) cpu usage by thread 'elasticsearch[uj_node_master][search][T#2]'\r\n     4/10 snapshots sharing following 15 elements\r\n       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:818)\r\n       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:535)\r\n       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:384)\r\n       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:113)\r\n       org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:372)\r\n       org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:385)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:368)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:365)\r\n       org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)\r\n       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)\r\n       org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:378)\r\n       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\r\n       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n       java.lang.Thread.run(Thread.java:745)\r\n     2/10 snapshots sharing following 20 elements\r\n       org.apache.lucene.codecs.compressing.LZ4.decompress(LZ4.java:90)\r\n       org.apache.lucene.codecs.compressing.CompressionMode$4.decompress(CompressionMode.java:138)\r\n       org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader$BlockState.document(CompressingStoredFieldsReader.java:560)\r\n       org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.document(CompressingStoredFieldsReader.java:576)\r\n       org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.visitDocument(CompressingStoredFieldsReader.java:583)\r\n       org.apache.lucene.index.CodecReader.document(CodecReader.java:81)\r\n       org.apache.lucene.index.FilterLeafReader.document(FilterLeafReader.java:405)\r\n       org.elasticsearch.search.fetch.FetchPhase.loadStoredFields(FetchPhase.java:413)\r\n       org.elasticsearch.search.fetch.FetchPhase.createSearchHit(FetchPhase.java:213)\r\n       org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:178)\r\n       org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:605)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$FetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:408)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$FetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:405)\r\n       org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)\r\n       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)\r\n       org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:378)\r\n       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\r\n       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n       java.lang.Thread.run(Thread.java:745)\r\n     2/10 snapshots sharing following 20 elements\r\n       org.elasticsearch.common.xcontent.support.AbstractXContentParser.readValue(AbstractXContentParser.java:307)\r\n       org.elasticsearch.common.xcontent.support.AbstractXContentParser.readMap(AbstractXContentParser.java:275)\r\n       org.elasticsearch.common.xcontent.support.AbstractXContentParser.readMap(AbstractXContentParser.java:246)\r\n       org.elasticsearch.common.xcontent.support.AbstractXContentParser.map(AbstractXContentParser.java:209)\r\n       org.elasticsearch.common.xcontent.XContentHelper.convertToMap(XContentHelper.java:83)\r\n       org.elasticsearch.search.lookup.SourceLookup.sourceAsMapAndType(SourceLookup.java:88)\r\n       org.elasticsearch.search.lookup.SourceLookup.loadSourceIfNeeded(SourceLookup.java:64)\r\n       org.elasticsearch.search.lookup.SourceLookup.extractRawValues(SourceLookup.java:130)\r\n       org.elasticsearch.search.fetch.FetchPhase.createSearchHit(FetchPhase.java:241)\r\n       org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:178)\r\n       org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:605)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$FetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:408)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$FetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:405)\r\n       org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)\r\n       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)\r\n       org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:378)\r\n       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\r\n       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n       java.lang.Thread.run(Thread.java:745)\r\n     2/10 snapshots sharing following 29 elements\r\n       org.apache.lucene.util.fst.FST.findTargetArc(FST.java:1186)\r\n       org.apache.lucene.codecs.blocktree.SegmentTermsEnum.seekExact(SegmentTermsEnum.java:483)\r\n       org.apache.lucene.index.TermContext.build(TermContext.java:94)\r\n       org.apache.lucene.search.TermQuery.createWeight(TermQuery.java:192)\r\n       org.apache.lucene.search.IndexSearcher.createWeight(IndexSearcher.java:904)\r\n       org.apache.lucene.search.BooleanWeight.<init>(BooleanWeight.java:57)\r\n       org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:239)\r\n       org.apache.lucene.search.IndexSearcher.createWeight(IndexSearcher.java:904)\r\n       org.apache.lucene.search.ConstantScoreQuery.createWeight(ConstantScoreQuery.java:119)\r\n       org.apache.lucene.search.IndexSearcher.createWeight(IndexSearcher.java:904)\r\n       org.apache.lucene.search.BooleanWeight.<init>(BooleanWeight.java:57)\r\n       org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:239)\r\n       org.apache.lucene.search.IndexSearcher.createWeight(IndexSearcher.java:904)\r\n       org.apache.lucene.search.IndexSearcher.createNormalizedWeight(IndexSearcher.java:887)\r\n       org.elasticsearch.search.internal.ContextIndexSearcher.createNormalizedWeight(ContextIndexSearcher.java:100)\r\n       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:535)\r\n       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:384)\r\n       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:113)\r\n       org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:372)\r\n       org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:385)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:368)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:365)\r\n       org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)\r\n       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)\r\n       org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:378)\r\n       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\r\n       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n       java.lang.Thread.run(Thread.java:745)\r\n   \r\n   46.6% (233ms out of 500ms) cpu usage by thread 'elasticsearch[uj_node_master][search][T#6]'\r\n     3/10 snapshots sharing following 29 elements\r\n       org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:219)\r\n       org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:172)\r\n       org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)\r\n       org.apache.lucene.search.LRUQueryCache.cacheImpl(LRUQueryCache.java:422)\r\n       org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.cache(LRUQueryCache.java:608)\r\n       org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorer(LRUQueryCache.java:625)\r\n       org.elasticsearch.indices.cache.query.IndicesQueryCache$CachingWeightWrapper.scorer(IndicesQueryCache.java:263)\r\n       org.apache.lucene.search.BooleanWeight.scorer(BooleanWeight.java:389)\r\n       org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorer(LRUQueryCache.java:628)\r\n       org.elasticsearch.indices.cache.query.IndicesQueryCache$CachingWeightWrapper.scorer(IndicesQueryCache.java:263)\r\n       org.apache.lucene.search.BooleanWeight.scorer(BooleanWeight.java:389)\r\n       org.apache.lucene.search.BooleanWeight.scorer(BooleanWeight.java:389)\r\n       org.apache.lucene.search.Weight.bulkScorer(Weight.java:135)\r\n       org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:370)\r\n       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:818)\r\n       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:535)\r\n       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:384)\r\n       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:113)\r\n       org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:372)\r\n       org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:385)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:368)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:365)\r\n       org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)\r\n       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)\r\n       org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:378)\r\n       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\r\n       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n       java.lang.Thread.run(Thread.java:745)\r\n     2/10 snapshots sharing following 30 elements\r\n       org.apache.lucene.util.BitSet.or(BitSet.java:94)\r\n       org.apache.lucene.util.FixedBitSet.or(FixedBitSet.java:271)\r\n       org.apache.lucene.util.DocIdSetBuilder.add(DocIdSetBuilder.java:87)\r\n       org.apache.lucene.search.MultiTermQueryConstantScoreWrapper$1.rewrite(MultiTermQueryConstantScoreWrapper.java:180)\r\n       org.apache.lucene.search.MultiTermQueryConstantScoreWrapper$1.scorer(MultiTermQueryConstantScoreWrapper.java:213)\r\n       org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorer(LRUQueryCache.java:628)\r\n       org.elasticsearch.indices.cache.query.IndicesQueryCache$CachingWeightWrapper.scorer(IndicesQueryCache.java:263)\r\n       org.apache.lucene.search.BooleanWeight.scorer(BooleanWeight.java:389)\r\n       org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorer(LRUQueryCache.java:628)\r\n       org.elasticsearch.indices.cache.query.IndicesQueryCache$CachingWeightWrapper.scorer(IndicesQueryCache.java:263)\r\n       org.apache.lucene.search.BooleanWeight.scorer(BooleanWeight.java:389)\r\n       org.apache.lucene.search.Weight.bulkScorer(Weight.java:135)\r\n       org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:370)\r\n       org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.bulkScorer(LRUQueryCache.java:655)\r\n       org.elasticsearch.indices.cache.query.IndicesQueryCache$CachingWeightWrapper.bulkScorer(IndicesQueryCache.java:269)\r\n       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:818)\r\n       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:535)\r\n       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:384)\r\n       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:113)\r\n       org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:372)\r\n       org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:385)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:368)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:365)\r\n       org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)\r\n       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)\r\n       org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:378)\r\n       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\r\n       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n       java.lang.Thread.run(Thread.java:745)\r\n     5/10 snapshots sharing following 27 elements\r\n       org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:219)\r\n       org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:172)\r\n       org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)\r\n       org.apache.lucene.search.LRUQueryCache.cacheImpl(LRUQueryCache.java:422)\r\n       org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.cache(LRUQueryCache.java:608)\r\n       org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorer(LRUQueryCache.java:625)\r\n       org.elasticsearch.indices.cache.query.IndicesQueryCache$CachingWeightWrapper.scorer(IndicesQueryCache.java:263)\r\n       org.apache.lucene.search.BooleanWeight.scorer(BooleanWeight.java:389)\r\n       org.apache.lucene.search.Weight.bulkScorer(Weight.java:135)\r\n       org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:370)\r\n       org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.bulkScorer(LRUQueryCache.java:655)\r\n       org.elasticsearch.indices.cache.query.IndicesQueryCache$CachingWeightWrapper.bulkScorer(IndicesQueryCache.java:269)\r\n       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:818)\r\n       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:535)\r\n       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:384)\r\n       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:113)\r\n       org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:372)\r\n       org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:385)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:368)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:365)\r\n       org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)\r\n       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)\r\n       org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:378)\r\n       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\r\n       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n       java.lang.Thread.run(Thread.java:745)\r\n   \r\n   44.5% (222.3ms out of 500ms) cpu usage by thread 'elasticsearch[uj_node_master][search][T#7]'\r\n     6/10 snapshots sharing following 29 elements\r\n       org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:219)\r\n       org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:172)\r\n       org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)\r\n       org.apache.lucene.search.LRUQueryCache.cacheImpl(LRUQueryCache.java:422)\r\n       org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.cache(LRUQueryCache.java:608)\r\n       org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorer(LRUQueryCache.java:625)\r\n       org.elasticsearch.indices.cache.query.IndicesQueryCache$CachingWeightWrapper.scorer(IndicesQueryCache.java:263)\r\n       org.apache.lucene.search.BooleanWeight.scorer(BooleanWeight.java:389)\r\n       org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorer(LRUQueryCache.java:628)\r\n       org.elasticsearch.indices.cache.query.IndicesQueryCache$CachingWeightWrapper.scorer(IndicesQueryCache.java:263)\r\n       org.apache.lucene.search.BooleanWeight.scorer(BooleanWeight.java:389)\r\n       org.apache.lucene.search.BooleanWeight.scorer(BooleanWeight.java:389)\r\n       org.apache.lucene.search.Weight.bulkScorer(Weight.java:135)\r\n       org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:370)\r\n       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:818)\r\n       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:535)\r\n       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:384)\r\n       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:113)\r\n       org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:372)\r\n       org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:385)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:368)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:365)\r\n       org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)\r\n       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)\r\n       org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:378)\r\n       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\r\n       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n       java.lang.Thread.run(Thread.java:745)\r\n     2/10 snapshots sharing following 25 elements\r\n       org.apache.lucene.search.MinShouldMatchSumScorer.advanceTail(MinShouldMatchSumScorer.java:215)\r\n       org.apache.lucene.search.MinShouldMatchSumScorer.advanceTail(MinShouldMatchSumScorer.java:225)\r\n       org.apache.lucene.search.MinShouldMatchSumScorer.doNext(MinShouldMatchSumScorer.java:250)\r\n       org.apache.lucene.search.MinShouldMatchSumScorer.access$200(MinShouldMatchSumScorer.java:48)\r\n       org.apache.lucene.search.MinShouldMatchSumScorer$2.advance(MinShouldMatchSumScorer.java:192)\r\n       org.apache.lucene.search.ConjunctionDISI.doNext(ConjunctionDISI.java:178)\r\n       org.apache.lucene.search.ConjunctionDISI.nextDoc(ConjunctionDISI.java:207)\r\n       org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:219)\r\n       org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:172)\r\n       org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)\r\n       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:821)\r\n       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:535)\r\n       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:384)\r\n       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:113)\r\n       org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:372)\r\n       org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:385)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:368)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:365)\r\n       org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)\r\n       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)\r\n       org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:378)\r\n       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\r\n       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n       java.lang.Thread.run(Thread.java:745)\r\n     2/10 snapshots sharing following 14 elements\r\n       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:535)\r\n       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:384)\r\n       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:113)\r\n       org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:372)\r\n       org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:385)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:368)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:365)\r\n       org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)\r\n       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)\r\n       org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:378)\r\n       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\r\n       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n       java.lang.Thread.run(Thread.java:745)\r\n\r\n2:\r\nHot threads at 2019-06-04T21:57:32.810Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:\r\n   \r\n   46.5% (232.5ms out of 500ms) cpu usage by thread 'elasticsearch[uj_node_master][search][T#5]'\r\n     3/10 snapshots sharing following 30 elements\r\n       org.apache.lucene.util.BitSet.or(BitSet.java:94)\r\n       org.apache.lucene.util.FixedBitSet.or(FixedBitSet.java:271)\r\n       org.apache.lucene.util.DocIdSetBuilder.add(DocIdSetBuilder.java:87)\r\n       org.apache.lucene.search.MultiTermQueryConstantScoreWrapper$1.rewrite(MultiTermQueryConstantScoreWrapper.java:173)\r\n       org.apache.lucene.search.MultiTermQueryConstantScoreWrapper$1.scorer(MultiTermQueryConstantScoreWrapper.java:213)\r\n       org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorer(LRUQueryCache.java:628)\r\n       org.elasticsearch.indices.cache.query.IndicesQueryCache$CachingWeightWrapper.scorer(IndicesQueryCache.java:263)\r\n       org.apache.lucene.search.BooleanWeight.scorer(BooleanWeight.java:389)\r\n       org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorer(LRUQueryCache.java:628)\r\n       org.elasticsearch.indices.cache.query.IndicesQueryCache$CachingWeightWrapper.scorer(IndicesQueryCache.java:263)\r\n       org.apache.lucene.search.BooleanWeight.scorer(BooleanWeight.java:389)\r\n       org.apache.lucene.search.Weight.bulkScorer(Weight.java:135)\r\n       org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:370)\r\n       org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.bulkScorer(LRUQueryCache.java:655)\r\n       org.elasticsearch.indices.cache.query.IndicesQueryCache$CachingWeightWrapper.bulkScorer(IndicesQueryCache.java:269)\r\n       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:818)\r\n       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:535)\r\n       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:384)\r\n       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:113)\r\n       org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:372)\r\n       org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:385)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:368)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:365)\r\n       org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)\r\n       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)\r\n       org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:378)\r\n       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\r\n       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n       java.lang.Thread.run(Thread.java:745)\r\n     5/10 snapshots sharing following 26 elements\r\n       org.apache.lucene.search.MultiTermQueryConstantScoreWrapper$1.scorer(MultiTermQueryConstantScoreWrapper.java:213)\r\n       org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorer(LRUQueryCache.java:628)\r\n       org.elasticsearch.indices.cache.query.IndicesQueryCache$CachingWeightWrapper.scorer(IndicesQueryCache.java:263)\r\n       org.apache.lucene.search.BooleanWeight.scorer(BooleanWeight.java:389)\r\n       org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorer(LRUQueryCache.java:628)\r\n       org.elasticsearch.indices.cache.query.IndicesQueryCache$CachingWeightWrapper.scorer(IndicesQueryCache.java:263)\r\n       org.apache.lucene.search.BooleanWeight.scorer(BooleanWeight.java:389)\r\n       org.apache.lucene.search.Weight.bulkScorer(Weight.java:135)\r\n       org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:370)\r\n       org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.bulkScorer(LRUQueryCache.java:655)\r\n       org.elasticsearch.indices.cache.query.IndicesQueryCache$CachingWeightWrapper.bulkScorer(IndicesQueryCache.java:269)\r\n       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:818)\r\n       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:535)\r\n       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:384)\r\n       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:113)\r\n       org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:372)\r\n       org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:385)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:368)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:365)\r\n       org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)\r\n       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)\r\n       org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:378)\r\n       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\r\n       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n       java.lang.Thread.run(Thread.java:745)\r\n     2/10 snapshots sharing following 21 elements\r\n       java.util.HashMap.hash(HashMap.java:338)\r\n       java.util.HashMap.put(HashMap.java:611)\r\n       org.apache.lucene.search.Multiset.add(Multiset.java:91)\r\n       org.apache.lucene.search.BooleanQuery.<init>(BooleanQuery.java:171)\r\n       org.apache.lucene.search.BooleanQuery.<init>(BooleanQuery.java:42)\r\n       org.apache.lucene.search.BooleanQuery$Builder.build(BooleanQuery.java:146)\r\n       org.elasticsearch.search.internal.DefaultSearchContext.buildFilteredQuery(DefaultSearchContext.java:247)\r\n       org.elasticsearch.search.internal.DefaultSearchContext.preProcess(DefaultSearchContext.java:229)\r\n       org.elasticsearch.search.query.QueryPhase.preProcess(QueryPhase.java:103)\r\n       org.elasticsearch.search.SearchService.createContext(SearchService.java:689)\r\n       org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:633)\r\n       org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:377)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:368)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:365)\r\n       org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)\r\n       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)\r\n       org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:378)\r\n       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\r\n       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n       java.lang.Thread.run(Thread.java:745)\r\n   \r\n   41.5% (207.7ms out of 500ms) cpu usage by thread 'elasticsearch[uj_node_master][search][T#7]'\r\n     2/10 snapshots sharing following 38 elements\r\n       java.util.TreeMap.getEntry(TreeMap.java:359)\r\n       java.util.TreeMap.get(TreeMap.java:278)\r\n       org.apache.lucene.codecs.blocktree.BlockTreeTermsReader.terms(BlockTreeTermsReader.java:289)\r\n       org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.terms(PerFieldPostingsFormat.java:285)\r\n       org.apache.lucene.index.LeafReader.terms(LeafReader.java:207)\r\n       org.apache.lucene.search.TermQuery$TermWeight.getTermsEnum(TermQuery.java:123)\r\n       org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:102)\r\n       org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorer(LRUQueryCache.java:628)\r\n       org.elasticsearch.indices.cache.query.IndicesQueryCache$CachingWeightWrapper.scorer(IndicesQueryCache.java:263)\r\n       org.apache.lucene.search.BooleanWeight.scorer(BooleanWeight.java:389)\r\n       org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorer(LRUQueryCache.java:628)\r\n       org.elasticsearch.indices.cache.query.IndicesQueryCache$CachingWeightWrapper.scorer(IndicesQueryCache.java:263)\r\n       org.apache.lucene.search.MultiTermQueryConstantScoreWrapper$1.scorer(MultiTermQueryConstantScoreWrapper.java:215)\r\n       org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorer(LRUQueryCache.java:628)\r\n       org.elasticsearch.indices.cache.query.IndicesQueryCache$CachingWeightWrapper.scorer(IndicesQueryCache.java:263)\r\n       org.apache.lucene.search.BooleanWeight.scorer(BooleanWeight.java:389)\r\n       org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorer(LRUQueryCache.java:628)\r\n       org.elasticsearch.indices.cache.query.IndicesQueryCache$CachingWeightWrapper.scorer(IndicesQueryCache.java:263)\r\n       org.apache.lucene.search.BooleanWeight.scorer(BooleanWeight.java:389)\r\n       org.apache.lucene.search.Weight.bulkScorer(Weight.java:135)\r\n       org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:370)\r\n       org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.bulkScorer(LRUQueryCache.java:655)\r\n       org.elasticsearch.indices.cache.query.IndicesQueryCache$CachingWeightWrapper.bulkScorer(IndicesQueryCache.java:269)\r\n       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:818)\r\n       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:535)\r\n       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:384)\r\n       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:113)\r\n       org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:372)\r\n       org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:385)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:368)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:365)\r\n       org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)\r\n       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)\r\n       org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:378)\r\n       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\r\n       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n       java.lang.Thread.run(Thread.java:745)\r\n     7/10 snapshots sharing following 26 elements\r\n       org.apache.lucene.search.MultiTermQueryConstantScoreWrapper$1.scorer(MultiTermQueryConstantScoreWrapper.java:213)\r\n       org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorer(LRUQueryCache.java:628)\r\n       org.elasticsearch.indices.cache.query.IndicesQueryCache$CachingWeightWrapper.scorer(IndicesQueryCache.java:263)\r\n       org.apache.lucene.search.BooleanWeight.scorer(BooleanWeight.java:389)\r\n       org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorer(LRUQueryCache.java:628)\r\n       org.elasticsearch.indices.cache.query.IndicesQueryCache$CachingWeightWrapper.scorer(IndicesQueryCache.java:263)\r\n       org.apache.lucene.search.BooleanWeight.scorer(BooleanWeight.java:389)\r\n       org.apache.lucene.search.Weight.bulkScorer(Weight.java:135)\r\n       org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:370)\r\n       org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.bulkScorer(LRUQueryCache.java:655)\r\n       org.elasticsearch.indices.cache.query.IndicesQueryCache$CachingWeightWrapper.bulkScorer(IndicesQueryCache.java:269)\r\n       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:818)\r\n       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:535)\r\n       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:384)\r\n       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:113)\r\n       org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:372)\r\n       org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:385)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:368)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:365)\r\n       org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)\r\n       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)\r\n       org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:378)\r\n       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\r\n       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n       java.lang.Thread.run(Thread.java:745)\r\n     unique snapshot\r\n       org.apache.lucene.codecs.blocktree.SegmentTermsEnum.seekExact(SegmentTermsEnum.java:542)\r\n       org.apache.lucene.index.TermContext.build(TermContext.java:94)\r\n       org.apache.lucene.search.TermQuery.createWeight(TermQuery.java:192)\r\n       org.apache.lucene.search.IndexSearcher.createWeight(IndexSearcher.java:904)\r\n       org.apache.lucene.search.BooleanWeight.<init>(BooleanWeight.java:57)\r\n       org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:239)\r\n       org.apache.lucene.search.IndexSearcher.createWeight(IndexSearcher.java:904)\r\n       org.apache.lucene.search.ConstantScoreQuery.createWeight(ConstantScoreQuery.java:119)\r\n       org.apache.lucene.search.IndexSearcher.createWeight(IndexSearcher.java:904)\r\n       org.apache.lucene.search.BooleanWeight.<init>(BooleanWeight.java:57)\r\n       org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:239)\r\n       org.apache.lucene.search.IndexSearcher.createWeight(IndexSearcher.java:904)\r\n       org.apache.lucene.search.IndexSearcher.createNormalizedWeight(IndexSearcher.java:887)\r\n       org.elasticsearch.search.internal.ContextIndexSearcher.createNormalizedWeight(ContextIndexSearcher.java:100)\r\n       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:535)\r\n       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:384)\r\n       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:113)\r\n       org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:372)\r\n       org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:385)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:368)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:365)\r\n       org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)\r\n       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)\r\n       org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:378)\r\n       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\r\n       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n       java.lang.Thread.run(Thread.java:745)\r\n   \r\n   34.1% (170.5ms out of 500ms) cpu usage by thread 'elasticsearch[uj_node_master][search][T#2]'\r\n     3/10 snapshots sharing following 14 elements\r\n       java.util.HashMap.hash(HashMap.java:338)\r\n       java.util.HashMap.get(HashMap.java:556)\r\n       org.elasticsearch.search.fetch.FetchPhase.createSearchHit(FetchPhase.java:247)\r\n       org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:178)\r\n       org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:605)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$FetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:408)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$FetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:405)\r\n       org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)\r\n       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)\r\n       org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:378)\r\n       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\r\n       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n       java.lang.Thread.run(Thread.java:745)\r\n     3/10 snapshots sharing following 26 elements\r\n       org.apache.lucene.search.MultiTermQueryConstantScoreWrapper$1.scorer(MultiTermQueryConstantScoreWrapper.java:213)\r\n       org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorer(LRUQueryCache.java:628)\r\n       org.elasticsearch.indices.cache.query.IndicesQueryCache$CachingWeightWrapper.scorer(IndicesQueryCache.java:263)\r\n       org.apache.lucene.search.BooleanWeight.scorer(BooleanWeight.java:389)\r\n       org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorer(LRUQueryCache.java:628)\r\n       org.elasticsearch.indices.cache.query.IndicesQueryCache$CachingWeightWrapper.scorer(IndicesQueryCache.java:263)\r\n       org.apache.lucene.search.BooleanWeight.scorer(BooleanWeight.java:389)\r\n       org.apache.lucene.search.Weight.bulkScorer(Weight.java:135)\r\n       org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:370)\r\n       org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.bulkScorer(LRUQueryCache.java:655)\r\n       org.elasticsearch.indices.cache.query.IndicesQueryCache$CachingWeightWrapper.bulkScorer(IndicesQueryCache.java:269)\r\n       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:818)\r\n       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:535)\r\n       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:384)\r\n       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:113)\r\n       org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:372)\r\n       org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:385)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:368)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:365)\r\n       org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)\r\n       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)\r\n       org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:378)\r\n       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\r\n       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n       java.lang.Thread.run(Thread.java:745)\r\n     4/10 snapshots sharing following 22 elements\r\n       org.apache.lucene.util.RoaringDocIdSet$ShortArrayDocIdSet$1.advance(RoaringDocIdSet.java:215)\r\n       org.apache.lucene.util.RoaringDocIdSet$Iterator.advance(RoaringDocIdSet.java:303)\r\n       org.apache.lucene.search.ConjunctionDISI.doNext(ConjunctionDISI.java:178)\r\n       org.apache.lucene.search.ConjunctionDISI.nextDoc(ConjunctionDISI.java:207)\r\n       org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:219)\r\n       org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:172)\r\n       org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)\r\n       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:821)\r\n       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:535)\r\n       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:384)\r\n       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:113)\r\n       org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:372)\r\n       org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:385)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:368)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:365)\r\n       org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)\r\n       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)\r\n       org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:378)\r\n       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\r\n       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n       java.lang.Thread.run(Thread.java:745)\r\n\r\n\r\n3:\r\nHot threads at 2019-06-04T21:57:46.200Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:\r\n   \r\n   47.4% (237ms out of 500ms) cpu usage by thread 'elasticsearch[uj_node_master][search][T#7]'\r\n     3/10 snapshots sharing following 30 elements\r\n       org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:219)\r\n       org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:172)\r\n       org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)\r\n       org.apache.lucene.search.LRUQueryCache.cacheImpl(LRUQueryCache.java:422)\r\n       org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.cache(LRUQueryCache.java:608)\r\n       org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorer(LRUQueryCache.java:625)\r\n       org.elasticsearch.indices.cache.query.IndicesQueryCache$CachingWeightWrapper.scorer(IndicesQueryCache.java:263)\r\n       org.apache.lucene.search.BooleanWeight.scorer(BooleanWeight.java:389)\r\n       org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorer(LRUQueryCache.java:628)\r\n       org.elasticsearch.indices.cache.query.IndicesQueryCache$CachingWeightWrapper.scorer(IndicesQueryCache.java:263)\r\n       org.apache.lucene.search.BooleanWeight.scorer(BooleanWeight.java:389)\r\n       org.apache.lucene.search.Weight.bulkScorer(Weight.java:135)\r\n       org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:370)\r\n       org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.bulkScorer(LRUQueryCache.java:655)\r\n       org.elasticsearch.indices.cache.query.IndicesQueryCache$CachingWeightWrapper.bulkScorer(IndicesQueryCache.java:269)\r\n       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:818)\r\n       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:535)\r\n       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:384)\r\n       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:113)\r\n       org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:372)\r\n       org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:385)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:368)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:365)\r\n       org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)\r\n       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)\r\n       org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:378)\r\n       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\r\n       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n       java.lang.Thread.run(Thread.java:745)\r\n     7/10 snapshots sharing following 14 elements\r\n       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:535)\r\n       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:384)\r\n       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:113)\r\n       org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:372)\r\n       org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:385)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:368)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:365)\r\n       org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)\r\n       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)\r\n       org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:378)\r\n       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\r\n       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n       java.lang.Thread.run(Thread.java:745)\r\n   \r\n   31.4% (156.7ms out of 500ms) cpu usage by thread 'elasticsearch[uj_node_master][search][T#6]'\r\n     4/10 snapshots sharing following 22 elements\r\n       org.apache.lucene.util.RoaringDocIdSet$ShortArrayDocIdSet$1.advance(RoaringDocIdSet.java:215)\r\n       org.apache.lucene.util.RoaringDocIdSet$Iterator.advance(RoaringDocIdSet.java:303)\r\n       org.apache.lucene.search.ConjunctionDISI.doNext(ConjunctionDISI.java:174)\r\n       org.apache.lucene.search.ConjunctionDISI.nextDoc(ConjunctionDISI.java:207)\r\n       org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:227)\r\n       org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:172)\r\n       org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)\r\n       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:821)\r\n       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:535)\r\n       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:384)\r\n       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:113)\r\n       org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:372)\r\n       org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:385)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:368)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:365)\r\n       org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)\r\n       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)\r\n       org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:378)\r\n       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\r\n       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n       java.lang.Thread.run(Thread.java:745)\r\n     2/10 snapshots sharing following 44 elements\r\n       sun.nio.ch.NativeThread.current(Native Method)\r\n       sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:46)\r\n       sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:737)\r\n       sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:727)\r\n       org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:179)\r\n       org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:342)\r\n       org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)\r\n       org.apache.lucene.store.DataInput.readVInt(DataInput.java:125)\r\n       org.apache.lucene.store.BufferedIndexInput.readVInt(BufferedIndexInput.java:221)\r\n       org.apache.lucene.codecs.blocktree.SegmentTermsEnumFrame.loadBlock(SegmentTermsEnumFrame.java:157)\r\n       org.apache.lucene.codecs.blocktree.SegmentTermsEnum.seekCeil(SegmentTermsEnum.java:754)\r\n       org.apache.lucene.index.FilteredTermsEnum.next(FilteredTermsEnum.java:216)\r\n       org.apache.lucene.search.MultiTermQueryConstantScoreWrapper$1.collectTerms(MultiTermQueryConstantScoreWrapper.java:121)\r\n       org.apache.lucene.search.MultiTermQueryConstantScoreWrapper$1.rewrite(MultiTermQueryConstantScoreWrapper.java:152)\r\n       org.apache.lucene.search.MultiTermQueryConstantScoreWrapper$1.bulkScorer(MultiTermQueryConstantScoreWrapper.java:199)\r\n       org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.cache(LRUQueryCache.java:604)\r\n       org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorer(LRUQueryCache.java:625)\r\n       org.elasticsearch.indices.cache.query.IndicesQueryCache$CachingWeightWrapper.scorer(IndicesQueryCache.java:263)\r\n       org.apache.lucene.search.BooleanWeight.scorer(BooleanWeight.java:389)\r\n       org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorer(LRUQueryCache.java:628)\r\n       org.elasticsearch.indices.cache.query.IndicesQueryCache$CachingWeightWrapper.scorer(IndicesQueryCache.java:263)\r\n       org.apache.lucene.search.BooleanWeight.scorer(BooleanWeight.java:389)\r\n       org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorer(LRUQueryCache.java:628)\r\n       org.elasticsearch.indices.cache.query.IndicesQueryCache$CachingWeightWrapper.scorer(IndicesQueryCache.java:263)\r\n       org.apache.lucene.search.BooleanWeight.scorer(BooleanWeight.java:389)\r\n       org.apache.lucene.search.Weight.bulkScorer(Weight.java:135)\r\n       org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:370)\r\n       org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.bulkScorer(LRUQueryCache.java:655)\r\n       org.elasticsearch.indices.cache.query.IndicesQueryCache$CachingWeightWrapper.bulkScorer(IndicesQueryCache.java:269)\r\n       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:818)\r\n       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:535)\r\n       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:384)\r\n       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:113)\r\n       org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:372)\r\n       org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:385)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:368)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:365)\r\n       org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)\r\n       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)\r\n       org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:378)\r\n       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\r\n       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n       java.lang.Thread.run(Thread.java:745)\r\n     2/10 snapshots sharing following 32 elements\r\n       sun.nio.ch.FileDispatcherImpl.pread0(Native Method)\r\n       sun.nio.ch.FileDispatcherImpl.pread(FileDispatcherImpl.java:52)\r\n       sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:220)\r\n       sun.nio.ch.IOUtil.read(IOUtil.java:197)\r\n       sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:741)\r\n       sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:727)\r\n       org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:179)\r\n       org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:342)\r\n       org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)\r\n       org.apache.lucene.codecs.lucene50.ForUtil.readBlock(ForUtil.java:187)\r\n       org.apache.lucene.codecs.lucene50.Lucene50PostingsReader$BlockDocsEnum.refillDocs(Lucene50PostingsReader.java:368)\r\n       org.apache.lucene.codecs.lucene50.Lucene50PostingsReader$BlockDocsEnum.advance(Lucene50PostingsReader.java:452)\r\n       org.apache.lucene.search.ConjunctionDISI.doNext(ConjunctionDISI.java:174)\r\n       org.apache.lucene.search.ConjunctionDISI.nextDoc(ConjunctionDISI.java:207)\r\n       org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:219)\r\n       org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:172)\r\n       org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)\r\n       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:821)\r\n       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:535)\r\n       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:384)\r\n       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:113)\r\n       org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:372)\r\n       org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:385)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:368)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:365)\r\n       org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)\r\n       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)\r\n       org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:378)\r\n       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\r\n       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n       java.lang.Thread.run(Thread.java:745)\r\n     2/10 snapshots sharing following 30 elements\r\n       org.apache.lucene.util.BitSet.or(BitSet.java:94)\r\n       org.apache.lucene.util.FixedBitSet.or(FixedBitSet.java:271)\r\n       org.apache.lucene.util.DocIdSetBuilder.add(DocIdSetBuilder.java:87)\r\n       org.apache.lucene.search.MultiTermQueryConstantScoreWrapper$1.rewrite(MultiTermQueryConstantScoreWrapper.java:173)\r\n       org.apache.lucene.search.MultiTermQueryConstantScoreWrapper$1.scorer(MultiTermQueryConstantScoreWrapper.java:213)\r\n       org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorer(LRUQueryCache.java:628)\r\n       org.elasticsearch.indices.cache.query.IndicesQueryCache$CachingWeightWrapper.scorer(IndicesQueryCache.java:263)\r\n       org.apache.lucene.search.BooleanWeight.scorer(BooleanWeight.java:389)\r\n       org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorer(LRUQueryCache.java:628)\r\n       org.elasticsearch.indices.cache.query.IndicesQueryCache$CachingWeightWrapper.scorer(IndicesQueryCache.java:263)\r\n       org.apache.lucene.search.BooleanWeight.scorer(BooleanWeight.java:389)\r\n       org.apache.lucene.search.Weight.bulkScorer(Weight.java:135)\r\n       org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:370)\r\n       org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.bulkScorer(LRUQueryCache.java:655)\r\n       org.elasticsearch.indices.cache.query.IndicesQueryCache$CachingWeightWrapper.bulkScorer(IndicesQueryCache.java:269)\r\n       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:818)\r\n       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:535)\r\n       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:384)\r\n       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:113)\r\n       org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:372)\r\n       org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:385)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:368)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:365)\r\n       org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)\r\n       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)\r\n       org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:378)\r\n       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\r\n       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n       java.lang.Thread.run(Thread.java:745)\r\n   \r\n   27.0% (134.7ms out of 500ms) cpu usage by thread 'elasticsearch[uj_node_master][search][T#1]'\r\n     2/10 snapshots sharing following 33 elements\r\n       org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:219)\r\n       org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:172)\r\n       org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)\r\n       org.apache.lucene.search.LRUQueryCache.cacheImpl(LRUQueryCache.java:422)\r\n       org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.cache(LRUQueryCache.java:608)\r\n       org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorer(LRUQueryCache.java:625)\r\n       org.elasticsearch.indices.cache.query.IndicesQueryCache$CachingWeightWrapper.scorer(IndicesQueryCache.java:263)\r\n       org.apache.lucene.search.BooleanWeight.scorer(BooleanWeight.java:389)\r\n       org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorer(LRUQueryCache.java:628)\r\n       org.elasticsearch.indices.cache.query.IndicesQueryCache$CachingWeightWrapper.scorer(IndicesQueryCache.java:263)\r\n       org.apache.lucene.search.BooleanWeight.scorer(BooleanWeight.java:389)\r\n       org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorer(LRUQueryCache.java:628)\r\n       org.elasticsearch.indices.cache.query.IndicesQueryCache$CachingWeightWrapper.scorer(IndicesQueryCache.java:263)\r\n       org.apache.lucene.search.BooleanWeight.scorer(BooleanWeight.java:389)\r\n       org.apache.lucene.search.Weight.bulkScorer(Weight.java:135)\r\n       org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:370)\r\n       org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.bulkScorer(LRUQueryCache.java:655)\r\n       org.elasticsearch.indices.cache.query.IndicesQueryCache$CachingWeightWrapper.bulkScorer(IndicesQueryCache.java:269)\r\n       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:818)\r\n       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:535)\r\n       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:384)\r\n       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:113)\r\n       org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:372)\r\n       org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:385)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:368)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:365)\r\n       org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)\r\n       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)\r\n       org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:378)\r\n       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\r\n       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n       java.lang.Thread.run(Thread.java:745)\r\n     2/10 snapshots sharing following 22 elements\r\n       org.apache.lucene.util.RoaringDocIdSet$ShortArrayDocIdSet$1.advance(RoaringDocIdSet.java:215)\r\n       org.apache.lucene.util.RoaringDocIdSet$Iterator.advance(RoaringDocIdSet.java:303)\r\n       org.apache.lucene.search.ConjunctionDISI.doNext(ConjunctionDISI.java:174)\r\n       org.apache.lucene.search.ConjunctionDISI.nextDoc(ConjunctionDISI.java:207)\r\n       org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:227)\r\n       org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:172)\r\n       org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)\r\n       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:821)\r\n       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:535)\r\n       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:384)\r\n       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:113)\r\n       org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:372)\r\n       org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:385)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:368)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:365)\r\n       org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)\r\n       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)\r\n       org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:378)\r\n       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\r\n       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n       java.lang.Thread.run(Thread.java:745)\r\n     2/10 snapshots sharing following 22 elements\r\n       org.apache.lucene.util.RoaringDocIdSet$ShortArrayDocIdSet$1.advance(RoaringDocIdSet.java:215)\r\n       org.apache.lucene.util.RoaringDocIdSet$Iterator.advance(RoaringDocIdSet.java:303)\r\n       org.apache.lucene.search.ConjunctionDISI.doNext(ConjunctionDISI.java:178)\r\n       org.apache.lucene.search.ConjunctionDISI.nextDoc(ConjunctionDISI.java:207)\r\n       org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:219)\r\n       org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:172)\r\n       org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)\r\n       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:821)\r\n       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:535)\r\n       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:384)\r\n       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:113)\r\n       org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:372)\r\n       org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:385)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:368)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:365)\r\n       org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)\r\n       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)\r\n       org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:378)\r\n       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\r\n       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n       java.lang.Thread.run(Thread.java:745)\r\n     2/10 snapshots sharing following 22 elements\r\n       org.apache.lucene.util.RoaringDocIdSet$ShortArrayDocIdSet$1.advance(RoaringDocIdSet.java:215)\r\n       org.apache.lucene.util.RoaringDocIdSet$Iterator.advance(RoaringDocIdSet.java:303)\r\n       org.apache.lucene.search.ConjunctionDISI.doNext(ConjunctionDISI.java:174)\r\n       org.apache.lucene.search.ConjunctionDISI.nextDoc(ConjunctionDISI.java:207)\r\n       org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:219)\r\n       org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:172)\r\n       org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)\r\n       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:821)\r\n       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:535)\r\n       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:384)\r\n       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:113)\r\n       org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:372)\r\n       org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:385)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:368)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:365)\r\n       org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)\r\n       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)\r\n       org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:378)\r\n       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\r\n       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n       java.lang.Thread.run(Thread.java:745)\r\n     2/10 snapshots sharing following 15 elements\r\n       org.elasticsearch.search.sort.SortParseElement.addCompoundSortField(SortParseElement.java:154)\r\n       org.elasticsearch.search.sort.SortParseElement.parse(SortParseElement.java:95)\r\n       org.elasticsearch.search.SearchService.parseSource(SearchService.java:856)\r\n       org.elasticsearch.search.SearchService.createContext(SearchService.java:667)\r\n       org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:633)\r\n       org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:377)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:368)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:365)\r\n       org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)\r\n       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)\r\n       org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:378)\r\n       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\r\n       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n       java.lang.Thread.run(Thread.java:745)"},"comment":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/comments/499252894","html_url":"https://github.com/elastic/elasticsearch/issues/42899#issuecomment-499252894","issue_url":"https://api.github.com/repos/elastic/elasticsearch/issues/42899","id":499252894,"node_id":"MDEyOklzc3VlQ29tbWVudDQ5OTI1Mjg5NA==","user":{"login":"jasontedor","id":4744941,"node_id":"MDQ6VXNlcjQ3NDQ5NDE=","avatar_url":"https://avatars3.githubusercontent.com/u/4744941?v=4","gravatar_id":"","url":"https://api.github.com/users/jasontedor","html_url":"https://github.com/jasontedor","followers_url":"https://api.github.com/users/jasontedor/followers","following_url":"https://api.github.com/users/jasontedor/following{/other_user}","gists_url":"https://api.github.com/users/jasontedor/gists{/gist_id}","starred_url":"https://api.github.com/users/jasontedor/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jasontedor/subscriptions","organizations_url":"https://api.github.com/users/jasontedor/orgs","repos_url":"https://api.github.com/users/jasontedor/repos","events_url":"https://api.github.com/users/jasontedor/events{/privacy}","received_events_url":"https://api.github.com/users/jasontedor/received_events","type":"User","site_admin":false},"created_at":"2019-06-05T20:58:20Z","updated_at":"2019-06-05T20:58:20Z","author_association":"MEMBER","body":"Thanks very much for your interest in Elasticsearch.\r\n\r\nThis appears to be a user question, and we'd like to direct these kinds of things to the [forums](https://discuss.elastic.co). If you can stop by there, we'd appreciate it. This allows us to use GitHub for verified bug reports, feature requests, and pull requests.\r\n\r\nThere's an active community in the [forums](https://discuss.elastic.co) that should be able to help get an answer to your question. As such, I hope you don't mind that I close this."}}
{"actor":{"display_login":"jasontedor"}}