{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/19879","repository_url":"https://api.github.com/repos/elastic/elasticsearch","labels_url":"https://api.github.com/repos/elastic/elasticsearch/issues/19879/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/issues/19879/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch/issues/19879/events","html_url":"https://github.com/elastic/elasticsearch/issues/19879","id":170019800,"number":19879,"title":"Memory leak when shard fails to allocate on missing synonym dictionary","user":{"login":"adrianocrestani","id":7266694,"avatar_url":"https://avatars.githubusercontent.com/u/7266694?v=3","gravatar_id":"","url":"https://api.github.com/users/adrianocrestani","html_url":"https://github.com/adrianocrestani","followers_url":"https://api.github.com/users/adrianocrestani/followers","following_url":"https://api.github.com/users/adrianocrestani/following{/other_user}","gists_url":"https://api.github.com/users/adrianocrestani/gists{/gist_id}","starred_url":"https://api.github.com/users/adrianocrestani/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/adrianocrestani/subscriptions","organizations_url":"https://api.github.com/users/adrianocrestani/orgs","repos_url":"https://api.github.com/users/adrianocrestani/repos","events_url":"https://api.github.com/users/adrianocrestani/events{/privacy}","received_events_url":"https://api.github.com/users/adrianocrestani/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2016-08-08T20:39:21Z","updated_at":"2016-08-08T21:36:48Z","closed_at":"2016-08-08T20:45:12Z","body":"**Elasticsearch version**: 2.1.1 and 2.3.5\r\n\r\n**Plugins installed**: []\r\n\r\n**JVM version**: Oracle JDK 1.8.0_60\r\n\r\n**OS version**: Ubuntu 4.8.2 (Trusty 14.04)\r\n\r\n**Description of the problem including expected versus actual behavior**:\r\n\r\nWe ran into a memory issue in our ES cluster when some shards tried to be allocated on certain nodes that were missing a synonym dictionary. The heap usage went from 15GB to 30GB and the behaviour was 100% reproducible.\r\n\r\nThose nodes did not OOM, but were spending a lot of time doing GCs. The whole cluster got very unresponsive where stats and search requests would take a long time to respond and often time out. The dictionary was missing on 4 out of 16 data nodes, only those 4 nodes started running out of memory, the other 12 nodes had the synonym dictionary in place and were able to allocate shards.\r\n\r\nCluster configuration: ES 2.1.1, 16 data nodes, 2 masters, 6 proxy nodes.\r\n\r\nWe were able to reproduce the memory leak problem with the steps below.\r\n\r\n**Steps to reproduce**:\r\n\r\n1) Install ES 2.3.5 on two different nodes\r\n\r\n2) Create a synonym dictionary using the command below on each node:\r\n```mkdir $ES_HOME/config/analysis/ && cat /usr/share/dict/words | sed -r 's/(.*)/\\1,\\1_synonym/' > $ES_HOME/config/analysis/synonym.txt```\r\n3) Start both nodes with 256MB of heap.\r\n```export ES_HEAP_SIZE=256m && $ES_HOME/bin/elasticsearch```\r\n4) Create an index using the following schema:\r\n```\r\ncurl -XPUT localhost:9200/test_index -d '{\r\n  \"settings\": {\r\n    \"similarity\": {\r\n      \"ta_bm25\": {\r\n        \"type\": \"BM25\",\r\n        \"b\": 0.1,\r\n        \"k1\": 20\r\n      }\r\n    },\r\n    \"index\": {\r\n      \"number_of_shards\": 3,\r\n      \"analysis\": {\r\n        \"filter\": {\r\n          \"my_synonym\": {\r\n            \"type\": \"synonym\",\r\n            \"synonyms_path\": \"analysis/synonym.txt\"\r\n          },\r\n          \"word_delimiter_filter\": {\r\n            \"type\": \"word_delimiter\",\r\n            \"catenate_words\": true,\r\n            \"catenate_all\": true,\r\n            \"preserve_original\": true,\r\n            \"stem_english_possessive\": false\r\n          },\r\n          \"abbreviation_filter\": {\r\n            \"type\": \"pattern_replace\",\r\n            \"pattern\": \"([a-zA-Z0-9]{1})[.]\",\r\n            \"replacement\": \"$1\"\r\n          },\r\n          \"ngram\": {\r\n            \"type\": \"edgeNGram\",\r\n            \"min_gram\": 1,\r\n            \"max_gram\": 12\r\n          },\r\n          \"stem_en\": {\r\n            \"type\": \"stemmer\",\r\n            \"name\": \"minimal_english\"\r\n          }\r\n        },\r\n        \"analyzer\": {\r\n          \"search_analyzer\": {\r\n            \"filter\": [\r\n              \"asciifolding\",\r\n              \"lowercase\",\r\n              \"abbreviation_filter\",\r\n              \"word_delimiter_filter\",\r\n              \"my_synonym\",\r\n              \"stem_en\"\r\n            ],\r\n            \"tokenizer\": \"whitespace\"\r\n          },\r\n          \"index_analyzer\": {\r\n            \"tokenizer\": \"whitespace\",\r\n            \"filter\": [\r\n              \"asciifolding\",\r\n              \"lowercase\",\r\n              \"abbreviation_filter\",\r\n              \"word_delimiter_filter\",\r\n              \"stem_en\"\r\n            ]\r\n          }\r\n        }\r\n      },\r\n      \"number_of_replicas\": 1\r\n    }\r\n  },\r\n  \"mappings\": {\r\n    \"my_type\": {\r\n      \"_source\": {\r\n        \"enabled\": false\r\n      },\r\n      \"_all\": {\r\n        \"enabled\": false\r\n      },\r\n      \"properties\": {\r\n        \"name\": {\r\n          \"type\": \"string\",\r\n          \"store\": \"no\",\r\n          \"index\": \"analyzed\",\r\n          \"analyzer\": \"index_analyzer\",\r\n          \"search_analyzer\": \"search_analyzer\",\r\n          \"similarity\": \"BM25\"\r\n        }\r\n      }\r\n    }\r\n  }\r\n}'\r\n```\r\n5) Create a test index:\r\n```for k in {1..10000} ; do for i in {1..100} ; do printf \"{\\\"index\\\": {\\\"_index\\\":\\\"test_index\\\", \\\"_type\\\":\\\"my_type\\\",\\\"_id\\\":\\\"${k}-${i}\\\"}}\\n{\\\"name\\\":\\\"$(shuf -n 5 /usr/share/dict/words | tr '\\n' ' ')\\\"}\\n\" ; done | curl -XPUT 'localhost:9200/_bulk' --data-binary @- ; done```\r\n\r\n6) Once the indexing is done, there should be one index called test_index, with one million documents, 3 shards and one replica. That means that both nodes should have 3 shards, one replica each. The command below should say that heap is around 40-50%.\r\n```curl -s 'localhost:9200/_cat/nodes?v&h=name,heap.current,heap.max,heap.percent'```\r\n\r\n7) Shutdown one of the nodes.\r\n\r\n8) Remove the dictionary file from that node and start it again.\r\n```\r\nrm -rf $ES_HOME/config/analysis/synonym.txt\r\nexport ES_HEAP_SIZE=256m && $ES_HOME/bin/elasticsearch\r\n```\r\n9) At this point, the node without the dictionary file will attempt to allocate the index shards, but will fail with the exception pasted log section below. ES attempts to do that over and over. Use the command below to keep track of the allocated heap.\r\n```curl -s 'localhost:9200/_cat/nodes?v&h=name,heap.current,heap.max,heap.percent'```\r\n\r\n10) After about 15-20 minutes, the heap will go above 95% and GC doesn't seem to help.\r\n\r\n11) To make things even worse, recreate the synonym dictionary (recreate, copy from a backed up file or from the other node).\r\n\r\n12) At this point, the node that is running out of memory will finally be able to allocate the those shards. Unfortunately, it will OOM.\r\n\r\n**Provide logs (if relevant)**:\r\n```\r\n[2016-08-08 15:15:11,172][WARN ][indices.cluster ] [Xi'an Chi Xan] [[test_index][0]] marking and sending shard failed due to [failed to create index] [16/1885]\r\n[test_index] IndexCreationException[failed to create index]; nested: IllegalArgumentException[IOException while reading synonyms_path_path: /home/acrestani/Downloads/elasticsearch-2.3.5/config/analysis\r\n/synonym.txt (No such file or directory)];\r\nat org.elasticsearch.indices.IndicesService.createIndex(IndicesService.java:360)\r\nat org.elasticsearch.indices.cluster.IndicesClusterStateService.applyNewIndices(IndicesClusterStateService.java:294)\r\nat org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java:163)\r\nat org.elasticsearch.cluster.service.InternalClusterService.runTasksForExecutor(InternalClusterService.java:610)\r\nat org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:772)\r\nat org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231)\r\nat org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194)\r\nat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\nat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\nat java.lang.Thread.run(Thread.java:745)\r\nCaused by: java.lang.IllegalArgumentException: IOException while reading synonyms_path_path: /home/acrestani/Downloads/elasticsearch-2.3.5/config/analysis/synonym.txt (No such file or directory)\r\nat org.elasticsearch.index.analysis.Analysis.getReaderFromFile(Analysis.java:288)\r\nat org.elasticsearch.index.analysis.SynonymTokenFilterFactory.<init>(SynonymTokenFilterFactory.java:64)\r\nat sun.reflect.GeneratedConstructorAccessor7.newInstance(Unknown Source)\r\nat sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)\r\nat java.lang.reflect.Constructor.newInstance(Constructor.java:422)\r\nat org.elasticsearch.common.inject.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:50)\r\nat org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:86)\r\nat org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:104)\r\nat org.elasticsearch.common.inject.FactoryProxy.get(FactoryProxy.java:54)\r\nat org.elasticsearch.common.inject.InjectorImpl$4$1.call(InjectorImpl.java:823)\r\nat org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:886)\r\nat org.elasticsearch.common.inject.InjectorImpl$4.get(InjectorImpl.java:818)\r\nat org.elasticsearch.common.inject.assistedinject.FactoryProvider2.invoke(FactoryProvider2.java:236)\r\nat com.sun.proxy.$Proxy15.create(Unknown Source)\r\nat org.elasticsearch.index.analysis.AnalysisService.<init>(AnalysisService.java:151)\r\nat org.elasticsearch.index.analysis.AnalysisService.<init>(AnalysisService.java:70)\r\nat sun.reflect.GeneratedConstructorAccessor5.newInstance(Unknown Source)\r\nat sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)\r\nat java.lang.reflect.Constructor.newInstance(Constructor.java:422)\r\nat org.elasticsearch.common.inject.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:50)\r\nat org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:86)\r\nat org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:104)\r\nat org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:47)\r\nat org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:886)\r\nat org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:43)\r\nat org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:59)\r\nat org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:46)\r\nat org.elasticsearch.common.inject.SingleParameterInjector.inject(SingleParameterInjector.java:42)\r\nat org.elasticsearch.common.inject.SingleParameterInjector.getAll(SingleParameterInjector.java:66)\r\nat org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:85)\r\nat org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:104)\r\nat org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:47)\r\nat org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:886)\r\nat org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:43)\r\nat org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:59)\r\nat org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:46)\r\nat org.elasticsearch.common.inject.SingleParameterInjector.inject(SingleParameterInjector.java:42)\r\nat org.elasticsearch.common.inject.SingleParameterInjector.getAll(SingleParameterInjector.java:66)\r\nat org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:85)\r\nat org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:104)\r\n```\r\n\r\nAfter deploying the synonym dictionary it prints this:\r\n\r\n```\r\njava.lang.OutOfMemoryError: Java heap space\r\nDumping heap to java_pid5205.hprof ...\r\nHeap dump file created [317848006 bytes in 1.196 secs]\r\n[2016-08-08 15:35:54,090][WARN ][threadpool ] [Xi'an Chi Xan] failed to run org.elasticsearch.discovery.zen.fd.MasterFaultDetection$MasterPinger@3eb528ce\r\njava.lang.OutOfMemoryError: Java heap space\r\nException in thread \"elasticsearch[Xi'an Chi Xan][http_server_boss][T#1]\" Exception in thread \"elasticsearch[Xi'an Chi Xan][generic][T#5]\" Exception in thread \"elasticsearch[Xi'an Chi Xan][transport_client_boss][T#1]\" java.lang.OutOfMemoryError: Java heap space\r\nat java.lang.AbstractStringBuilder.<init>(AbstractStringBuilder.java:68)\r\nat java.lang.StringBuilder.<init>(StringBuilder.java:101)\r\nat org.elasticsearch.common.logging.support.LoggerMessageFormat.format(LoggerMessageFormat.java:52)\r\nat org.elasticsearch.common.logging.support.AbstractESLogger.warn(AbstractESLogger.java:109)\r\nat org.elasticsearch.transport.netty.NettyInternalESLogger.warn(NettyInternalESLogger.java:83)\r\nat org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:340)\r\nat org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)\r\nat org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)\r\nat org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)\r\nat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\nat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\nat java.lang.Thread.run(Thread.java:745)\r\njava.lang.OutOfMemoryError: Java heap space\r\njava.lang.OutOfMemoryError: Java heap space\r\n[2016-08-08 15:35:54,095][WARN ][transport.netty ] [Xi'an Chi Xan] exception caught on transport layer [[id: 0x20cb9904, /10.33.154.216:60026 => /10.96.160.99:9300]], closing connection\r\njava.lang.IllegalStateException: Message not fully read (request) for requestId [69030], action [internal:discovery/zen/fd/ping], readerIndex [75] vs expected [181]; resetting\r\nat org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:121)\r\nat org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)\r\nat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\r\nat org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)\r\nat org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)\r\nat org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)\r\nat org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)\r\nat org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)\r\nat org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)\r\nat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\r\nat org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)\r\nat org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:75)\r\nat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\r\nat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)\r\nat org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)\r\nat org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)\r\nat org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)\r\nat org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)\r\nat org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)\r\nat org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)\r\nat org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)\r\nat org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)\r\nat org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)\r\nat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\nat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\nat java.lang.Thread.run(Thread.java:745)\r\n```"},"comment":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/comments/238385536","html_url":"https://github.com/elastic/elasticsearch/issues/19879#issuecomment-238385536","issue_url":"https://api.github.com/repos/elastic/elasticsearch/issues/19879","id":238385536,"user":{"login":"adrianocrestani","id":7266694,"avatar_url":"https://avatars.githubusercontent.com/u/7266694?v=3","gravatar_id":"","url":"https://api.github.com/users/adrianocrestani","html_url":"https://github.com/adrianocrestani","followers_url":"https://api.github.com/users/adrianocrestani/followers","following_url":"https://api.github.com/users/adrianocrestani/following{/other_user}","gists_url":"https://api.github.com/users/adrianocrestani/gists{/gist_id}","starred_url":"https://api.github.com/users/adrianocrestani/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/adrianocrestani/subscriptions","organizations_url":"https://api.github.com/users/adrianocrestani/orgs","repos_url":"https://api.github.com/users/adrianocrestani/repos","events_url":"https://api.github.com/users/adrianocrestani/events{/privacy}","received_events_url":"https://api.github.com/users/adrianocrestani/received_events","type":"User","site_admin":false},"created_at":"2016-08-08T21:36:48Z","updated_at":"2016-08-08T21:36:48Z","body":"@abeyad thanks!\r\n\r\nHere is something else to support my statement: \"Our analysis tells that your application is suffering from memory leak. It can cause OutOfMemoryError, JVM to freeze, poor response time and high CPU consumption.\"\r\n\r\nhttp://gceasy.io/my-gc-report.jsp?p=L2hvbWUvcmFtL3VwbG9hZC9pbnN0YW5jZTEvc2hhcmVkLzIwMTYtOC04L2djLmxvZy0yMS0zNS00Nw=="}}
{"actor":{"display_login":"adrianocrestani"}}