<p dir="auto"><strong>Elasticsearch version</strong>: 2.1.1 and 2.3.5</p>
<p dir="auto"><strong>Plugins installed</strong>: []</p>
<p dir="auto"><strong>JVM version</strong>: Oracle JDK 1.8.0_60</p>
<p dir="auto"><strong>OS version</strong>: Ubuntu 4.8.2 (Trusty 14.04)</p>
<p dir="auto"><strong>Description of the problem including expected versus actual behavior</strong>:</p>
<p dir="auto">We ran into a memory issue in our ES cluster when some shards tried to be allocated on certain nodes that were missing a synonym dictionary. The heap usage went from 15GB to 30GB and the behaviour was 100% reproducible.</p>
<p dir="auto">Those nodes did not OOM, but were spending a lot of time doing GCs. The whole cluster got very unresponsive where stats and search requests would take a long time to respond and often time out. The dictionary was missing on 4 out of 16 data nodes, only those 4 nodes started running out of memory, the other 12 nodes had the synonym dictionary in place and were able to allocate shards.</p>
<p dir="auto">Cluster configuration: ES 2.1.1, 16 data nodes, 2 masters, 6 proxy nodes.</p>
<p dir="auto">We were able to reproduce the memory leak problem with the steps below.</p>
<p dir="auto"><strong>Steps to reproduce</strong>:</p>
<ol dir="auto">
<li>
<p dir="auto">Install ES 2.3.5 on two different nodes</p>
</li>
<li>
<p dir="auto">Create a synonym dictionary using the command below on each node:<br>
<code class="notranslate">mkdir $ES_HOME/config/analysis/ &amp;&amp; cat /usr/share/dict/words | sed -r 's/(.*)/\1,\1_synonym/' &gt; $ES_HOME/config/analysis/synonym.txt</code></p>
</li>
<li>
<p dir="auto">Start both nodes with 256MB of heap.<br>
<code class="notranslate">export ES_HEAP_SIZE=256m &amp;&amp; $ES_HOME/bin/elasticsearch</code></p>
</li>
<li>
<p dir="auto">Create an index using the following schema:</p>
</li>
</ol>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="curl -XPUT localhost:9200/test_index -d '{
  &quot;settings&quot;: {
    &quot;similarity&quot;: {
      &quot;ta_bm25&quot;: {
        &quot;type&quot;: &quot;BM25&quot;,
        &quot;b&quot;: 0.1,
        &quot;k1&quot;: 20
      }
    },
    &quot;index&quot;: {
      &quot;number_of_shards&quot;: 3,
      &quot;analysis&quot;: {
        &quot;filter&quot;: {
          &quot;my_synonym&quot;: {
            &quot;type&quot;: &quot;synonym&quot;,
            &quot;synonyms_path&quot;: &quot;analysis/synonym.txt&quot;
          },
          &quot;word_delimiter_filter&quot;: {
            &quot;type&quot;: &quot;word_delimiter&quot;,
            &quot;catenate_words&quot;: true,
            &quot;catenate_all&quot;: true,
            &quot;preserve_original&quot;: true,
            &quot;stem_english_possessive&quot;: false
          },
          &quot;abbreviation_filter&quot;: {
            &quot;type&quot;: &quot;pattern_replace&quot;,
            &quot;pattern&quot;: &quot;([a-zA-Z0-9]{1})[.]&quot;,
            &quot;replacement&quot;: &quot;$1&quot;
          },
          &quot;ngram&quot;: {
            &quot;type&quot;: &quot;edgeNGram&quot;,
            &quot;min_gram&quot;: 1,
            &quot;max_gram&quot;: 12
          },
          &quot;stem_en&quot;: {
            &quot;type&quot;: &quot;stemmer&quot;,
            &quot;name&quot;: &quot;minimal_english&quot;
          }
        },
        &quot;analyzer&quot;: {
          &quot;search_analyzer&quot;: {
            &quot;filter&quot;: [
              &quot;asciifolding&quot;,
              &quot;lowercase&quot;,
              &quot;abbreviation_filter&quot;,
              &quot;word_delimiter_filter&quot;,
              &quot;my_synonym&quot;,
              &quot;stem_en&quot;
            ],
            &quot;tokenizer&quot;: &quot;whitespace&quot;
          },
          &quot;index_analyzer&quot;: {
            &quot;tokenizer&quot;: &quot;whitespace&quot;,
            &quot;filter&quot;: [
              &quot;asciifolding&quot;,
              &quot;lowercase&quot;,
              &quot;abbreviation_filter&quot;,
              &quot;word_delimiter_filter&quot;,
              &quot;stem_en&quot;
            ]
          }
        }
      },
      &quot;number_of_replicas&quot;: 1
    }
  },
  &quot;mappings&quot;: {
    &quot;my_type&quot;: {
      &quot;_source&quot;: {
        &quot;enabled&quot;: false
      },
      &quot;_all&quot;: {
        &quot;enabled&quot;: false
      },
      &quot;properties&quot;: {
        &quot;name&quot;: {
          &quot;type&quot;: &quot;string&quot;,
          &quot;store&quot;: &quot;no&quot;,
          &quot;index&quot;: &quot;analyzed&quot;,
          &quot;analyzer&quot;: &quot;index_analyzer&quot;,
          &quot;search_analyzer&quot;: &quot;search_analyzer&quot;,
          &quot;similarity&quot;: &quot;BM25&quot;
        }
      }
    }
  }
}'"><pre class="notranslate"><code class="notranslate">curl -XPUT localhost:9200/test_index -d '{
  "settings": {
    "similarity": {
      "ta_bm25": {
        "type": "BM25",
        "b": 0.1,
        "k1": 20
      }
    },
    "index": {
      "number_of_shards": 3,
      "analysis": {
        "filter": {
          "my_synonym": {
            "type": "synonym",
            "synonyms_path": "analysis/synonym.txt"
          },
          "word_delimiter_filter": {
            "type": "word_delimiter",
            "catenate_words": true,
            "catenate_all": true,
            "preserve_original": true,
            "stem_english_possessive": false
          },
          "abbreviation_filter": {
            "type": "pattern_replace",
            "pattern": "([a-zA-Z0-9]{1})[.]",
            "replacement": "$1"
          },
          "ngram": {
            "type": "edgeNGram",
            "min_gram": 1,
            "max_gram": 12
          },
          "stem_en": {
            "type": "stemmer",
            "name": "minimal_english"
          }
        },
        "analyzer": {
          "search_analyzer": {
            "filter": [
              "asciifolding",
              "lowercase",
              "abbreviation_filter",
              "word_delimiter_filter",
              "my_synonym",
              "stem_en"
            ],
            "tokenizer": "whitespace"
          },
          "index_analyzer": {
            "tokenizer": "whitespace",
            "filter": [
              "asciifolding",
              "lowercase",
              "abbreviation_filter",
              "word_delimiter_filter",
              "stem_en"
            ]
          }
        }
      },
      "number_of_replicas": 1
    }
  },
  "mappings": {
    "my_type": {
      "_source": {
        "enabled": false
      },
      "_all": {
        "enabled": false
      },
      "properties": {
        "name": {
          "type": "string",
          "store": "no",
          "index": "analyzed",
          "analyzer": "index_analyzer",
          "search_analyzer": "search_analyzer",
          "similarity": "BM25"
        }
      }
    }
  }
}'
</code></pre></div>
<ol start="5" dir="auto">
<li>
<p dir="auto">Create a test index:<br>
<code class="notranslate">for k in {1..10000} ; do for i in {1..100} ; do printf "{\"index\": {\"_index\":\"test_index\", \"_type\":\"my_type\",\"_id\":\"${k}-${i}\"}}\n{\"name\":\"$(shuf -n 5 /usr/share/dict/words | tr '\n' ' ')\"}\n" ; done | curl -XPUT 'localhost:9200/_bulk' --data-binary @- ; done</code></p>
</li>
<li>
<p dir="auto">Once the indexing is done, there should be one index called test_index, with one million documents, 3 shards and one replica. That means that both nodes should have 3 shards, one replica each. The command below should say that heap is around 40-50%.<br>
<code class="notranslate">curl -s 'localhost:9200/_cat/nodes?v&amp;h=name,heap.current,heap.max,heap.percent'</code></p>
</li>
<li>
<p dir="auto">Shutdown one of the nodes.</p>
</li>
<li>
<p dir="auto">Remove the dictionary file from that node and start it again.</p>
</li>
</ol>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="rm -rf $ES_HOME/config/analysis/synonym.txt
export ES_HEAP_SIZE=256m &amp;&amp; $ES_HOME/bin/elasticsearch"><pre class="notranslate"><code class="notranslate">rm -rf $ES_HOME/config/analysis/synonym.txt
export ES_HEAP_SIZE=256m &amp;&amp; $ES_HOME/bin/elasticsearch
</code></pre></div>
<ol start="9" dir="auto">
<li>
<p dir="auto">At this point, the node without the dictionary file will attempt to allocate the index shards, but will fail with the exception pasted log section below. ES attempts to do that over and over. Use the command below to keep track of the allocated heap.<br>
<code class="notranslate">curl -s 'localhost:9200/_cat/nodes?v&amp;h=name,heap.current,heap.max,heap.percent'</code></p>
</li>
<li>
<p dir="auto">After about 15-20 minutes, the heap will go above 95% and GC doesn't seem to help.</p>
</li>
<li>
<p dir="auto">To make things even worse, recreate the synonym dictionary (recreate, copy from a backed up file or from the other node).</p>
</li>
<li>
<p dir="auto">At this point, the node that is running out of memory will finally be able to allocate the those shards. Unfortunately, it will OOM.</p>
</li>
</ol>
<p dir="auto"><strong>Provide logs (if relevant)</strong>:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[2016-08-08 15:15:11,172][WARN ][indices.cluster ] [Xi'an Chi Xan] [[test_index][0]] marking and sending shard failed due to [failed to create index] [16/1885]
[test_index] IndexCreationException[failed to create index]; nested: IllegalArgumentException[IOException while reading synonyms_path_path: /home/acrestani/Downloads/elasticsearch-2.3.5/config/analysis
/synonym.txt (No such file or directory)];
at org.elasticsearch.indices.IndicesService.createIndex(IndicesService.java:360)
at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyNewIndices(IndicesClusterStateService.java:294)
at org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java:163)
at org.elasticsearch.cluster.service.InternalClusterService.runTasksForExecutor(InternalClusterService.java:610)
at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:772)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: IOException while reading synonyms_path_path: /home/acrestani/Downloads/elasticsearch-2.3.5/config/analysis/synonym.txt (No such file or directory)
at org.elasticsearch.index.analysis.Analysis.getReaderFromFile(Analysis.java:288)
at org.elasticsearch.index.analysis.SynonymTokenFilterFactory.&lt;init&gt;(SynonymTokenFilterFactory.java:64)
at sun.reflect.GeneratedConstructorAccessor7.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at org.elasticsearch.common.inject.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:50)
at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:86)
at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:104)
at org.elasticsearch.common.inject.FactoryProxy.get(FactoryProxy.java:54)
at org.elasticsearch.common.inject.InjectorImpl$4$1.call(InjectorImpl.java:823)
at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:886)
at org.elasticsearch.common.inject.InjectorImpl$4.get(InjectorImpl.java:818)
at org.elasticsearch.common.inject.assistedinject.FactoryProvider2.invoke(FactoryProvider2.java:236)
at com.sun.proxy.$Proxy15.create(Unknown Source)
at org.elasticsearch.index.analysis.AnalysisService.&lt;init&gt;(AnalysisService.java:151)
at org.elasticsearch.index.analysis.AnalysisService.&lt;init&gt;(AnalysisService.java:70)
at sun.reflect.GeneratedConstructorAccessor5.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at org.elasticsearch.common.inject.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:50)
at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:86)
at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:104)
at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:47)
at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:886)
at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:43)
at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:59)
at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:46)
at org.elasticsearch.common.inject.SingleParameterInjector.inject(SingleParameterInjector.java:42)
at org.elasticsearch.common.inject.SingleParameterInjector.getAll(SingleParameterInjector.java:66)
at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:85)
at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:104)
at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:47)
at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:886)
at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:43)
at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:59)
at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:46)
at org.elasticsearch.common.inject.SingleParameterInjector.inject(SingleParameterInjector.java:42)
at org.elasticsearch.common.inject.SingleParameterInjector.getAll(SingleParameterInjector.java:66)
at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:85)
at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:104)"><pre class="notranslate"><code class="notranslate">[2016-08-08 15:15:11,172][WARN ][indices.cluster ] [Xi'an Chi Xan] [[test_index][0]] marking and sending shard failed due to [failed to create index] [16/1885]
[test_index] IndexCreationException[failed to create index]; nested: IllegalArgumentException[IOException while reading synonyms_path_path: /home/acrestani/Downloads/elasticsearch-2.3.5/config/analysis
/synonym.txt (No such file or directory)];
at org.elasticsearch.indices.IndicesService.createIndex(IndicesService.java:360)
at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyNewIndices(IndicesClusterStateService.java:294)
at org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java:163)
at org.elasticsearch.cluster.service.InternalClusterService.runTasksForExecutor(InternalClusterService.java:610)
at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:772)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: IOException while reading synonyms_path_path: /home/acrestani/Downloads/elasticsearch-2.3.5/config/analysis/synonym.txt (No such file or directory)
at org.elasticsearch.index.analysis.Analysis.getReaderFromFile(Analysis.java:288)
at org.elasticsearch.index.analysis.SynonymTokenFilterFactory.&lt;init&gt;(SynonymTokenFilterFactory.java:64)
at sun.reflect.GeneratedConstructorAccessor7.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at org.elasticsearch.common.inject.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:50)
at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:86)
at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:104)
at org.elasticsearch.common.inject.FactoryProxy.get(FactoryProxy.java:54)
at org.elasticsearch.common.inject.InjectorImpl$4$1.call(InjectorImpl.java:823)
at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:886)
at org.elasticsearch.common.inject.InjectorImpl$4.get(InjectorImpl.java:818)
at org.elasticsearch.common.inject.assistedinject.FactoryProvider2.invoke(FactoryProvider2.java:236)
at com.sun.proxy.$Proxy15.create(Unknown Source)
at org.elasticsearch.index.analysis.AnalysisService.&lt;init&gt;(AnalysisService.java:151)
at org.elasticsearch.index.analysis.AnalysisService.&lt;init&gt;(AnalysisService.java:70)
at sun.reflect.GeneratedConstructorAccessor5.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at org.elasticsearch.common.inject.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:50)
at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:86)
at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:104)
at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:47)
at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:886)
at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:43)
at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:59)
at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:46)
at org.elasticsearch.common.inject.SingleParameterInjector.inject(SingleParameterInjector.java:42)
at org.elasticsearch.common.inject.SingleParameterInjector.getAll(SingleParameterInjector.java:66)
at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:85)
at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:104)
at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:47)
at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:886)
at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:43)
at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:59)
at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:46)
at org.elasticsearch.common.inject.SingleParameterInjector.inject(SingleParameterInjector.java:42)
at org.elasticsearch.common.inject.SingleParameterInjector.getAll(SingleParameterInjector.java:66)
at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:85)
at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:104)
</code></pre></div>
<p dir="auto">After deploying the synonym dictionary it prints this:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid5205.hprof ...
Heap dump file created [317848006 bytes in 1.196 secs]
[2016-08-08 15:35:54,090][WARN ][threadpool ] [Xi'an Chi Xan] failed to run org.elasticsearch.discovery.zen.fd.MasterFaultDetection$MasterPinger@3eb528ce
java.lang.OutOfMemoryError: Java heap space
Exception in thread &quot;elasticsearch[Xi'an Chi Xan][http_server_boss][T#1]&quot; Exception in thread &quot;elasticsearch[Xi'an Chi Xan][generic][T#5]&quot; Exception in thread &quot;elasticsearch[Xi'an Chi Xan][transport_client_boss][T#1]&quot; java.lang.OutOfMemoryError: Java heap space
at java.lang.AbstractStringBuilder.&lt;init&gt;(AbstractStringBuilder.java:68)
at java.lang.StringBuilder.&lt;init&gt;(StringBuilder.java:101)
at org.elasticsearch.common.logging.support.LoggerMessageFormat.format(LoggerMessageFormat.java:52)
at org.elasticsearch.common.logging.support.AbstractESLogger.warn(AbstractESLogger.java:109)
at org.elasticsearch.transport.netty.NettyInternalESLogger.warn(NettyInternalESLogger.java:83)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:340)
at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
[2016-08-08 15:35:54,095][WARN ][transport.netty ] [Xi'an Chi Xan] exception caught on transport layer [[id: 0x20cb9904, /10.33.154.216:60026 =&gt; /10.96.160.99:9300]], closing connection
java.lang.IllegalStateException: Message not fully read (request) for requestId [69030], action [internal:discovery/zen/fd/ping], readerIndex [75] vs expected [181]; resetting
at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:121)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:75)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)"><pre class="notranslate"><code class="notranslate">java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid5205.hprof ...
Heap dump file created [317848006 bytes in 1.196 secs]
[2016-08-08 15:35:54,090][WARN ][threadpool ] [Xi'an Chi Xan] failed to run org.elasticsearch.discovery.zen.fd.MasterFaultDetection$MasterPinger@3eb528ce
java.lang.OutOfMemoryError: Java heap space
Exception in thread "elasticsearch[Xi'an Chi Xan][http_server_boss][T#1]" Exception in thread "elasticsearch[Xi'an Chi Xan][generic][T#5]" Exception in thread "elasticsearch[Xi'an Chi Xan][transport_client_boss][T#1]" java.lang.OutOfMemoryError: Java heap space
at java.lang.AbstractStringBuilder.&lt;init&gt;(AbstractStringBuilder.java:68)
at java.lang.StringBuilder.&lt;init&gt;(StringBuilder.java:101)
at org.elasticsearch.common.logging.support.LoggerMessageFormat.format(LoggerMessageFormat.java:52)
at org.elasticsearch.common.logging.support.AbstractESLogger.warn(AbstractESLogger.java:109)
at org.elasticsearch.transport.netty.NettyInternalESLogger.warn(NettyInternalESLogger.java:83)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:340)
at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
[2016-08-08 15:35:54,095][WARN ][transport.netty ] [Xi'an Chi Xan] exception caught on transport layer [[id: 0x20cb9904, /10.33.154.216:60026 =&gt; /10.96.160.99:9300]], closing connection
java.lang.IllegalStateException: Message not fully read (request) for requestId [69030], action [internal:discovery/zen/fd/ping], readerIndex [75] vs expected [181]; resetting
at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:121)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:75)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
</code></pre></div>
<p dir="auto">Allocating shards to a node can fail for various reasons. When an allocation fails, we currently ignore the node for that shard during the next allocation round. However, this means that:</p>
<ul dir="auto">
<li>subsequent rounds consider the node for allocating the shard again.</li>
<li>other shards are still allocated to the node (in particular the balancer tries to put shards on that node with the failed shard as its weight becomes smaller).<br>
This is particularly bad if the node is permanently broken, leading to a never-ending series of failed allocations. Ultimately this affects the stability of the cluster.</li>
</ul>