{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/19576","repository_url":"https://api.github.com/repos/elastic/elasticsearch","labels_url":"https://api.github.com/repos/elastic/elasticsearch/issues/19576/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/issues/19576/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch/issues/19576/events","html_url":"https://github.com/elastic/elasticsearch/issues/19576","id":167365740,"number":19576,"title":"OOM -> LockObtainFailedException -> All data lost","user":{"login":"DaveChapman","id":5517266,"avatar_url":"https://avatars.githubusercontent.com/u/5517266?v=3","gravatar_id":"","url":"https://api.github.com/users/DaveChapman","html_url":"https://github.com/DaveChapman","followers_url":"https://api.github.com/users/DaveChapman/followers","following_url":"https://api.github.com/users/DaveChapman/following{/other_user}","gists_url":"https://api.github.com/users/DaveChapman/gists{/gist_id}","starred_url":"https://api.github.com/users/DaveChapman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/DaveChapman/subscriptions","organizations_url":"https://api.github.com/users/DaveChapman/orgs","repos_url":"https://api.github.com/users/DaveChapman/repos","events_url":"https://api.github.com/users/DaveChapman/events{/privacy}","received_events_url":"https://api.github.com/users/DaveChapman/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/feedback_needed","name":"feedback_needed","color":"d4c5f9"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-07-25T13:27:44Z","updated_at":"2016-07-25T16:47:52Z","closed_at":null,"body":"Hi,\r\n\r\nWe experienced an issue today where we saw OOM errors, LockObtainFailedException exceptions and some others (shown in the traces below). We restarted the node (this is a cluster of a single node hosting a number of indices with 5 shards in each index). Following the restart ALL the data was gone from all indices and a check of the file system showed less than 1MB in the data directories of Elastic (previously this would have been Gigs of data).\r\n\r\nI've used the title above as it looks very similar to this issue: \r\nhttps://github.com/elastic/elasticsearch/issues/12041\r\nbut I think its similar to this also.....\r\nhttp://stackoverflow.com/questions/32494095/all-the-data-suddenly-removed-from-the-elasticsearch-cluster\r\n\r\nPlease find my responses to your standard questions below.\r\n\r\nThanks in advance for any help!\r\n\r\n**Elasticsearch version**: 2.3.2\r\n\r\n**JVM version**:\r\njava version \"1.8.0_91\"\r\nJava(TM) SE Runtime Environment (build 1.8.0_91-b14)\r\nJava HotSpot(TM) 64-Bit Server VM (build 25.91-b14, mixed mode)\r\n\r\n**OS version**: CentOS Linux release 7.2.1511 (Core)\r\n\r\n**Description of the problem including expected versus actual behavior**:\r\nAfter restart due to out of memory all index data directories are missing, the index directories are still present, but contain only _state, no data.\r\n\r\n**Provide logs (if relevant)**:\r\n[WARN ][indices.breaker.request  ] [request] New used memory 2179536984 [2gb] for data of [<reused_arrays>] would be larger than configured breaker: 2102132736 [1.9gb], breaking\r\n\r\n[DEBUG][action.search            ] [<host>] [<index>][4], node[9wlPuDpBSB6oGedoRiI7-Q], [P], v[4], s[STARTED], a[id=wtkeXgBiRv-o9DVnrbOGnw]: Failed to execute [org.elasticsearch.action.search.SearchRequest@1eb229c6] lastShard [true]\r\nRemoteTransportException[[<host>][x.x.x.x:9300][indices:data/read/search[phase/query]]]; nested: QueryPhaseExecutionException[Query Failed [Failed to execute main query]]; nested: CircuitBreakingException[[request] Data too large, data for [<reused_arrays>] would be larger than limit of [2102132736/1.9gb]];\r\nCaused by: QueryPhaseExecutionException[Query Failed [Failed to execute main query]]; nested: CircuitBreakingException[[request] Data too large, data for [<reused_arrays>] would be larger than limit of [2102132736/1.9gb]];\r\n\tat org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:409)\r\n\tat org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:113)\r\n\tat org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:366)\r\n\tat org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:378)\r\n\tat org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:368)\r\n\tat org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:365)\r\n\tat org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)\r\n\tat org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:75)\r\n\tat org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:376)\r\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n\tat java.lang.Thread.run(Thread.java:745)\r\nCaused by: CircuitBreakingException[[request] Data too large, data for [<reused_arrays>] would be larger than limit of [2102132736/1.9gb]]\r\n\tat org.elasticsearch.common.breaker.ChildMemoryCircuitBreaker.circuitBreak(ChildMemoryCircuitBreaker.java:97)\r\n\tat org.elasticsearch.common.breaker.ChildMemoryCircuitBreaker.addEstimateBytesAndMaybeBreak(ChildMemoryCircuitBreaker.java:147)\r\n\tat org.elasticsearch.common.util.BigArrays.adjustBreaker(BigArrays.java:396)\r\n\tat org.elasticsearch.common.util.BigArrays.validate(BigArrays.java:433)\r\n\tat org.elasticsearch.common.util.BigArrays.newByteArray(BigArrays.java:458)\r\n\tat org.elasticsearch.common.util.BigArrays.resize(BigArrays.java:475)\r\n\tat org.elasticsearch.common.util.BigArrays.grow(BigArrays.java:489)\r\n\tat org.elasticsearch.search.aggregations.metrics.cardinality.HyperLogLogPlusPlus.ensureCapacity(HyperLogLogPlusPlus.java:197)\r\n\tat org.elasticsearch.search.aggregations.metrics.cardinality.HyperLogLogPlusPlus.collect(HyperLogLogPlusPlus.java:230)\r\n\tat org.elasticsearch.search.aggregations.metrics.cardinality.CardinalityAggregator$DirectCollector.collect(CardinalityAggregator.java:203)\r\n\tat org.elasticsearch.search.aggregations.LeafBucketCollector$3.collect(LeafBucketCollector.java:73)\r\n\tat org.elasticsearch.search.aggregations.bucket.BucketsAggregator.collectExistingBucket(BucketsAggregator.java:80)\r\n\tat org.elasticsearch.search.aggregations.bucket.BucketsAggregator.collectBucket(BucketsAggregator.java:72)\r\n\tat org.elasticsearch.search.aggregations.bucket.range.RangeAggregator$1.collect(RangeAggregator.java:184)\r\n\tat org.elasticsearch.search.aggregations.bucket.range.RangeAggregator$1.collect(RangeAggregator.java:138)\r\n\tat org.elasticsearch.search.aggregations.bucket.BucketsAggregator.collectExistingBucket(BucketsAggregator.java:80)\r\n\tat org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$2.collect(GlobalOrdinalsStringTermsAggregator.java:130)\r\n\tat org.elasticsearch.search.aggregations.LeafBucketCollector.collect(LeafBucketCollector.java:88)\r\n\tat org.apache.lucene.search.MultiCollector$MultiLeafCollector.collect(MultiCollector.java:174)\r\n\tat org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:221)\r\n\tat org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:172)\r\n\tat org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)\r\n\tat org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:821)\r\n\tat org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:535)\r\n\tat org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:384)\r\n\t... 12 more\r\n\r\n[DEBUG][action.search            ] [<host>] [<index>][0], node[9wlPuDpBSB6oGedoRiI7-Q], [P], v[4], s[STARTED], a[id=jsJyGbCqQW677Mcg3OK1fg]: Failed to execute [org.elasticsearch.action.search.SearchRequest@1eb229c6] lastShard [true]\r\nRemoteTransportException[[<host>][x.x.x.x:9300][indices:data/read/search[phase/query]]]; nested: QueryPhaseExecutionException[Query Failed [Failed to execute main query]]; nested: OutOfMemoryError[Java heap space];\r\nCaused by: QueryPhaseExecutionException[Query Failed [Failed to execute main query]]; nested: OutOfMemoryError[Java heap space];\r\n\tat org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:409)\r\n\tat org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:113)\r\n\tat org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:366)\r\n\tat org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:378)\r\n\tat org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:368)\r\n\tat org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:365)\r\n\tat org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)\r\n\tat org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:75)\r\n\tat org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:376)\r\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n\tat java.lang.Thread.run(Thread.java:745)\r\nCaused by: java.lang.OutOfMemoryError: Java heap space\r\n\tat org.elasticsearch.cache.recycler.PageCacheRecycler$1.newInstance(PageCacheRecycler.java:102)\r\n\tat org.elasticsearch.cache.recycler.PageCacheRecycler$1.newInstance(PageCacheRecycler.java:99)\r\n\tat org.elasticsearch.common.recycler.DequeRecycler.obtain(DequeRecycler.java:53)\r\n\tat org.elasticsearch.common.recycler.AbstractRecycler.obtain(AbstractRecycler.java:33)\r\n\tat org.elasticsearch.common.recycler.DequeRecycler.obtain(DequeRecycler.java:28)\r\n\tat org.elasticsearch.common.recycler.FilterRecycler.obtain(FilterRecycler.java:39)\r\n\tat org.elasticsearch.common.recycler.Recyclers$3.obtain(Recyclers.java:119)\r\n\tat org.elasticsearch.common.recycler.FilterRecycler.obtain(FilterRecycler.java:39)\r\n\tat org.elasticsearch.cache.recycler.PageCacheRecycler.bytePage(PageCacheRecycler.java:150)\r\n\tat org.elasticsearch.common.util.AbstractBigArray.newBytePage(AbstractBigArray.java:108)\r\n\tat org.elasticsearch.common.util.BigByteArray.<init>(BigByteArray.java:45)\r\n\tat org.elasticsearch.common.util.BigArrays.newByteArray(BigArrays.java:451)\r\n\tat org.elasticsearch.common.util.BigArrays.resize(BigArrays.java:475)\r\n\tat org.elasticsearch.common.util.BigArrays.grow(BigArrays.java:489)\r\n\tat org.elasticsearch.search.aggregations.metrics.cardinality.HyperLogLogPlusPlus.ensureCapacity(HyperLogLogPlusPlus.java:197)\r\n\tat org.elasticsearch.search.aggregations.metrics.cardinality.HyperLogLogPlusPlus.collect(HyperLogLogPlusPlus.java:230)\r\n\tat org.elasticsearch.search.aggregations.metrics.cardinality.CardinalityAggregator$DirectCollector.collect(CardinalityAggregator.java:203)\r\n\tat org.elasticsearch.search.aggregations.LeafBucketCollector$3.collect(LeafBucketCollector.java:73)\r\n\tat org.elasticsearch.search.aggregations.bucket.BucketsAggregator.collectExistingBucket(BucketsAggregator.java:80)\r\n\tat org.elasticsearch.search.aggregations.bucket.BucketsAggregator.collectBucket(BucketsAggregator.java:72)\r\n\tat org.elasticsearch.search.aggregations.bucket.range.RangeAggregator$1.collect(RangeAggregator.java:184)\r\n\tat org.elasticsearch.search.aggregations.bucket.range.RangeAggregator$1.collect(RangeAggregator.java:138)\r\n\tat org.elasticsearch.search.aggregations.bucket.BucketsAggregator.collectExistingBucket(BucketsAggregator.java:80)\r\n\tat org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$2.collect(GlobalOrdinalsStringTermsAggregator.java:130)\r\n\tat org.elasticsearch.search.aggregations.LeafBucketCollector.collect(LeafBucketCollector.java:88)\r\n\tat org.apache.lucene.search.MultiCollector$MultiLeafCollector.collect(MultiCollector.java:174)\r\n\tat org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:221)\r\n\tat org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:172)\r\n\tat org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)\r\n\tat org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:821)\r\n\tat org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:535)\r\n\tat org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:384)\r\n\r\n[WARN ][index.engine             ] [<host>] [<index>][0] failed engine [refresh failed]\r\njava.lang.OutOfMemoryError: Java heap space\r\n\r\n[DEBUG][action.bulk              ] [<host>] failed to execute [BulkShardRequest to [<index>] containing [1] requests] on [[<index>][4]]\r\n[<index>][[<index>][4]] EngineClosedException[CurrentState[CLOSED] Closed]\r\n\tat org.elasticsearch.index.engine.Engine.ensureOpen(Engine.java:329)\r\n\tat org.elasticsearch.index.engine.InternalEngine.get(InternalEngine.java:317)\r\n\tat org.elasticsearch.index.shard.IndexShard.get(IndexShard.java:652)\r\n\tat org.elasticsearch.index.get.ShardGetService.innerGet(ShardGetService.java:173)\r\n\tat org.elasticsearch.index.get.ShardGetService.get(ShardGetService.java:86)\r\n\tat org.elasticsearch.action.update.UpdateHelper.prepare(UpdateHelper.java:77)\r\n\tat org.elasticsearch.action.bulk.TransportShardBulkAction.shardUpdateOperation(TransportShardBulkAction.java:383)\r\n\tat org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:191)\r\n\tat org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:68)\r\n\tat org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryPhase.doRun(TransportReplicationAction.java:639)\r\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\r\n\tat org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:279)\r\n\tat org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:271)\r\n\tat org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:75)\r\n\tat org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:376)\r\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n\tat java.lang.Thread.run(Thread.java:745)\r\n\r\n[WARN ][index.shard              ] [<host>] [<index>][4] Failed to perform scheduled engine refresh\r\n[<index>][[<index>][4]] RefreshFailedEngineException[Refresh failed]; nested: OutOfMemoryError[Java heap space];\r\n\tat org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:680)\r\n\tat org.elasticsearch.index.shard.IndexShard.refresh(IndexShard.java:661)\r\n\tat org.elasticsearch.index.shard.IndexShard$EngineRefresher$1.run(IndexShard.java:1349)\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n\tat java.lang.Thread.run(Thread.java:745)\r\nCaused by: java.lang.OutOfMemoryError: Java heap space\r\n\r\n[WARN ][indices.cluster          ] [<host>] [[<index>][2]] marking and sending shard failed due to [engine failure, reason [refresh failed]]\r\njava.lang.OutOfMemoryError: Java heap space\r\n\r\n[WARN ][cluster.action.shard     ] [<host>] [<index>][2] received shard failed for target shard [[<index>][2], node[9wlPuDpBSB6oGedoRiI7-Q], [P], v[4], s[STARTED], a[id=1SRVPoVZTlaB3Qdt2lyCSQ]], indexUUID [qmKpqmQ2RYKaIzlqxXya6w], message [engine failure, reason [refresh failed]], failure [OutOfMemoryError[Java heap space]]\r\njava.lang.OutOfMemoryError: Java heap space\r\n\r\n[WARN ][indices.memory           ] [<host>] failed to set shard [<index>][0] index buffer to [62.6mb]\r\norg.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed\r\n\tat org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:720)\r\n\tat org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:734)\r\n\tat org.apache.lucene.index.IndexWriter.ramBytesUsed(IndexWriter.java:475)\r\n\tat org.elasticsearch.index.engine.InternalEngine.indexWriterRAMBytesUsed(InternalEngine.java:948)\r\n\tat org.elasticsearch.index.shard.IndexShard.updateBufferSize(IndexShard.java:1149)\r\n\tat org.elasticsearch.indices.memory.IndexingMemoryController.updateShardBuffers(IndexingMemoryController.java:232)\r\n\tat org.elasticsearch.indices.memory.IndexingMemoryController$ShardsIndicesStatusChecker.run(IndexingMemoryController.java:286)\r\n\tat org.elasticsearch.threadpool.ThreadPool$LoggingRunnable.run(ThreadPool.java:640)\r\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\r\n\tat java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)\r\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)\r\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n\tat java.lang.Thread.run(Thread.java:745)\r\nCaused by: java.lang.OutOfMemoryError: Java heap space\r\n\r\n[WARN ][indices.cluster          ] [<host>] [[<index>][4]] marking and sending shard failed due to [failed to create shard]\r\n[<index>][[<index>][4]] ElasticsearchException[failed to create shard]; nested: LockObtainFailedException[Can't lock shard [<index>][4], timed out after 5000ms];\r\n\tat org.elasticsearch.index.IndexService.createShard(IndexService.java:389)\r\n\tat org.elasticsearch.indices.cluster.IndicesClusterStateService.applyInitializingShard(IndicesClusterStateService.java:601)\r\n\tat org.elasticsearch.indices.cluster.IndicesClusterStateService.applyNewOrUpdatedShards(IndicesClusterStateService.java:501)\r\n\tat org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java:166)\r\n\tat org.elasticsearch.cluster.service.InternalClusterService.runTasksForExecutor(InternalClusterService.java:610)\r\n\tat org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:772)\r\n\tat org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231)\r\n\tat org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194)\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n\tat java.lang.Thread.run(Thread.java:745)\r\nCaused by: org.apache.lucene.store.LockObtainFailedException: Can't lock shard [lowercasekey-poi-geoshape-1466686817402][4], timed out after 5000ms\r\n\tat org.elasticsearch.env.NodeEnvironment$InternalShardLock.acquire(NodeEnvironment.java:609)\r\n\tat org.elasticsearch.env.NodeEnvironment.shardLock(NodeEnvironment.java:537)\r\n\tat org.elasticsearch.index.IndexService.createShard(IndexService.java:306)\r\n\t... 10 more\r\n\r\n[WARN ][cluster.action.shard     ] [<host>] [<index>][4] received shard failed for target shard [[<index>][4], node[9wlPuDpBSB6oGedoRiI7-Q], [P], v[5], s[INITIALIZING], a[id=oOVK7CO-TsSLK7aWor8q5w], unassigned_info[[reason=ALLOCATION_FAILED], at[2016-07-25T09:26:03.243Z], details[engine failure, reason [refresh failed], failure OutOfMemoryError[Java heap space]]]], indexUUID [qmKpqmQ2RYKaIzlqxXya6w], message [failed to create shard], failure [ElasticsearchException[failed to create shard]; nested: LockObtainFailedException[Can't lock shard [<index>][4], timed out after 5000ms]; ]\r\n[<index>][[<index>][4]] ElasticsearchException[failed to create shard]; nested: LockObtainFailedException[Can't lock shard [<index>][4], timed out after 5000ms];\r\n\tat org.elasticsearch.index.IndexService.createShard(IndexService.java:389)\r\n\tat org.elasticsearch.indices.cluster.IndicesClusterStateService.applyInitializingShard(IndicesClusterStateService.java:601)\r\n\tat org.elasticsearch.indices.cluster.IndicesClusterStateService.applyNewOrUpdatedShards(IndicesClusterStateService.java:501)\r\n\tat org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java:166)\r\n\tat org.elasticsearch.cluster.service.InternalClusterService.runTasksForExecutor(InternalClusterService.java:610)\r\n\tat org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:772)\r\n\tat org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231)\r\n\tat org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194)\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n\tat java.lang.Thread.run(Thread.java:745)\r\nCaused by: org.apache.lucene.store.LockObtainFailedException: Can't lock shard [<index>][4], timed out after 5000ms\r\n\tat org.elasticsearch.env.NodeEnvironment$InternalShardLock.acquire(NodeEnvironment.java:609)\r\n\tat org.elasticsearch.env.NodeEnvironment.shardLock(NodeEnvironment.java:537)\r\n\tat org.elasticsearch.index.IndexService.createShard(IndexService.java:306)\r\n\t... 10 more\r\n\r\n[WARN ][netty.channel.DefaultChannelPipeline] An exception was thrown by an exception handler.\r\njava.util.concurrent.RejectedExecutionException: Worker has already been shutdown\r\n\tat org.jboss.netty.channel.socket.nio.AbstractNioSelector.registerTask(AbstractNioSelector.java:120)\r\n\tat org.jboss.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:72)\r\n\tat org.jboss.netty.channel.socket.nio.NioWorker.executeInIoThread(NioWorker.java:36)\r\n\tat org.jboss.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:56)\r\n\tat org.jboss.netty.channel.socket.nio.NioWorker.executeInIoThread(NioWorker.java:36)\r\n\tat org.jboss.netty.channel.socket.nio.AbstractNioChannelSink.execute(AbstractNioChannelSink.java:34)\r\n\tat org.jboss.netty.channel.DefaultChannelPipeline.execute(DefaultChannelPipeline.java:636)\r\n\tat org.jboss.netty.channel.Channels.fireExceptionCaughtLater(Channels.java:496)\r\n\tat org.jboss.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:46)\r\n\tat org.jboss.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:658)\r\n\tat org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:781)\r\n\tat org.jboss.netty.channel.Channels.write(Channels.java:725)\r\n\tat org.jboss.netty.handler.codec.oneone.OneToOneEncoder.doEncode(OneToOneEncoder.java:71)\r\n\tat org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59)\r\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)\r\n\tat org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784)\r\n\tat org.elasticsearch.http.netty.pipelining.HttpPipeliningHandler.handleDownstream(HttpPipeliningHandler.java:87)\r\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)\r\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)\r\n\tat org.elasticsearch.http.netty.NettyHttpChannel.sendResponse(NettyHttpChannel.java:146)\r\n\tat org.elasticsearch.rest.action.support.RestResponseListener.processResponse(RestResponseListener.java:43)\r\n\tat org.elasticsearch.rest.action.support.RestActionListener.onResponse(RestActionListener.java:49)\r\n\tat org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:89)\r\n\tat org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:85)\r\n\tat org.elasticsearch.action.bulk.TransportBulkAction$2.finishHim(TransportBulkAction.java:356)\r\n\tat org.elasticsearch.action.bulk.TransportBulkAction$2.onFailure(TransportBulkAction.java:351)\r\n\tat org.elasticsearch.action.support.TransportAction$1.onFailure(TransportAction.java:95)\r\n\tat org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.finishAsFailed(TransportReplicationAction.java:567)\r\n\tat org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$2.onClusterServiceClose(TransportReplicationAction.java:552)\r\n\tat org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onClose(ClusterStateObserver.java:222)\r\n\tat org.elasticsearch.cluster.service.InternalClusterService.add(InternalClusterService.java:282)\r\n\tat org.elasticsearch.cluster.ClusterStateObserver.waitForNextChange(ClusterStateObserver.java:153)\r\n\tat org.elasticsearch.cluster.ClusterStateObserver.waitForNextChange(ClusterStateObserver.java:98)\r\n\tat org.elasticsearch.cluster.ClusterStateObserver.waitForNextChange(ClusterStateObserver.java:90)\r\n\tat org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.retry(TransportReplicationAction.java:544)\r\n\tat org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.retryBecauseUnavailable(TransportReplicationAction.java:596)\r\n\tat org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.doRun(TransportReplicationAction.java:465)\r\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\r\n\tat org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$2.onNewClusterState(TransportReplicationAction.java:547)\r\n\tat org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.postAdded(ClusterStateObserver.java:206)\r\n\tat org.elasticsearch.cluster.service.InternalClusterService$1.run(InternalClusterService.java:296)\r\n\tat org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231)\r\n\tat org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194)\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n\tat java.lang.Thread.run(Thread.java:745)\r\n\r\n[WARN ][rest.suppressed          ] /<index>/_search Params: {index=<index>}\r\nFailed to execute phase [query], all shards failed; shardFailures {[9wlPuDpBSB6oGedoRiI7-Q][<index>][0]: TransportException[transport stopped, action: indices:data/read/search[phase/query]]}{[9wlPuDpBSB6oGedoRiI7-Q][<index>][1]: TransportException[transport stopped, action: indices:data/read/search[phase/query]]}{[9wlPuDpBSB6oGedoRiI7-Q][<index>][2]: TransportException[transport stopped, action: indices:data/read/search[phase/query]]}{[9wlPuDpBSB6oGedoRiI7-Q][<index>][3]: TransportException[transport stopped, action: indices:data/read/search[phase/query]]}{[9wlPuDpBSB6oGedoRiI7-Q][<index>][4]: TransportException[transport stopped, action: indices:data/read/search[phase/query]]}\r\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.onFirstPhaseResult(AbstractSearchAsyncAction.java:206)\r\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:152)\r\n\tat org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:46)\r\n\tat org.elasticsearch.transport.TransportService$2.run(TransportService.java:206)\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n\tat java.lang.Thread.run(Thread.java:745)\r\nCaused by: TransportException[transport stopped, action: indices:data/read/search[phase/query]]\r\n\t... 4 more\r\n\r\n[INFO ][node                     ] [<host>] stopped\r\n[INFO ][node                     ] [<host>] closing ...\r\n[INFO ][node                     ] [<host>] version[2.3.2], pid[26270], build[b9e4a6a/2016-04-21T16:03:47Z]\r\n[INFO ][node                     ] [<host>] initializing ...\r\n\r\n[INFO ][node                     ] [<host>] initialised\r\n[INFO ][node                     ] [<host>] starting ...\r\n[INFO ][transport                ] [<host>] publish_address {10.10.20.10:9300}, bound_addresses {[::]:9300}\r\n[INFO ][discovery                ] [<host>] dev-elastic/RSMtDui2TvS4pzY6vvdJng\r\n[INFO ][node                     ] [<host>] closed\r\n[INFO ][cluster.service          ] [<host>] new_master {<host>}{RSMtDui2TvS4pzY6vvdJng}{x.x.x.x}{x.x.x.x:9300}{master=true}, reason: zen-disco-join(elected_as_master, [0] joins received)\r\n[INFO ][http                     ] [<host>] publish_address {x.x.x.x:9200}, bound_addresses {[::]:9200}\r\n[INFO ][node                     ] [<host>] started\r\n[INFO ][gateway                  ] [<host>] recovered [0] indices into cluster_state"},"comment":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/comments/235011509","html_url":"https://github.com/elastic/elasticsearch/issues/19576#issuecomment-235011509","issue_url":"https://api.github.com/repos/elastic/elasticsearch/issues/19576","id":235011509,"user":{"login":"DaveChapman","id":5517266,"avatar_url":"https://avatars.githubusercontent.com/u/5517266?v=3","gravatar_id":"","url":"https://api.github.com/users/DaveChapman","html_url":"https://github.com/DaveChapman","followers_url":"https://api.github.com/users/DaveChapman/followers","following_url":"https://api.github.com/users/DaveChapman/following{/other_user}","gists_url":"https://api.github.com/users/DaveChapman/gists{/gist_id}","starred_url":"https://api.github.com/users/DaveChapman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/DaveChapman/subscriptions","organizations_url":"https://api.github.com/users/DaveChapman/orgs","repos_url":"https://api.github.com/users/DaveChapman/repos","events_url":"https://api.github.com/users/DaveChapman/events{/privacy}","received_events_url":"https://api.github.com/users/DaveChapman/received_events","type":"User","site_admin":false},"created_at":"2016-07-25T16:47:52Z","updated_at":"2016-07-25T16:47:52Z","body":"Hi, an update to this: the data seems to still be on disk (I had forgotten that we were writing to a number of directories).\r\n\r\nAnyway our configuration is as follows:\r\n\r\n`path.data: /data0/data,/data1/data,/data2/data`\r\n\r\nAnd our data directories are sized as:\r\n\r\n`924K\t/data0/data`\r\n`72G\t/data1/data`\r\n`90G\t/data2/data`\r\n\r\nHowever, data is not being returned from our searches before the time we restarted Elastic this morning (new items are visible however). The cluster state is yellow."}}
{"actor":{"display_login":"DaveChapman"}}