{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/30919","repository_url":"https://api.github.com/repos/elastic/elasticsearch","labels_url":"https://api.github.com/repos/elastic/elasticsearch/issues/30919/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/issues/30919/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch/issues/30919/events","html_url":"https://github.com/elastic/elasticsearch/issues/30919","id":327183634,"number":30919,"title":"Elasticsearch some nodes report such errors when full restart cluster.","user":{"login":"johnyannj","id":8110450,"avatar_url":"https://avatars3.githubusercontent.com/u/8110450?v=4","gravatar_id":"","url":"https://api.github.com/users/johnyannj","html_url":"https://github.com/johnyannj","followers_url":"https://api.github.com/users/johnyannj/followers","following_url":"https://api.github.com/users/johnyannj/following{/other_user}","gists_url":"https://api.github.com/users/johnyannj/gists{/gist_id}","starred_url":"https://api.github.com/users/johnyannj/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/johnyannj/subscriptions","organizations_url":"https://api.github.com/users/johnyannj/orgs","repos_url":"https://api.github.com/users/johnyannj/repos","events_url":"https://api.github.com/users/johnyannj/events{/privacy}","received_events_url":"https://api.github.com/users/johnyannj/received_events","type":"User","site_admin":false},"labels":[{"id":111624690,"url":"https://api.github.com/repos/elastic/elasticsearch/labels/feedback_needed","name":"feedback_needed","color":"d4c5f9","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-05-29T06:21:32Z","updated_at":"2018-05-29T10:36:54Z","closed_at":null,"author_association":"NONE","body":"\r\nversion: 6.2.2\r\nOS: CentOS\r\njdk: 1.8\r\nnodes: 6\r\n\r\nWe are testing the full restart of the elasticsearch cluster。\r\nwe found some nodes has follow errors in logs when it starts recovery.\r\n\r\nThe error log is as follows：\r\n\r\nDoes it mean data loss or normal logic?\r\n\r\n```\r\n[2018-05-28T17:12:50,350][WARN ][o.e.g.GatewayAllocator$InternalReplicaShardAllocator] [es_node1] [app20142][23]: failed to list shard for shard_store on node [Duy8P4B7SpKpJPWux9FaaA]\r\norg.elasticsearch.action.FailedNodeException: Failed node [Duy8P4B7SpKpJPWux9FaaA]\r\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:239) ~[elasticsearch-6.2.2.jar:6.2.2]\r\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$200(TransportNodesAction.java:153) ~[elasticsearch-6.2.2.jar:6.2.2]\r\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:211) ~[elasticsearch-6.2.2.jar:6.2.2]\r\n\tat org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1098) ~[elasticsearch-6.2.2.jar:6.2.2]\r\n\tat org.elasticsearch.transport.TcpTransport.lambda$handleException$33(TcpTransport.java:1478) ~[elasticsearch-6.2.2.jar:6.2.2]\r\n\tat org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:135) [elasticsearch-6.2.2.jar:6.2.2]\r\n\tat org.elasticsearch.transport.TcpTransport.handleException(TcpTransport.java:1476) [elasticsearch-6.2.2.jar:6.2.2]\r\n\tat org.elasticsearch.transport.TcpTransport.handlerResponseError(TcpTransport.java:1468) [elasticsearch-6.2.2.jar:6.2.2]\r\n\tat org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1398) [elasticsearch-6.2.2.jar:6.2.2]\r\n\tat org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:64) [transport-netty4-6.2.2.jar:6.2.2]\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]\r\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]\r\n\tat io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) [netty-codec-4.1.16.Final.jar:4.1.16.Final]\r\n\tat io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297) [netty-codec-4.1.16.Final.jar:4.1.16.Final]\r\n\tat io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413) [netty-codec-4.1.16.Final.jar:4.1.16.Final]\r\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) [netty-codec-4.1.16.Final.jar:4.1.16.Final]\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]\r\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]\r\n\tat io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241) [netty-handler-4.1.16.Final.jar:4.1.16.Final]\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]\r\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]\r\n\tat io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final]\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]\r\n\tat io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final]\r\n\tat io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final]\r\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]\r\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final]\r\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final]\r\n\tat io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]\r\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]\r\n\tat java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]\r\nCaused by: org.elasticsearch.transport.RemoteTransportException: [es_node6][192.168.64.58:9300][internal:cluster/nodes/indices/shard/store[n]]\r\nCaused by: org.elasticsearch.ElasticsearchException: Failed to list store metadata for shard [[app20142][23]]\r\n\tat org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[elasticsearch-6.2.2.jar:6.2.2]\r\n\tat org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:61) ~[elasticsearch-6.2.2.jar:6.2.2]\r\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:140) ~[elasticsearch-6.2.2.jar:6.2.2]\r\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:262) ~[elasticsearch-6.2.2.jar:6.2.2]\r\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:258) ~[elasticsearch-6.2.2.jar:6.2.2]\r\n\tat org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) ~[elasticsearch-6.2.2.jar:6.2.2]\r\n\tat org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1555) ~[elasticsearch-6.2.2.jar:6.2.2]\r\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) ~[elasticsearch-6.2.2.jar:6.2.2]\r\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.2.2.jar:6.2.2]\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_171]\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_171]\r\n\t... 1 more\r\nCaused by: java.io.FileNotFoundException: no segments* file found in store(MMapDirectory@/disk/data1/data/nodes/0/indices/XqfshaweRp-5GAXbK7C_qA/23/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@39cb082a): files: [recovery.l2K98nCnS_CG69RJlGHtZA._10d7.dii, recovery.l2K98nCnS_CG69RJlGHtZA._10d7.dim, recovery.l2K98nCnS_CG69RJlGHtZA._10d7.fdt, recovery.l2K98nCnS_CG69RJlGHtZA._10d7.fdx, recovery.l2K98nCnS_CG69RJlGHtZA._10d7.fnm, recovery.l2K98nCnS_CG69RJlGHtZA._10d7.si, recovery.l2K98nCnS_CG69RJlGHtZA._10d7_Lucene50_0.doc, recovery.l2K98nCnS_CG69RJlGHtZA._10d7_Lucene50_0.tip, recovery.l2K98nCnS_CG69RJlGHtZA._10d7_Lucene70_0.dvm, recovery.l2K98nCnS_CG69RJlGHtZA._11gk.dii, recovery.l2K98nCnS_CG69RJlGHtZA._11gk.dim, recovery.l2K98nCnS_CG69RJlGHtZA._11gk.fdt, recovery.l2K98nCnS_CG69RJlGHtZA._11gk.fdx, recovery.l2K98nCnS_CG69RJlGHtZA._11gk.fnm, recovery.l2K98nCnS_CG69RJlGHtZA._11gk.si, recovery.l2K98nCnS_CG69RJlGHtZA._11gk_Lucene50_0.doc, recovery.l2K98nCnS_CG69RJlGHtZA._11gk_Lucene50_0.tip, recovery.l2K98nCnS_CG69RJlGHtZA._11gk_Lucene70_0.dvm, recovery.l2K98nCnS_CG69RJlGHtZA._12as.dii, recovery.l2K98nCnS_CG69RJlGHtZA._12as.dim, recovery.l2K98nCnS_CG69RJlGHtZA._12as.fdt, recovery.l2K98nCnS_CG69RJlGHtZA._12as.fdx, recovery.l2K98nCnS_CG69RJlGHtZA._12as.fnm, recovery.l2K98nCnS_CG69RJlGHtZA._12as.si, recovery.l2K98nCnS_CG69RJlGHtZA._12as_Lucene50_0.doc, recovery.l2K98nCnS_CG69RJlGHtZA._12as_Lucene50_0.tip, recovery.l2K98nCnS_CG69RJlGHtZA._12as_Lucene70_0.dvm, recovery.l2K98nCnS_CG69RJlGHtZA._13cw.dii, recovery.l2K98nCnS_CG69RJlGHtZA._13cw.dim, recovery.l2K98nCnS_CG69RJlGHtZA._13cw.fdt, recovery.l2K98nCnS_CG69RJlGHtZA._13cw.fdx, recovery.l2K98nCnS_CG69RJlGHtZA._13cw.fnm, recovery.l2K98nCnS_CG69RJlGHtZA._13cw.si, recovery.l2K98nCnS_CG69RJlGHtZA._13cw_Lucene50_0.doc, recovery.l2K98nCnS_CG69RJlGHtZA._13cw_Lucene50_0.tip, recovery.l2K98nCnS_CG69RJlGHtZA._13cw_Lucene70_0.dvm, recovery.l2K98nCnS_CG69RJlGHtZA._149n.cfe, recovery.l2K98nCnS_CG69RJlGHtZA._149n.si, recovery.l2K98nCnS_CG69RJlGHtZA._15f1.cfe, recovery.l2K98nCnS_CG69RJlGHtZA._15f1.si, recovery.l2K98nCnS_CG69RJlGHtZA._16co.cfe, recovery.l2K98nCnS_CG69RJlGHtZA._16co.si, recovery.l2K98nCnS_CG69RJlGHtZA._17lo.cfe, recovery.l2K98nCnS_CG69RJlGHtZA._17lo.cfs, recovery.l2K98nCnS_CG69RJlGHtZA._17lo.si, recovery.l2K98nCnS_CG69RJlGHtZA._17oi.cfe, recovery.l2K98nCnS_CG69RJlGHtZA._17oi.si, recovery.l2K98nCnS_CG69RJlGHtZA._17r2.cfe, recovery.l2K98nCnS_CG69RJlGHtZA._17r2.cfs, recovery.l2K98nCnS_CG69RJlGHtZA._17r2.si, recovery.l2K98nCnS_CG69RJlGHtZA._17wd.cfe, recovery.l2K98nCnS_CG69RJlGHtZA._17wd.si, recovery.l2K98nCnS_CG69RJlGHtZA._17yv.cfe, recovery.l2K98nCnS_CG69RJlGHtZA._17yv.si, recovery.l2K98nCnS_CG69RJlGHtZA._1833.cfe, recovery.l2K98nCnS_CG69RJlGHtZA._1833.cfs, recovery.l2K98nCnS_CG69RJlGHtZA._1833.si, recovery.l2K98nCnS_CG69RJlGHtZA._185w.cfe, recovery.l2K98nCnS_CG69RJlGHtZA._185w.si, recovery.l2K98nCnS_CG69RJlGHtZA._18b8.cfe, recovery.l2K98nCnS_CG69RJlGHtZA._18b8.si, recovery.l2K98nCnS_CG69RJlGHtZA._18h0.cfe, recovery.l2K98nCnS_CG69RJlGHtZA._18h0.si, recovery.l2K98nCnS_CG69RJlGHtZA._18ja.cfe, recovery.l2K98nCnS_CG69RJlGHtZA._18ja.si, recovery.l2K98nCnS_CG69RJlGHtZA._18jz.cfe, recovery.l2K98nCnS_CG69RJlGHtZA._18jz.cfs, recovery.l2K98nCnS_CG69RJlGHtZA._18jz.si, recovery.l2K98nCnS_CG69RJlGHtZA._18ku.cfe, recovery.l2K98nCnS_CG69RJlGHtZA._18ku.cfs, recovery.l2K98nCnS_CG69RJlGHtZA._18ku.si, recovery.l2K98nCnS_CG69RJlGHtZA._18mo.cfe, recovery.l2K98nCnS_CG69RJlGHtZA._18mo.cfs, recovery.l2K98nCnS_CG69RJlGHtZA._18mo.si, recovery.l2K98nCnS_CG69RJlGHtZA._18op.cfe, recovery.l2K98nCnS_CG69RJlGHtZA._18op.si, recovery.l2K98nCnS_CG69RJlGHtZA._18pr.cfe, recovery.l2K98nCnS_CG69RJlGHtZA._18pr.cfs, recovery.l2K98nCnS_CG69RJlGHtZA._18pr.si, recovery.l2K98nCnS_CG69RJlGHtZA._18ql.cfe, recovery.l2K98nCnS_CG69RJlGHtZA._18ql.cfs, recovery.l2K98nCnS_CG69RJlGHtZA._18ql.si, recovery.l2K98nCnS_CG69RJlGHtZA._18rg.cfe, recovery.l2K98nCnS_CG69RJlGHtZA._18rg.cfs, recovery.l2K98nCnS_CG69RJlGHtZA._18rg.si, recovery.l2K98nCnS_CG69RJlGHtZA._18ri.cfe, recovery.l2K98nCnS_CG69RJlGHtZA._18ri.cfs, recovery.l2K98nCnS_CG69RJlGHtZA._18ri.si, recovery.l2K98nCnS_CG69RJlGHtZA._18rz.cfe, recovery.l2K98nCnS_CG69RJlGHtZA._18rz.cfs, recovery.l2K98nCnS_CG69RJlGHtZA._18rz.si, recovery.l2K98nCnS_CG69RJlGHtZA._18s1.cfe, recovery.l2K98nCnS_CG69RJlGHtZA._18s1.cfs, recovery.l2K98nCnS_CG69RJlGHtZA._18s1.si, recovery.l2K98nCnS_CG69RJlGHtZA._18si.cfe, recovery.l2K98nCnS_CG69RJlGHtZA._18si.cfs, recovery.l2K98nCnS_CG69RJlGHtZA._18si.si, recovery.l2K98nCnS_CG69RJlGHtZA._18sj.cfe, recovery.l2K98nCnS_CG69RJlGHtZA._18sj.cfs, recovery.l2K98nCnS_CG69RJlGHtZA._18sj.si, recovery.l2K98nCnS_CG69RJlGHtZA._18sk.cfe, recovery.l2K98nCnS_CG69RJlGHtZA._18sk.cfs, recovery.l2K98nCnS_CG69RJlGHtZA._18sk.si, recovery.l2K98nCnS_CG69RJlGHtZA._18sm.cfe, recovery.l2K98nCnS_CG69RJlGHtZA._18sm.cfs, recovery.l2K98nCnS_CG69RJlGHtZA._18sm.si, recovery.l2K98nCnS_CG69RJlGHtZA._18sn.cfe, recovery.l2K98nCnS_CG69RJlGHtZA._18sn.cfs, recovery.l2K98nCnS_CG69RJlGHtZA._18sn.si, recovery.l2K98nCnS_CG69RJlGHtZA._18sp.cfe, recovery.l2K98nCnS_CG69RJlGHtZA._18sp.cfs, recovery.l2K98nCnS_CG69RJlGHtZA._18sp.si, recovery.l2K98nCnS_CG69RJlGHtZA._18sq.cfe, recovery.l2K98nCnS_CG69RJlGHtZA._18sq.cfs, recovery.l2K98nCnS_CG69RJlGHtZA._18sq.si, recovery.l2K98nCnS_CG69RJlGHtZA._18ss.cfe, recovery.l2K98nCnS_CG69RJlGHtZA._18ss.cfs, recovery.l2K98nCnS_CG69RJlGHtZA._18ss.si, recovery.l2K98nCnS_CG69RJlGHtZA._18st.cfe, recovery.l2K98nCnS_CG69RJlGHtZA._18st.cfs, recovery.l2K98nCnS_CG69RJlGHtZA._18st.si, recovery.l2K98nCnS_CG69RJlGHtZA._18su.cfe, recovery.l2K98nCnS_CG69RJlGHtZA._18su.si, recovery.l2K98nCnS_CG69RJlGHtZA._18t2.cfe, recovery.l2K98nCnS_CG69RJlGHtZA._18t2.cfs, recovery.l2K98nCnS_CG69RJlGHtZA._18t2.si, recovery.l2K98nCnS_CG69RJlGHtZA._18tc.cfe, recovery.l2K98nCnS_CG69RJlGHtZA._18tc.cfs, recovery.l2K98nCnS_CG69RJlGHtZA._18tc.si, recovery.l2K98nCnS_CG69RJlGHtZA._18tn.cfe, recovery.l2K98nCnS_CG69RJlGHtZA._18tn.cfs, recovery.l2K98nCnS_CG69RJlGHtZA._18tn.si, recovery.l2K98nCnS_CG69RJlGHtZA._18tt.cfe, recovery.l2K98nCnS_CG69RJlGHtZA._18tt.cfs, recovery.l2K98nCnS_CG69RJlGHtZA._18tt.si, recovery.l2K98nCnS_CG69RJlGHtZA._18tv.cfe, recovery.l2K98nCnS_CG69RJlGHtZA._18tv.cfs, recovery.l2K98nCnS_CG69RJlGHtZA._18tv.si, recovery.l2K98nCnS_CG69RJlGHtZA._18ty.cfe, recovery.l2K98nCnS_CG69RJlGHtZA._18ty.cfs, recovery.l2K98nCnS_CG69RJlGHtZA._18ty.si, recovery.l2K98nCnS_CG69RJlGHtZA._18u0.cfe, recovery.l2K98nCnS_CG69RJlGHtZA._18u0.cfs, recovery.l2K98nCnS_CG69RJlGHtZA._18u0.si, recovery.l2K98nCnS_CG69RJlGHtZA._18u1.cfe, recovery.l2K98nCnS_CG69RJlGHtZA._18u1.cfs, recovery.l2K98nCnS_CG69RJlGHtZA._18u1.si, recovery.l2K98nCnS_CG69RJlGHtZA._18u3.cfe, recovery.l2K98nCnS_CG69RJlGHtZA._18u3.cfs, recovery.l2K98nCnS_CG69RJlGHtZA._18u3.si, recovery.l2K98nCnS_CG69RJlGHtZA._18u4.cfe, recovery.l2K98nCnS_CG69RJlGHtZA._18u4.cfs, recovery.l2K98nCnS_CG69RJlGHtZA._18u4.si, recovery.l2K98nCnS_CG69RJlGHtZA._18u5.cfe, recovery.l2K98nCnS_CG69RJlGHtZA._18u5.cfs, recovery.l2K98nCnS_CG69RJlGHtZA._18u5.si, recovery.l2K98nCnS_CG69RJlGHtZA._18u6.cfe, recovery.l2K98nCnS_CG69RJlGHtZA._18u6.cfs, recovery.l2K98nCnS_CG69RJlGHtZA._18u6.si, recovery.l2K98nCnS_CG69RJlGHtZA._18u7.cfe, recovery.l2K98nCnS_CG69RJlGHtZA._18u7.cfs, recovery.l2K98nCnS_CG69RJlGHtZA._18u7.si, recovery.l2K98nCnS_CG69RJlGHtZA._18u8.cfe, recovery.l2K98nCnS_CG69RJlGHtZA._18u8.cfs, recovery.l2K98nCnS_CG69RJlGHtZA._18u8.si, recovery.l2K98nCnS_CG69RJlGHtZA._18u9.cfe, recovery.l2K98nCnS_CG69RJlGHtZA._18u9.cfs, recovery.l2K98nCnS_CG69RJlGHtZA._18u9.si, recovery.l2K98nCnS_CG69RJlGHtZA._18ua.cfe, recovery.l2K98nCnS_CG69RJlGHtZA._18ua.cfs, recovery.l2K98nCnS_CG69RJlGHtZA._18ua.si, recovery.l2K98nCnS_CG69RJlGHtZA._n8t.dii, recovery.l2K98nCnS_CG69RJlGHtZA._n8t.dim, recovery.l2K98nCnS_CG69RJlGHtZA._n8t.fdt, recovery.l2K98nCnS_CG69RJlGHtZA._n8t.fdx, recovery.l2K98nCnS_CG69RJlGHtZA._n8t.fnm, recovery.l2K98nCnS_CG69RJlGHtZA._n8t.si, recovery.l2K98nCnS_CG69RJlGHtZA._n8t_Lucene50_0.doc, recovery.l2K98nCnS_CG69RJlGHtZA._n8t_Lucene50_0.tim, recovery.l2K98nCnS_CG69RJlGHtZA._n8t_Lucene50_0.tip, recovery.l2K98nCnS_CG69RJlGHtZA._n8t_Lucene70_0.dvd, recovery.l2K98nCnS_CG69RJlGHtZA._n8t_Lucene70_0.dvm, recovery.l2K98nCnS_CG69RJlGHtZA._vo5.dii, recovery.l2K98nCnS_CG69RJlGHtZA._vo5.dim, recovery.l2K98nCnS_CG69RJlGHtZA._vo5.fdt, recovery.l2K98nCnS_CG69RJlGHtZA._vo5.fdx, recovery.l2K98nCnS_CG69RJlGHtZA._vo5.fnm, recovery.l2K98nCnS_CG69RJlGHtZA._vo5.si, recovery.l2K98nCnS_CG69RJlGHtZA._vo5_Lucene50_0.doc, recovery.l2K98nCnS_CG69RJlGHtZA._vo5_Lucene50_0.tip, recovery.l2K98nCnS_CG69RJlGHtZA._vo5_Lucene70_0.dvm, recovery.l2K98nCnS_CG69RJlGHtZA._x0t.dii, recovery.l2K98nCnS_CG69RJlGHtZA._x0t.dim, recovery.l2K98nCnS_CG69RJlGHtZA._x0t.fdt, recovery.l2K98nCnS_CG69RJlGHtZA._x0t.fdx, recovery.l2K98nCnS_CG69RJlGHtZA._x0t.fnm, recovery.l2K98nCnS_CG69RJlGHtZA._x0t.si, recovery.l2K98nCnS_CG69RJlGHtZA._x0t_Lucene50_0.doc, recovery.l2K98nCnS_CG69RJlGHtZA._x0t_Lucene50_0.tip, recovery.l2K98nCnS_CG69RJlGHtZA._x0t_Lucene70_0.dvm, recovery.l2K98nCnS_CG69RJlGHtZA._y3g.dii, recovery.l2K98nCnS_CG69RJlGHtZA._y3g.dim, recovery.l2K98nCnS_CG69RJlGHtZA._y3g.fdt, recovery.l2K98nCnS_CG69RJlGHtZA._y3g.fdx, recovery.l2K98nCnS_CG69RJlGHtZA._y3g.fnm, recovery.l2K98nCnS_CG69RJlGHtZA._y3g.si, recovery.l2K98nCnS_CG69RJlGHtZA._y3g_Lucene50_0.doc, recovery.l2K98nCnS_CG69RJlGHtZA._y3g_Lucene50_0.tim, recovery.l2K98nCnS_CG69RJlGHtZA._y3g_Lucene50_0.tip, recovery.l2K98nCnS_CG69RJlGHtZA._y3g_Lucene70_0.dvd, recovery.l2K98nCnS_CG69RJlGHtZA._y3g_Lucene70_0.dvm, recovery.l2K98nCnS_CG69RJlGHtZA._yo6.dii, recovery.l2K98nCnS_CG69RJlGHtZA._yo6.dim, recovery.l2K98nCnS_CG69RJlGHtZA._yo6.fdt, recovery.l2K98nCnS_CG69RJlGHtZA._yo6.fdx, recovery.l2K98nCnS_CG69RJlGHtZA._yo6.fnm, recovery.l2K98nCnS_CG69RJlGHtZA._yo6.si, recovery.l2K98nCnS_CG69RJlGHtZA._yo6_Lucene50_0.doc, recovery.l2K98nCnS_CG69RJlGHtZA._yo6_Lucene50_0.tip, recovery.l2K98nCnS_CG69RJlGHtZA._yo6_Lucene70_0.dvm, recovery.l2K98nCnS_CG69RJlGHtZA._zf8.dii, recovery.l2K98nCnS_CG69RJlGHtZA._zf8.dim, recovery.l2K98nCnS_CG69RJlGHtZA._zf8.fdt, recovery.l2K98nCnS_CG69RJlGHtZA._zf8.fdx, recovery.l2K98nCnS_CG69RJlGHtZA._zf8.fnm, recovery.l2K98nCnS_CG69RJlGHtZA._zf8.si, recovery.l2K98nCnS_CG69RJlGHtZA._zf8_Lucene50_0.doc, recovery.l2K98nCnS_CG69RJlGHtZA._zf8_Lucene50_0.tip, recovery.l2K98nCnS_CG69RJlGHtZA._zf8_Lucene70_0.dvm, recovery.l2K98nCnS_CG69RJlGHtZA.segments_74, write.lock]\r\n\tat org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:670) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]\r\n\tat org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:627) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]\r\n\tat org.apache.lucene.index.SegmentInfos.readLatestCommit(SegmentInfos.java:434) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]\r\n\tat org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:123) ~[elasticsearch-6.2.2.jar:6.2.2]\r\n\tat org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:202) ~[elasticsearch-6.2.2.jar:6.2.2]\r\n\tat org.elasticsearch.index.store.Store.access$200(Store.java:130) ~[elasticsearch-6.2.2.jar:6.2.2]\r\n\tat org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:859) ~[elasticsearch-6.2.2.jar:6.2.2]\r\n\tat org.elasticsearch.index.store.Store$MetadataSnapshot.<init>(Store.java:792) ~[elasticsearch-6.2.2.jar:6.2.2]\r\n\tat org.elasticsearch.index.store.Store.getMetadata(Store.java:288) ~[elasticsearch-6.2.2.jar:6.2.2]\r\n\tat org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1143) ~[elasticsearch-6.2.2.jar:6.2.2]\r\n\tat org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:125) ~[elasticsearch-6.2.2.jar:6.2.2]\r\n\tat org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:109) ~[elasticsearch-6.2.2.jar:6.2.2]\r\n\tat org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:61) ~[elasticsearch-6.2.2.jar:6.2.2]\r\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:140) ~[elasticsearch-6.2.2.jar:6.2.2]\r\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:262) ~[elasticsearch-6.2.2.jar:6.2.2]\r\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:258) ~[elasticsearch-6.2.2.jar:6.2.2]\r\n\tat org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) ~[elasticsearch-6.2.2.jar:6.2.2]\r\n\tat org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1555) ~[elasticsearch-6.2.2.jar:6.2.2]\r\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) ~[elasticsearch-6.2.2.jar:6.2.2]\r\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.2.2.jar:6.2.2]\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_171]\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_171]\r\n\t... 1 more\r\n```\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/comments/392731667","html_url":"https://github.com/elastic/elasticsearch/issues/30919#issuecomment-392731667","issue_url":"https://api.github.com/repos/elastic/elasticsearch/issues/30919","id":392731667,"user":{"login":"markharwood","id":170925,"avatar_url":"https://avatars0.githubusercontent.com/u/170925?v=4","gravatar_id":"","url":"https://api.github.com/users/markharwood","html_url":"https://github.com/markharwood","followers_url":"https://api.github.com/users/markharwood/followers","following_url":"https://api.github.com/users/markharwood/following{/other_user}","gists_url":"https://api.github.com/users/markharwood/gists{/gist_id}","starred_url":"https://api.github.com/users/markharwood/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/markharwood/subscriptions","organizations_url":"https://api.github.com/users/markharwood/orgs","repos_url":"https://api.github.com/users/markharwood/repos","events_url":"https://api.github.com/users/markharwood/events{/privacy}","received_events_url":"https://api.github.com/users/markharwood/received_events","type":"User","site_admin":false},"created_at":"2018-05-29T10:36:54Z","updated_at":"2018-05-29T10:36:54Z","author_association":"CONTRIBUTOR","body":">Does it mean data loss or normal logic?\r\n\r\nLooking at the stack trace there is a missing Lucene file which is not good. Did this index/shard have any docs in it before shutdown? Can you list the current files in the `data1/data/nodes/0/indices/XqfshaweRp-5GAXbK7C_qA/23/index` directory?"}}
{"actor":{"display_login":"markharwood"}}