{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/18279","repository_url":"https://api.github.com/repos/elastic/elasticsearch","labels_url":"https://api.github.com/repos/elastic/elasticsearch/issues/18279/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/issues/18279/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch/issues/18279/events","html_url":"https://github.com/elastic/elasticsearch/issues/18279","id":154281136,"number":18279,"title":"Hot swappable path.data disks","user":{"login":"PhaedrusTheGreek","id":4387023,"avatar_url":"https://avatars.githubusercontent.com/u/4387023?v=3","gravatar_id":"","url":"https://api.github.com/users/PhaedrusTheGreek","html_url":"https://github.com/PhaedrusTheGreek","followers_url":"https://api.github.com/users/PhaedrusTheGreek/followers","following_url":"https://api.github.com/users/PhaedrusTheGreek/following{/other_user}","gists_url":"https://api.github.com/users/PhaedrusTheGreek/gists{/gist_id}","starred_url":"https://api.github.com/users/PhaedrusTheGreek/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/PhaedrusTheGreek/subscriptions","organizations_url":"https://api.github.com/users/PhaedrusTheGreek/orgs","repos_url":"https://api.github.com/users/PhaedrusTheGreek/repos","events_url":"https://api.github.com/users/PhaedrusTheGreek/events{/privacy}","received_events_url":"https://api.github.com/users/PhaedrusTheGreek/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/:Store","name":":Store","color":"009800"},{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/enhancement","name":"enhancement","color":"4a4ea8"},{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/resiliency","name":"resiliency","color":"009800"}],"state":"open","locked":false,"assignee":{"login":"s1monw","id":973334,"avatar_url":"https://avatars.githubusercontent.com/u/973334?v=3","gravatar_id":"","url":"https://api.github.com/users/s1monw","html_url":"https://github.com/s1monw","followers_url":"https://api.github.com/users/s1monw/followers","following_url":"https://api.github.com/users/s1monw/following{/other_user}","gists_url":"https://api.github.com/users/s1monw/gists{/gist_id}","starred_url":"https://api.github.com/users/s1monw/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/s1monw/subscriptions","organizations_url":"https://api.github.com/users/s1monw/orgs","repos_url":"https://api.github.com/users/s1monw/repos","events_url":"https://api.github.com/users/s1monw/events{/privacy}","received_events_url":"https://api.github.com/users/s1monw/received_events","type":"User","site_admin":false},"milestone":null,"comments":5,"created_at":"2016-05-11T16:04:28Z","updated_at":"2016-05-13T13:12:41Z","closed_at":null,"body":"It seems that when making use of `path.data` over multiple physical disks, that when a disk is removed, the system should recover automatically.   Currently, searches and or indexing requests over missing shards throw exceptions, and no allocation/recovery occurs.   The only way to bring the data back online is to restart the node, or to reinsert the original disk with existing data.  \r\n\r\nIt would be great if Elasticsearch could:\r\n* Automatically recover when disks are removed\r\n* Automatically make use of a newly returned empty disk\r\n\r\nSteps to Test / Reproduce:\r\n\r\n1) Set up `path.data` over 2 disks, and start 2 elasticsearch nodes locally\r\n\r\n```\r\npath.data: [\"/Volumes/KINGSTON\", \"/Volumes/SDCARD\"]\r\n```\r\n\r\n2) Index some data over 5 shards.\r\n\r\n```\r\nindex    shard prirep state   docs  store ip        node\r\ntest1003 4     r      STARTED    2 10.1kb 127.0.0.1 Jacqueline Falsworth\r\ntest1003 4     p      STARTED    2 10.1kb 127.0.0.1 Vindicator\r\ntest1003 3     r      STARTED    6 24.4kb 127.0.0.1 Jacqueline Falsworth\r\ntest1003 3     p      STARTED    6 24.5kb 127.0.0.1 Vindicator\r\ntest1003 1     r      STARTED   10 40.6kb 127.0.0.1 Jacqueline Falsworth\r\ntest1003 1     p      STARTED   10 45.5kb 127.0.0.1 Vindicator\r\ntest1003 2     r      STARTED    2 10.1kb 127.0.0.1 Jacqueline Falsworth\r\ntest1003 2     p      STARTED    2 10.1kb 127.0.0.1 Vindicator\r\ntest1003 0     r      STARTED    3 10.1kb 127.0.0.1 Jacqueline Falsworth\r\ntest1003 0     p      STARTED    3 10.1kb 127.0.0.1 Vindicator\r\n```\r\n\r\n3) Remove the disk that contains most/all of the data\r\n\r\nExceptions start to show in logs\r\n\r\n```\r\n2016-05-11 11:50:18,961][DEBUG][action.admin.indices.stats] [Vindicator] [indices:monitor/stats] failed to execute operation for shard [[[test1003/01ABN7pTQDCoTa80WMdAvg]][0], node[AMr_NWrVSFCuNV-YCOfsVg], [P], s[STARTED], a[id=IMwYwgWrTLCZYa08WJRNvg]]\r\nElasticsearchException[failed to refresh store stats]; nested: NoSuchFileException[/Volumes/KINGSTON/elasticsearch/nodes/0/indices/01ABN7pTQDCoTa80WMdAvg/0/index];\r\n\tat org.elasticsearch.index.store.Store$StoreStatsCache.refresh(Store.java:1411)\r\n\tat org.elasticsearch.index.store.Store$StoreStatsCache.refresh(Store.java:1396)\r\n\tat org.elasticsearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:54)\r\n\tat org.elasticsearch.index.store.Store.stats(Store.java:321)\r\n\tat org.elasticsearch.index.shard.IndexShard.storeStats(IndexShard.java:632)\r\n\tat org.elasticsearch.action.admin.indices.stats.CommonStats.<init>(CommonStats.java:137)\r\n\tat org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:166)\r\n\tat org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:47)\r\n\tat org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.onShardOperation(TransportBroadcastByNodeAction.java:414)\r\n\tat org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:393)\r\n\tat org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:380)\r\n\tat org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)\r\n\tat org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:65)\r\n\tat org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:376)\r\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:468)\r\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n\tat java.lang.Thread.run(Thread.java:745)\r\nCaused by: java.nio.file.NoSuchFileException: /Volumes/KINGSTON/elasticsearch/nodes/0/indices/01ABN7pTQDCoTa80WMdAvg/0/index\r\n\tat sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)\r\n\tat sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)\r\n\tat sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)\r\n\tat sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:407)\r\n\tat java.nio.file.Files.newDirectoryStream(Files.java:457)\r\n\tat org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:215)\r\n\tat org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:234)\r\n\tat org.elasticsearch.index.store.FsDirectoryService$1.listAll(FsDirectoryService.java:135)\r\n\tat org.apache.lucene.store.FilterDirectory.listAll(FilterDirectory.java:57)\r\n\tat org.apache.lucene.store.FilterDirectory.listAll(FilterDirectory.java:57)\r\n\tat org.elasticsearch.index.store.Store$StoreStatsCache.estimateSize(Store.java:1417)\r\n\tat org.elasticsearch.index.store.Store$StoreStatsCache.refresh(Store.java:1409)\r\n\t... 18 more\r\n[2016-05-11 11:50:26,796][WARN ][monitor.fs               ] [Vindicator] Failed to fetch fs stats - returning empty instance\r\n```\r\n\r\nbut `_cat/shards` shows everything is OK\r\n\r\n```\r\nindex    shard prirep state   docs store ip        node\r\ntest1003 4     r      STARTED            127.0.0.1 Jacqueline Falsworth\r\ntest1003 4     p      STARTED            127.0.0.1 Vindicator\r\ntest1003 3     r      STARTED            127.0.0.1 Jacqueline Falsworth\r\ntest1003 3     p      STARTED            127.0.0.1 Vindicator\r\ntest1003 1     r      STARTED            127.0.0.1 Jacqueline Falsworth\r\ntest1003 1     p      STARTED            127.0.0.1 Vindicator\r\ntest1003 2     r      STARTED            127.0.0.1 Jacqueline Falsworth\r\ntest1003 2     p      STARTED            127.0.0.1 Vindicator\r\ntest1003 0     r      STARTED            127.0.0.1 Jacqueline Falsworth\r\ntest1003 0     p      STARTED            127.0.0.1 Vindicator\r\n```\r\n\r\n3) Post a `_refresh`\r\n\r\nNo change\r\n\r\n4) Index some data\r\n\r\n```\r\n{\r\n   \"error\": {\r\n      \"root_cause\": [\r\n         {\r\n            \"type\": \"index_failed_engine_exception\",\r\n            \"reason\": \"Index failed for [test1003#AVSghrSCuf6DFWq498vy]\",\r\n            \"index_uuid\": \"01ABN7pTQDCoTa80WMdAvg\",\r\n            \"shard\": \"1\",\r\n            \"index\": \"test1003\"\r\n         }\r\n      ],\r\n      \"type\": \"index_failed_engine_exception\",\r\n      \"reason\": \"Index failed for [test1003#AVSghrSCuf6DFWq498vy]\",\r\n      \"index_uuid\": \"01ABN7pTQDCoTa80WMdAvg\",\r\n      \"shard\": \"1\",\r\n      \"index\": \"test1003\",\r\n      \"caused_by\": {\r\n         \"type\": \"i_o_exception\",\r\n         \"reason\": \"Input/output error: NIOFSIndexInput(path=\\\"/Volumes/KINGSTON/elasticsearch/nodes/0/indices/01ABN7pTQDCoTa80WMdAvg/1/index/_a.cfs\\\") [slice=_a_Lucene50_0.tim]\",\r\n         \"caused_by\": {\r\n            \"type\": \"i_o_exception\",\r\n            \"reason\": \"Input/output error\"\r\n         }\r\n      }\r\n   },\r\n   \"status\": 500\r\n}\r\n```\r\n\r\nLogs show an exception\r\n\r\n```\r\n[2016-05-11 11:52:26,911][DEBUG][action.admin.indices.stats] [Vindicator] [indices:monitor/stats] failed to execute operation for shard [[[test1003/01ABN7pTQDCoTa80WMdAvg]][0], node[AMr_NWrVSFCuNV-YCOfsVg], [P], s[STARTED], a[id=IMwYwgWrTLCZYa08WJRNvg]]\r\nElasticsearchException[failed to refresh store stats]; nested: NoSuchFileException[/Volumes/KINGSTON/elasticsearch/nodes/0/indices/01ABN7pTQDCoTa80WMdAvg/0/index];\r\n\tat org.elasticsearch.index.store.Store$StoreStatsCache.refresh(Store.java:1411)\r\n\tat org.elasticsearch.index.store.Store$StoreStatsCache.refresh(Store.java:1396)\r\n\tat org.elasticsearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:54)\r\n\tat org.elasticsearch.index.store.Store.stats(Store.java:321)\r\n\tat org.elasticsearch.index.shard.IndexShard.storeStats(IndexShard.java:632)\r\n\tat org.elasticsearch.action.admin.indices.stats.CommonStats.<init>(CommonStats.java:137)\r\n\tat org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:166)\r\n\tat org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:47)\r\n\tat org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.onShardOperation(TransportBroadcastByNodeAction.java:414)\r\n\tat org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:393)\r\n\tat org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:380)\r\n\tat org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)\r\n\tat org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:65)\r\n\tat org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:376)\r\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:468)\r\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n\tat java.lang.Thread.run(Thread.java:745)\r\nCaused by: java.nio.file.NoSuchFileException: /Volumes/KINGSTON/elasticsearch/nodes/0/indices/01ABN7pTQDCoTa80WMdAvg/0/index\r\n\tat sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)\r\n\tat sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)\r\n\tat sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)\r\n\tat sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:407)\r\n\tat java.nio.file.Files.newDirectoryStream(Files.java:457)\r\n\tat org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:215)\r\n\tat org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:234)\r\n\tat org.elasticsearch.index.store.FsDirectoryService$1.listAll(FsDirectoryService.java:135)\r\n\tat org.apache.lucene.store.FilterDirectory.listAll(FilterDirectory.java:57)\r\n\tat org.apache.lucene.store.FilterDirectory.listAll(FilterDirectory.java:57)\r\n\tat org.elasticsearch.index.store.Store$StoreStatsCache.estimateSize(Store.java:1417)\r\n\tat org.elasticsearch.index.store.Store$StoreStatsCache.refresh(Store.java:1409)\r\n\t... 18 more\r\n```\r\n\r\n`_cat/shards` still show all shards STARTED\r\n\r\n```\r\nindex    shard prirep state   docs store ip        node\r\ntest1003 4     r      STARTED            127.0.0.1 Jacqueline Falsworth\r\ntest1003 4     p      STARTED            127.0.0.1 Vindicator\r\ntest1003 3     r      STARTED            127.0.0.1 Jacqueline Falsworth\r\ntest1003 3     p      STARTED            127.0.0.1 Vindicator\r\ntest1003 1     r      STARTED            127.0.0.1 Jacqueline Falsworth\r\ntest1003 1     p      STARTED            127.0.0.1 Vindicator\r\ntest1003 2     r      STARTED            127.0.0.1 Jacqueline Falsworth\r\ntest1003 2     p      STARTED            127.0.0.1 Vindicator\r\ntest1003 0     r      STARTED            127.0.0.1 Jacqueline Falsworth\r\ntest1003 0     p      STARTED            127.0.0.1 Vindicator\r\n```\r\n\r\n\r\n5) Wait 5 minutes,  Search some data:\r\n\r\nNo change\r\n\r\n```\r\n{\r\n   \"took\": 15,\r\n   \"timed_out\": false,\r\n   \"_shards\": {\r\n      \"total\": 5,\r\n      \"successful\": 3,\r\n      \"failed\": 2,\r\n      \"failures\": [\r\n         {\r\n            \"shard\": 0,\r\n            \"index\": \"test1003\",\r\n            \"node\": \"AMr_NWrVSFCuNV-YCOfsVg\",\r\n            \"reason\": {\r\n               \"type\": \"i_o_exception\",\r\n               \"reason\": \"Input/output error: NIOFSIndexInput(path=\\\"/Volumes/KINGSTON/elasticsearch/nodes/0/indices/01ABN7pTQDCoTa80WMdAvg/0/index/_0.cfs\\\") [slice=_0.fdt]\",\r\n               \"caused_by\": {\r\n                  \"type\": \"i_o_exception\",\r\n                  \"reason\": \"Input/output error\"\r\n               }\r\n            }\r\n         },\r\n         {\r\n            \"shard\": 1,\r\n            \"index\": \"test1003\",\r\n            \"node\": \"wK5mnEIaT82Wz3wdTAjv6Q\",\r\n            \"reason\": {\r\n               \"type\": \"i_o_exception\",\r\n               \"reason\": \"Input/output error: NIOFSIndexInput(path=\\\"/Volumes/KINGSTON/elasticsearch/nodes/1/indices/01ABN7pTQDCoTa80WMdAvg/1/index/_2.cfs\\\") [slice=_2.fdt]\",\r\n               \"caused_by\": {\r\n                  \"type\": \"i_o_exception\",\r\n                  \"reason\": \"Input/output error\"\r\n               }\r\n            }\r\n         }\r\n      ]\r\n   },\r\n   \"hits\": {\r\n      \"total\": 23,\r\n      \"max_score\": 1,\r\n      \"hits\": []\r\n   }\r\n}\r\n```\r\n\r\n```\r\nindex    shard prirep state   docs store ip        node\r\ntest1003 4     r      STARTED            127.0.0.1 Jacqueline Falsworth\r\ntest1003 4     p      STARTED            127.0.0.1 Vindicator\r\ntest1003 3     r      STARTED            127.0.0.1 Jacqueline Falsworth\r\ntest1003 3     p      STARTED            127.0.0.1 Vindicator\r\ntest1003 1     r      STARTED            127.0.0.1 Jacqueline Falsworth\r\ntest1003 1     p      STARTED            127.0.0.1 Vindicator\r\ntest1003 2     r      STARTED            127.0.0.1 Jacqueline Falsworth\r\ntest1003 2     p      STARTED            127.0.0.1 Vindicator\r\ntest1003 0     r      STARTED            127.0.0.1 Jacqueline Falsworth\r\ntest1003 0     p      STARTED            127.0.0.1 Vindicator\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/comments/219038755","html_url":"https://github.com/elastic/elasticsearch/issues/18279#issuecomment-219038755","issue_url":"https://api.github.com/repos/elastic/elasticsearch/issues/18279","id":219038755,"user":{"login":"PhaedrusTheGreek","id":4387023,"avatar_url":"https://avatars.githubusercontent.com/u/4387023?v=3","gravatar_id":"","url":"https://api.github.com/users/PhaedrusTheGreek","html_url":"https://github.com/PhaedrusTheGreek","followers_url":"https://api.github.com/users/PhaedrusTheGreek/followers","following_url":"https://api.github.com/users/PhaedrusTheGreek/following{/other_user}","gists_url":"https://api.github.com/users/PhaedrusTheGreek/gists{/gist_id}","starred_url":"https://api.github.com/users/PhaedrusTheGreek/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/PhaedrusTheGreek/subscriptions","organizations_url":"https://api.github.com/users/PhaedrusTheGreek/orgs","repos_url":"https://api.github.com/users/PhaedrusTheGreek/repos","events_url":"https://api.github.com/users/PhaedrusTheGreek/events{/privacy}","received_events_url":"https://api.github.com/users/PhaedrusTheGreek/received_events","type":"User","site_admin":false},"created_at":"2016-05-13T13:12:41Z","updated_at":"2016-05-13T13:12:41Z","body":"Definitely we don't want to *introduce* any resiliency issues.    Some manual intervention makes sense, but restarting a node can sometimes take a long time.    Should there be something like delayed allocation on marking a path.data as failed? - there is the case of something like NFS, where a network problem might make the drive appear to come and go."}}