{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/18558","repository_url":"https://api.github.com/repos/elastic/elasticsearch","labels_url":"https://api.github.com/repos/elastic/elasticsearch/issues/18558/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/issues/18558/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch/issues/18558/events","html_url":"https://github.com/elastic/elasticsearch/issues/18558","id":156603941,"number":18558,"title":"[CI] org.elasticsearch.indices.IndicesServiceTests.testDeleteIndexStore fails due to state file not existing","user":{"login":"dakrone","id":19060,"avatar_url":"https://avatars.githubusercontent.com/u/19060?v=3","gravatar_id":"","url":"https://api.github.com/users/dakrone","html_url":"https://github.com/dakrone","followers_url":"https://api.github.com/users/dakrone/followers","following_url":"https://api.github.com/users/dakrone/following{/other_user}","gists_url":"https://api.github.com/users/dakrone/gists{/gist_id}","starred_url":"https://api.github.com/users/dakrone/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dakrone/subscriptions","organizations_url":"https://api.github.com/users/dakrone/orgs","repos_url":"https://api.github.com/users/dakrone/repos","events_url":"https://api.github.com/users/dakrone/events{/privacy}","received_events_url":"https://api.github.com/users/dakrone/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/test","name":"test","color":"5319e7"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2016-05-24T20:40:51Z","updated_at":"2016-05-26T11:37:14Z","closed_at":null,"body":"https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-intake/636/console\r\n\r\nThe failure is:\r\n\r\n```\r\nERROR   0.14s J1 | IndicesServiceTests.testDeleteIndexStore <<< FAILURES!\r\n   > Throwable #1: ElasticsearchException[java.io.IOException: failed to read [id:1, legacy:false, file:/var/lib/jenkins/workspace/elastic+elasticsearch+master+multijob-intake/core/build/testrun/test/J1/temp/org.elasticsearch.indices.IndicesServiceTests_678304BAB99F89CE-001/tempDir-011/data/single-node-cluster-CHILD_VM=[1]-CLUSTER_SEED=[3730156636843346220]-HASH=[2CC2E218290088]/nodes/0/_state/global-1.st]]; nested: IOException[failed to read [id:1, legacy:false, file:/var/lib/jenkins/workspace/elastic+elasticsearch+master+multijob-intake/core/build/testrun/test/J1/temp/org.elasticsearch.indices.IndicesServiceTests_678304BAB99F89CE-001/tempDir-011/data/single-node-cluster-CHILD_VM=[1]-CLUSTER_SEED=[3730156636843346220]-HASH=[2CC2E218290088]/nodes/0/_state/global-1.st]]; nested: NoSuchFileException[/var/lib/jenkins/workspace/elastic+elasticsearch+master+multijob-intake/core/build/testrun/test/J1/temp/org.elasticsearch.indices.IndicesServiceTests_678304BAB99F89CE-001/tempDir-011/data/single-node-cluster-CHILD_VM=[1]-CLUSTER_SEED=[3730156636843346220]-HASH=[2CC2E218290088]/nodes/0/_state/global-1.st];\r\n   > \tat __randomizedtesting.SeedInfo.seed([678304BAB99F89CE:D34003C59E7578EF]:0)\r\n   > \tat org.elasticsearch.ExceptionsHelper.maybeThrowRuntimeAndSuppress(ExceptionsHelper.java:164)\r\n   > \tat org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:321)\r\n   > \tat org.elasticsearch.gateway.MetaStateService.loadGlobalState(MetaStateService.java:109)\r\n   > \tat org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:54)\r\n   > \tat org.elasticsearch.gateway.GatewayMetaState.loadMetaState(GatewayMetaState.java:102)\r\n   > \tat org.elasticsearch.indices.IndicesServiceTests.testDeleteIndexStore(IndicesServiceTests.java:128)\r\n   > \tat java.lang.Thread.run(Thread.java:745)\r\n   > Caused by: java.io.IOException: failed to read [id:1, legacy:false, file:/var/lib/jenkins/workspace/elastic+elasticsearch+master+multijob-intake/core/build/testrun/test/J1/temp/org.elasticsearch.indices.IndicesServiceTests_678304BAB99F89CE-001/tempDir-011/data/single-node-cluster-CHILD_VM=[1]-CLUSTER_SEED=[3730156636843346220]-HASH=[2CC2E218290088]/nodes/0/_state/global-1.st]\r\n   > \tat org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:316)\r\n   > \t... 41 more\r\n   > Caused by: java.nio.file.NoSuchFileException: /var/lib/jenkins/workspace/elastic+elasticsearch+master+multijob-intake/core/build/testrun/test/J1/temp/org.elasticsearch.indices.IndicesServiceTests_678304BAB99F89CE-001/tempDir-011/data/single-node-cluster-CHILD_VM=[1]-CLUSTER_SEED=[3730156636843346220]-HASH=[2CC2E218290088]/nodes/0/_state/global-1.st\r\n   > \tat sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)\r\n   > \tat sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)\r\n   > \tat sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)\r\n   > \tat sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)\r\n   > \tat org.apache.lucene.mockfile.FilterFileSystemProvider.newByteChannel(FilterFileSystemProvider.java:212)\r\n   > \tat org.apache.lucene.mockfile.FilterFileSystemProvider.newByteChannel(FilterFileSystemProvider.java:212)\r\n   > \tat org.apache.lucene.mockfile.FilterFileSystemProvider.newByteChannel(FilterFileSystemProvider.java:212)\r\n   > \tat org.apache.lucene.mockfile.HandleTrackingFS.newByteChannel(HandleTrackingFS.java:240)\r\n   > \tat org.apache.lucene.mockfile.FilterFileSystemProvider.newByteChannel(FilterFileSystemProvider.java:212)\r\n   > \tat org.apache.lucene.mockfile.HandleTrackingFS.newByteChannel(HandleTrackingFS.java:240)\r\n   > \tat org.apache.lucene.mockfile.FilterFileSystemProvider.newByteChannel(FilterFileSystemProvider.java:212)\r\n   > \tat java.nio.file.Files.newByteChannel(Files.java:361)\r\n   > \tat java.nio.file.Files.newByteChannel(Files.java:407)\r\n   > \tat org.apache.lucene.store.SimpleFSDirectory.openInput(SimpleFSDirectory.java:77)\r\n   > \tat org.elasticsearch.gateway.MetaDataStateFormat.read(MetaDataStateFormat.java:182)\r\n   > \tat org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:311)\r\n   > \t... 41 more\r\n```\r\n\r\nHowever, it does not reproduce, here's the reproduction line:\r\n\r\n```\r\ngradle :core:test -Dtests.seed=678304BAB99F89CE -Dtests.class=org.elasticsearch.indices.IndicesServiceTests -Dtests.method=\"testDeleteIndexStore\" -Dtests.security.manager=true -Dtests.locale=sv-SE -Dtests.timezone=Asia/Macau\r\n```"},"comment":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/comments/221847588","html_url":"https://github.com/elastic/elasticsearch/issues/18558#issuecomment-221847588","issue_url":"https://api.github.com/repos/elastic/elasticsearch/issues/18558","id":221847588,"user":{"login":"ywelsch","id":3718355,"avatar_url":"https://avatars.githubusercontent.com/u/3718355?v=3","gravatar_id":"","url":"https://api.github.com/users/ywelsch","html_url":"https://github.com/ywelsch","followers_url":"https://api.github.com/users/ywelsch/followers","following_url":"https://api.github.com/users/ywelsch/following{/other_user}","gists_url":"https://api.github.com/users/ywelsch/gists{/gist_id}","starred_url":"https://api.github.com/users/ywelsch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ywelsch/subscriptions","organizations_url":"https://api.github.com/users/ywelsch/orgs","repos_url":"https://api.github.com/users/ywelsch/repos","events_url":"https://api.github.com/users/ywelsch/events{/privacy}","received_events_url":"https://api.github.com/users/ywelsch/received_events","type":"User","site_admin":false},"created_at":"2016-05-26T11:37:14Z","updated_at":"2016-05-26T11:37:14Z","body":"The lines that lead to the failure are\r\n```\r\nassertAcked(client().admin().indices().prepareDelete(\"test\"));\r\nmeta = gwMetaState.loadMetaState();\r\n```\r\n\r\nTo understand why this fails, it is important to know that acknowledgment of deletion can happen before the new cluster state containing the deletion has been written to disk, even if the cluster is only single-node.\r\nNow, the way `loadMetaState` works is by listing the directory contents to find the *latest* meta state file.  Latest means state file with highest version identifier attached to it. In a second step, it tries to read the file. If the meta state file has been replaced in the meanwhile by an updated version (and the old version deleted), we try to read the file that has been deleted.\r\n\r\nloadMetaState() is only safe to call from the cluster state update thread, the same thread where we write the state files.\r\n\r\nTo fix the test, we could add the line\r\n```\r\nassertBusy(() -> assertThat(client().admin().cluster().preparePendingClusterTasks().get().getPendingTasks(), empty()));\r\n```\r\nbetween the index deletion and loadMetaState(). @bleskes what do you think?"}}
{"actor":{"display_login":"ywelsch"}}