{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/50651","repository_url":"https://api.github.com/repos/elastic/elasticsearch","labels_url":"https://api.github.com/repos/elastic/elasticsearch/issues/50651/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/issues/50651/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch/issues/50651/events","html_url":"https://github.com/elastic/elasticsearch/issues/50651","id":545716047,"node_id":"MDU6SXNzdWU1NDU3MTYwNDc=","number":50651,"title":"ML test errors with AssertingRandom","user":{"login":"davidkyle","id":2353640,"node_id":"MDQ6VXNlcjIzNTM2NDA=","avatar_url":"https://avatars1.githubusercontent.com/u/2353640?v=4","gravatar_id":"","url":"https://api.github.com/users/davidkyle","html_url":"https://github.com/davidkyle","followers_url":"https://api.github.com/users/davidkyle/followers","following_url":"https://api.github.com/users/davidkyle/following{/other_user}","gists_url":"https://api.github.com/users/davidkyle/gists{/gist_id}","starred_url":"https://api.github.com/users/davidkyle/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davidkyle/subscriptions","organizations_url":"https://api.github.com/users/davidkyle/orgs","repos_url":"https://api.github.com/users/davidkyle/repos","events_url":"https://api.github.com/users/davidkyle/events{/privacy}","received_events_url":"https://api.github.com/users/davidkyle/received_events","type":"User","site_admin":false},"labels":[{"id":912833043,"node_id":"MDU6TGFiZWw5MTI4MzMwNDM=","url":"https://api.github.com/repos/elastic/elasticsearch/labels/:ml","name":":ml","color":"0e8a16","default":false,"description":"Machine learning"},{"id":148612629,"node_id":"MDU6TGFiZWwxNDg2MTI2Mjk=","url":"https://api.github.com/repos/elastic/elasticsearch/labels/%3Etest-failure","name":">test-failure","color":"207de5","default":false,"description":"Triaged test failures from CI"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2020-01-06T13:04:41Z","updated_at":"2020-01-06T13:04:44Z","closed_at":null,"author_association":"MEMBER","body":"https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-unix-compatibility/os=sles-12&&immutable/496/console\r\nhttps://gradle-enterprise.elastic.co/s/ggiermd5izfvs/tests/lf2lfu4ufazso-numqxpqxfbj2o\r\n\r\n`AssertingRandom` throws an `IllegalStateException` when the random is used by a different thread to the one it was created in. The exception then takes the node down causing the test to fail.  \r\n\r\n\r\n```\r\njava.lang.IllegalStateException: This Random was created for/by another thread (Thread[TEST-BasicDistributedJobsIT.testFailOverBasics_withDataFeeder-seed#[7B64EE984D68EA0C],5,TGRP-BasicDistributedJobsIT]). Random instances must not be shared (acquire per-thread). Current thread: Thread[elasticsearch[node_t0][ml_datafeed][T#1],5,TGRP-BasicDistributedJobsIT]\r\n\tat com.carrotsearch.randomizedtesting.AssertingRandom.checkValid(AssertingRandom.java:152) ~[randomizedtesting-runner-2.7.4.jar:?]\r\n\tat com.carrotsearch.randomizedtesting.AssertingRandom.nextInt(AssertingRandom.java:86) ~[randomizedtesting-runner-2.7.4.jar:?]\r\n\tat org.elasticsearch.xpack.ml.utils.persistence.ResultsPersisterService.bulkIndexWithRetry(ResultsPersisterService.java:121) ~[main/:?]\r\n\tat org.elasticsearch.xpack.ml.utils.persistence.ResultsPersisterService.indexWithRetry(ResultsPersisterService.java:82) ~[main/:?]\r\n\tat org.elasticsearch.xpack.ml.job.persistence.JobResultsPersister$Persistable.persist(JobResultsPersister.java:399) ~[main/:?]\r\n\tat org.elasticsearch.xpack.ml.job.persistence.JobResultsPersister.persistDatafeedTimingStats(JobResultsPersister.java:363) ~[main/:?]\r\n\tat org.elasticsearch.xpack.ml.datafeed.DatafeedTimingStatsReporter.flush(DatafeedTimingStatsReporter.java:104) [main/:?]\r\n\tat org.elasticsearch.xpack.ml.datafeed.DatafeedTimingStatsReporter.flushIfDifferSignificantly(DatafeedTimingStatsReporter.java:96) [main/:?]\r\n\tat org.elasticsearch.xpack.ml.datafeed.DatafeedTimingStatsReporter.reportSearchDuration(DatafeedTimingStatsReporter.java:64) [main/:?]\r\n\tat org.elasticsearch.xpack.ml.datafeed.extractor.chunked.ChunkedDataExtractor$DataSummaryFactory.newAggregatedDataSummary(ChunkedDataExtractor.java:233) [main/:?]\r\n\tat org.elasticsearch.xpack.ml.datafeed.extractor.chunked.ChunkedDataExtractor$DataSummaryFactory.buildDataSummary(ChunkedDataExtractor.java:202) [main/:?]\r\n\tat org.elasticsearch.xpack.ml.datafeed.extractor.chunked.ChunkedDataExtractor.setUpChunkedSearch(ChunkedDataExtractor.java:118) [main/:?]\r\n\tat org.elasticsearch.xpack.ml.datafeed.extractor.chunked.ChunkedDataExtractor.next(ChunkedDataExtractor.java:111) [main/:?]\r\n\tat org.elasticsearch.xpack.ml.datafeed.DatafeedJob.run(DatafeedJob.java:343) [main/:?]\r\n\tat org.elasticsearch.xpack.ml.datafeed.DatafeedJob.runLookBack(DatafeedJob.java:146) [main/:?]\r\n\tat org.elasticsearch.xpack.ml.datafeed.DatafeedManager$Holder.executeLookBack(DatafeedManager.java:408) [main/:?]\r\n\tat org.elasticsearch.xpack.ml.datafeed.DatafeedManager$2.doRun(DatafeedManager.java:176) [main/:?]\r\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]\r\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]\r\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]\r\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:629) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]\r\n\tat java.lang.Thread.run(Thread.java:834) [?:?]\r\nCaused by: com.carrotsearch.randomizedtesting.StackTraceHolder: Original allocation stack for this Random (allocated by Thread[TEST-BasicDistributedJobsIT.testFailOverBasics_withDataFeeder-seed#[7B64EE984D68EA0C],5,TGRP-BasicDistributedJobsIT])\r\n\tat java.lang.Thread.getStackTrace(Thread.java:1606) ~[?:?]\r\n\tat com.carrotsearch.randomizedtesting.AssertingRandom.<init>(AssertingRandom.java:40) ~[randomizedtesting-runner-2.7.4.jar:?]\r\n\tat com.carrotsearch.randomizedtesting.Randomness.<init>(Randomness.java:28) ~[randomizedtesting-runner-2.7.4.jar:?]\r\n\tat com.carrotsearch.randomizedtesting.Randomness.<init>(Randomness.java:35) ~[randomizedtesting-runner-2.7.4.jar:?]\r\n\tat com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:823) ~[randomizedtesting-runner-2.7.4.jar:?]\r\n\tat com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883) ~[randomizedtesting-runner-2.7.4.jar:?]\r\n\tat com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894) ~[randomizedtesting-runner-2.7.4.jar:?]\r\n\tat org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[lucene-test-framework-8.4.0-snapshot-08b8d116f8f.jar:8.4.0-snapshot-08b8d116f8f 08b8d116f8ffacf35a6b05ff4d37f2263b712347 - ivera - 2019-12-12 10:52:54]\r\n\tat com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.7.4.jar:?]\r\n\tat org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) ~[lucene-test-framework-8.4.0-snapshot-08b8d116f8f.jar:8.4.0-snapshot-08b8d116f8f 08b8d116f8ffacf35a6b05ff4d37f2263b712347 - ivera - 2019-12-12 10:52:54]\r\n\tat com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.7.4.jar:?]\r\n\tat com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.7.4.jar:?]\r\n\tat com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.7.4.jar:?]\r\n\tat com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.7.4.jar:?]\r\n\tat org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) ~[lucene-test-framework-8.4.0-snapshot-08b8d116f8f.jar:8.4.0-snapshot-08b8d116f8f 08b8d116f8ffacf35a6b05ff4d37f2263b712347 - ivera - 2019-12-12 10:52:54]\r\n\tat org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[lucene-test-framework-8.4.0-snapshot-08b8d116f8f.jar:8.4.0-snapshot-08b8d116f8f 08b8d116f8ffacf35a6b05ff4d37f2263b712347 - ivera - 2019-12-12 10:52:54]\r\n\tat org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[lucene-test-framework-8.4.0-snapshot-08b8d116f8f.jar:8.4.0-snapshot-08b8d116f8f 08b8d116f8ffacf35a6b05ff4d37f2263b712347 - ivera - 2019-12-12 10:52:54]\r\n\tat org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) ~[lucene-test-framework-8.4.0-snapshot-08b8d116f8f.jar:8.4.0-snapshot-08b8d116f8f 08b8d116f8ffacf35a6b05ff4d37f2263b712347 - ivera - 2019-12-12 10:52:54]\r\n\tat com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.7.4.jar:?]\r\n\tat com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.7.4.jar:?]\r\n\t... 1 more\r\n```\r\n\r\n\r\nDoes not reproduce\r\n```\r\n./gradlew ':x-pack:plugin:ml:internalClusterTest' --tests \"org.elasticsearch.xpack.ml.integration.BasicDistributedJobsIT.testFailOverBasics_withDataFeeder\" \\\r\n  -Dtests.seed=7B64EE984D68EA0C \\\r\n  -Dtests.security.manager=true \\\r\n  -Dtests.locale=fi \\\r\n  -Dtests.timezone=Africa/Johannesburg \\\r\n  -Dcompiler.java=13\r\n```"},"comment":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/comments/571130681","html_url":"https://github.com/elastic/elasticsearch/issues/50651#issuecomment-571130681","issue_url":"https://api.github.com/repos/elastic/elasticsearch/issues/50651","id":571130681,"node_id":"MDEyOklzc3VlQ29tbWVudDU3MTEzMDY4MQ==","user":{"login":"elasticmachine","id":15837671,"node_id":"MDQ6VXNlcjE1ODM3Njcx","avatar_url":"https://avatars3.githubusercontent.com/u/15837671?v=4","gravatar_id":"","url":"https://api.github.com/users/elasticmachine","html_url":"https://github.com/elasticmachine","followers_url":"https://api.github.com/users/elasticmachine/followers","following_url":"https://api.github.com/users/elasticmachine/following{/other_user}","gists_url":"https://api.github.com/users/elasticmachine/gists{/gist_id}","starred_url":"https://api.github.com/users/elasticmachine/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/elasticmachine/subscriptions","organizations_url":"https://api.github.com/users/elasticmachine/orgs","repos_url":"https://api.github.com/users/elasticmachine/repos","events_url":"https://api.github.com/users/elasticmachine/events{/privacy}","received_events_url":"https://api.github.com/users/elasticmachine/received_events","type":"User","site_admin":false},"created_at":"2020-01-06T13:04:43Z","updated_at":"2020-01-06T13:04:43Z","author_association":"COLLABORATOR","body":"Pinging @elastic/ml-core (:ml)"}}
{"actor":{"display_login":"elasticmachine"}}