{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/30456","repository_url":"https://api.github.com/repos/elastic/elasticsearch","labels_url":"https://api.github.com/repos/elastic/elasticsearch/issues/30456/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/issues/30456/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch/issues/30456/events","html_url":"https://github.com/elastic/elasticsearch/issues/30456","id":321188898,"node_id":"MDU6SXNzdWUzMjExODg4OTg=","number":30456,"title":"UpgradeClusterClientYamlTestSuiteIT fails intermittently ","user":{"login":"atorok","id":2565652,"node_id":"MDQ6VXNlcjI1NjU2NTI=","avatar_url":"https://avatars1.githubusercontent.com/u/2565652?v=4","gravatar_id":"","url":"https://api.github.com/users/atorok","html_url":"https://github.com/atorok","followers_url":"https://api.github.com/users/atorok/followers","following_url":"https://api.github.com/users/atorok/following{/other_user}","gists_url":"https://api.github.com/users/atorok/gists{/gist_id}","starred_url":"https://api.github.com/users/atorok/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/atorok/subscriptions","organizations_url":"https://api.github.com/users/atorok/orgs","repos_url":"https://api.github.com/users/atorok/repos","events_url":"https://api.github.com/users/atorok/events{/privacy}","received_events_url":"https://api.github.com/users/atorok/received_events","type":"User","site_admin":false},"labels":[{"id":912833043,"node_id":"MDU6TGFiZWw5MTI4MzMwNDM=","url":"https://api.github.com/repos/elastic/elasticsearch/labels/:ml","name":":ml","color":"0e8a16","default":false},{"id":148612629,"node_id":"MDU6TGFiZWwxNDg2MTI2Mjk=","url":"https://api.github.com/repos/elastic/elasticsearch/labels/%3Etest-failure","name":">test-failure","color":"207de5","default":false},{"id":912911731,"node_id":"MDU6TGFiZWw5MTI5MTE3MzE=","url":"https://api.github.com/repos/elastic/elasticsearch/labels/v6.4.0","name":"v6.4.0","color":"DDDDDD","default":false},{"id":484855464,"node_id":"MDU6TGFiZWw0ODQ4NTU0NjQ=","url":"https://api.github.com/repos/elastic/elasticsearch/labels/v7.0.0","name":"v7.0.0","color":"dddddd","default":false}],"state":"open","locked":false,"assignee":{"login":"davidkyle","id":2353640,"node_id":"MDQ6VXNlcjIzNTM2NDA=","avatar_url":"https://avatars1.githubusercontent.com/u/2353640?v=4","gravatar_id":"","url":"https://api.github.com/users/davidkyle","html_url":"https://github.com/davidkyle","followers_url":"https://api.github.com/users/davidkyle/followers","following_url":"https://api.github.com/users/davidkyle/following{/other_user}","gists_url":"https://api.github.com/users/davidkyle/gists{/gist_id}","starred_url":"https://api.github.com/users/davidkyle/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davidkyle/subscriptions","organizations_url":"https://api.github.com/users/davidkyle/orgs","repos_url":"https://api.github.com/users/davidkyle/repos","events_url":"https://api.github.com/users/davidkyle/events{/privacy}","received_events_url":"https://api.github.com/users/davidkyle/received_events","type":"User","site_admin":false},"assignees":[{"login":"davidkyle","id":2353640,"node_id":"MDQ6VXNlcjIzNTM2NDA=","avatar_url":"https://avatars1.githubusercontent.com/u/2353640?v=4","gravatar_id":"","url":"https://api.github.com/users/davidkyle","html_url":"https://github.com/davidkyle","followers_url":"https://api.github.com/users/davidkyle/followers","following_url":"https://api.github.com/users/davidkyle/following{/other_user}","gists_url":"https://api.github.com/users/davidkyle/gists{/gist_id}","starred_url":"https://api.github.com/users/davidkyle/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davidkyle/subscriptions","organizations_url":"https://api.github.com/users/davidkyle/orgs","repos_url":"https://api.github.com/users/davidkyle/repos","events_url":"https://api.github.com/users/davidkyle/events{/privacy}","received_events_url":"https://api.github.com/users/davidkyle/received_events","type":"User","site_admin":false}],"milestone":null,"comments":28,"created_at":"2018-05-08T13:34:54Z","updated_at":"2018-05-31T08:35:37Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"Discovered while building `#30367` [Failed](https://gradle.com/s/54rh47i5qtii2)\r\n\r\nReproducible using :\r\n```\r\n./gradlew :x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#mixedClusterTestRunner\r\n-Dtests.seed=D2CB09031F59783F\r\n-Dtests.class=org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT\r\n-Dtests.security.manager=true -Dtests.locale=en-BB -Dtests.timezone=Australia/ACT\r\n-Dtests.rest.suite=mixed_cluster\r\n```\r\nSaw different failures while trying to reproduce it in isolation: \r\n\r\n```\r\n 2> REPRODUCE WITH: ./gradlew :x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#mixedClusterTestRunner -Dtests.seed=D2CB09031F59783F -Dtests.class=org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT -Dtests.method=\"test {p0=mixed_cluster/30_ml_jobs_crud/Test get old cluster job}\" -Dtests.security.manager=true -Dtests.locale=en-BB -Dtests.timezone=Australia/ACT -Dtests.rest.suite=mixed_cluster\r\nFAILURE 1.40s | UpgradeClusterClientYamlTestSuiteIT.test {p0=mixed_cluster/30_ml_jobs_crud/Test get old cluster job} <<< FAILURES!\r\n   > Throwable #1: java.lang.AssertionError: Failure at [mixed_cluster/30_ml_jobs_crud:33]: field [jobs.0.node] doesn't have a true value\r\n   > Expected: not null\r\n   >      but: was null\r\n   >    at __randomizedtesting.SeedInfo.seed([D2CB09031F59783F:5A9F36D9B1A515C7]:0)\r\n   >    at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:343)\r\n   >    at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:320)\r\n   >    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\r\n   >    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\r\n   >    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\n   >    at java.base/java.lang.reflect.Method.invoke(Method.java:564)\r\n   >    at java.base/java.lang.Thread.run(Thread.java:844)\r\n   > Caused by: java.lang.AssertionError: field [jobs.0.node] doesn't have a true value\r\n   > Expected: not null\r\n   >      but: was null\r\n   >    at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)\r\n   >    at org.elasticsearch.test.rest.yaml.section.IsTrueAssertion.doAssert(IsTrueAssertion.java:55)\r\n   >    at org.elasticsearch.test.rest.yaml.section.Assertion.execute(Assertion.java:76)\r\n   >    at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:336)\r\n   >    ... 37 more\r\n```\r\nand\r\n```\r\n> Task :x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#oldClusterTestRunner\r\n==> Test Summary: 1 suite, 10 tests, 1 ignored (1 assumption)\r\n   [junit4] JVM J0:     1.08 ..    22.93 =    21.85s\r\n   [junit4] Execution time total: 22 seconds\r\n   [junit4] Tests summary: 1 suite, 10 tests, 1 ignored (1 assumption)\r\n\r\n> Task :x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#mixedClusterTestRunner\r\n   [junit4] <JUnit4> says ciao! Master seed: D2CB09031F59783F\r\n==> Test Info: seed=D2CB09031F59783F; jvm=1; suite=1\r\n\r\nSuite: org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT\r\n  1> [2018-05-08T21:32:10,505][INFO ][o.e.u.UpgradeClusterClientYamlTestSuiteIT] [test {p0=mixed_cluster/40_ml_datafeed_crud/Put job and datafeed in mixed cluster}]: before test\r\n  1> [2018-05-08T21:32:10,636][INFO ][o.e.u.UpgradeClusterClientYamlTestSuiteIT] initializing REST clients against [http://[::1]:33111]\r\n  1> [2018-05-08T21:32:11,479][INFO ][o.e.u.UpgradeClusterClientYamlTestSuiteIT] initializing yaml client, minimum es version: [6.4.0] master version: [6.4.0] hosts: [http://[::1]:33111]\r\n  1> [2018-05-08T21:32:12,210][INFO ][o.e.u.UpgradeClusterClientYamlTestSuiteIT] [test {p0=mixed_cluster/40_ml_datafeed_crud/Put job and datafeed in mixed cluster}]: after test\r\n  1> [2018-05-08T21:32:12,234][INFO ][o.e.u.UpgradeClusterClientYamlTestSuiteIT] [test {p0=mixed_cluster/20_security/Verify user and role in mixed cluster}]: before test\r\n  1> [2018-05-08T21:32:12,965][INFO ][o.e.u.UpgradeClusterClientYamlTestSuiteIT] [test {p0=mixed_cluster/20_security/Verify user and role in mixed cluster}]: after test\r\n  1> [2018-05-08T21:32:12,974][INFO ][o.e.u.UpgradeClusterClientYamlTestSuiteIT] [test {p0=mixed_cluster/10_basic/Index data and search on the mixed cluster}]: before test\r\n  1> [2018-05-08T21:32:13,795][INFO ][o.e.u.UpgradeClusterClientYamlTestSuiteIT] [test {p0=mixed_cluster/10_basic/Index data and search on the mixed cluster}]: after test\r\n  1> [2018-05-08T21:32:13,805][INFO ][o.e.u.UpgradeClusterClientYamlTestSuiteIT] [test {p0=mixed_cluster/30_ml_jobs_crud/Test get old cluster job}]: before test\r\n  1> [2018-05-08T21:32:14,042][INFO ][o.e.u.UpgradeClusterClientYamlTestSuiteIT] Stash dump on test failure [{\r\n  1>   \"stash\" : {\r\n  1>     \"body\" : {\r\n  1>       \"count\" : 1,\r\n  1>       \"jobs\" : [\r\n  1>         {\r\n  1>           \"job_id\" : \"old-cluster-job\",\r\n  1>           \"data_counts\" : {\r\n  1>             \"job_id\" : \"old-cluster-job\",\r\n  1>             \"processed_record_count\" : 0,\r\n  1>             \"processed_field_count\" : 0,\r\n  1>             \"input_bytes\" : 0,\r\n  1>             \"input_field_count\" : 0,\r\n  1>             \"invalid_date_count\" : 0,\r\n  1>             \"missing_field_count\" : 0,\r\n  1>             \"out_of_order_timestamp_count\" : 0,\r\n  1>             \"empty_bucket_count\" : 0,\r\n  1>             \"sparse_bucket_count\" : 0,\r\n  1>             \"bucket_count\" : 0,\r\n  1>             \"input_record_count\" : 0\r\n  1>           },\r\n  1>           \"model_size_stats\" : {\r\n  1>             \"job_id\" : \"old-cluster-job\",\r\n  1>             \"result_type\" : \"model_size_stats\",\r\n  1>             \"model_bytes\" : 146088,\r\n  1>             \"total_by_field_count\" : 4,\r\n  1>             \"total_over_field_count\" : 0,\r\n  1>             \"total_partition_field_count\" : 2,\r\n  1>             \"bucket_allocation_failures_count\" : 0,\r\n  1>             \"memory_status\" : \"ok\",\r\n  1>             \"log_time\" : 1525779101000,\r\n  1>             \"timestamp\" : 1403481600000\r\n  1>           },\r\n  1>           \"state\" : \"closed\"\r\n  1>         }\r\n  1>       ]\r\n  1>     }\r\n  1>   }\r\n  1> }]\r\n  1> [2018-05-08T21:32:14,168][INFO ][o.e.u.UpgradeClusterClientYamlTestSuiteIT] [test {p0=mixed_cluster/30_ml_jobs_crud/Test get old cluster job}]: after test\r\n  2> REPRODUCE WITH: ./gradlew :x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#mixedClusterTestRunner -Dtests.seed=D2CB09031F59783F -Dtests.class=org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT -Dtests.method=\"test {p0=mixed_cluster/30_ml_jobs_crud/Test get old cluster job}\" -Dtests.security.manager=true -Dtests.locale=en-BB -Dtests.timezone=Australia/ACT -Dtests.rest.suite=mixed_cluster\r\nFAILURE 0.38s | UpgradeClusterClientYamlTestSuiteIT.test {p0=mixed_cluster/30_ml_jobs_crud/Test get old cluster job} <<< FAILURES!\r\n   > Throwable #1: java.lang.AssertionError: Failure at [mixed_cluster/30_ml_jobs_crud:19]: jobs.0.data_counts.processed_record_count didn't match expected value:\r\n   > jobs.0.data_counts.processed_record_count: expected [2] but was [0]\r\n   >    at __randomizedtesting.SeedInfo.seed([D2CB09031F59783F:5A9F36D9B1A515C7]:0)\r\n   >    at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:343)\r\n   >    at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:320)\r\n   >    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\r\n   >    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\r\n   >    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\n   >    at java.base/java.lang.reflect.Method.invoke(Method.java:564)\r\n   >    at java.base/java.lang.Thread.run(Thread.java:844)\r\n   > Caused by: java.lang.AssertionError: jobs.0.data_counts.processed_record_count didn't match expected value:\r\n   > jobs.0.data_counts.processed_record_count: expected [2] but was [0]\r\n   >    at org.elasticsearch.test.rest.yaml.section.MatchAssertion.doAssert(MatchAssertion.java:87)\r\n   >    at org.elasticsearch.test.rest.yaml.section.Assertion.execute(Assertion.java:76)\r\n   >    at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:336)\r\n   >    ... 37 more\r\n  1> [2018-05-08T21:32:14,183][INFO ][o.e.u.UpgradeClusterClientYamlTestSuiteIT] [test {p0=mixed_cluster/50_token_auth/Get the indexed token and use if to authenticate}]: before test\r\n  1> [2018-05-08T21:32:14,671][INFO ][o.e.u.UpgradeClusterClientYamlTestSuiteIT] There are still tasks running after this test that might break subsequent tests [internal:index/shard/recovery/prepare_translog, internal:index/shard/recovery/start_recovery, internal:index/shard/recovery/translog_ops].\r\n  1> [2018-05-08T21:32:14,671][INFO ][o.e.u.UpgradeClusterClientYamlTestSuiteIT] [test {p0=mixed_cluster/50_token_auth/Get the indexed token and use if to authenticate}]: after test\r\n  1> [2018-05-08T21:32:14,704][INFO ][o.e.u.UpgradeClusterClientYamlTestSuiteIT] [test {p0=mixed_cluster/30_ml_jobs_crud/Test get job with rules}]: before test\r\n  1> [2018-05-08T21:32:15,014][INFO ][o.e.u.UpgradeClusterClientYamlTestSuiteIT] There are still tasks running after this test that might break subsequent tests [internal:index/shard/recovery/start_recovery, internal:index/shard/recovery/translog_ops].\r\n  1> [2018-05-08T21:32:15,014][INFO ][o.e.u.UpgradeClusterClientYamlTestSuiteIT] [test {p0=mixed_cluster/30_ml_jobs_crud/Test get job with rules}]: after test\r\n  1> [2018-05-08T21:32:15,045][INFO ][o.e.u.UpgradeClusterClientYamlTestSuiteIT] [test {p0=mixed_cluster/20_security/verify users for default password migration in mixed cluster}]: before test\r\n  1> [2018-05-08T21:32:15,431][INFO ][o.e.u.UpgradeClusterClientYamlTestSuiteIT] [test {p0=mixed_cluster/20_security/verify users for default password migration in mixed cluster}]: after test\r\n  1> [2018-05-08T21:32:15,440][INFO ][o.e.u.UpgradeClusterClientYamlTestSuiteIT] [test {p0=mixed_cluster/10_basic/Start scroll in mixed cluster for upgraded}]: before test\r\n  1> [2018-05-08T21:32:17,380][INFO ][o.e.u.UpgradeClusterClientYamlTestSuiteIT] [test {p0=mixed_cluster/10_basic/Start scroll in mixed cluster for upgraded}]: after test\r\n  1> [2018-05-08T21:32:17,398][INFO ][o.e.u.UpgradeClusterClientYamlTestSuiteIT] [test {p0=mixed_cluster/10_basic/Basic scroll mixed}]: before test\r\n  1> [2018-05-08T21:32:18,322][INFO ][o.e.u.UpgradeClusterClientYamlTestSuiteIT] [test {p0=mixed_cluster/10_basic/Basic scroll mixed}]: after test\r\n  1> [2018-05-08T21:32:18,337][INFO ][o.e.u.UpgradeClusterClientYamlTestSuiteIT] [test {p0=mixed_cluster/30_ml_jobs_crud/Create a job in the mixed cluster and write some data}]: before test\r\n  1> [2018-05-08T21:32:19,989][INFO ][o.e.u.UpgradeClusterClientYamlTestSuiteIT] [test {p0=mixed_cluster/30_ml_jobs_crud/Create a job in the mixed cluster and write some data}]: after test\r\n  1> [2018-05-08T21:32:19,995][INFO ][o.e.u.UpgradeClusterClientYamlTestSuiteIT] [test {p0=mixed_cluster/40_ml_datafeed_crud/Test old cluster datafeed}]: before test\r\n  1> [2018-05-08T21:32:20,135][INFO ][o.e.u.UpgradeClusterClientYamlTestSuiteIT] [test {p0=mixed_cluster/40_ml_datafeed_crud/Test old cluster datafeed}]: after test\r\n  2> NOTE: leaving temporary files on disk at: /home/alpar.torok/elasticsearch/x-pack/qa/rolling-upgrade/with-system-key/build/testrun/v6.4.0-SNAPSHOT#mixedClusterTestRunner/J0/temp/org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT_D2CB09031F59783F-001\r\n  2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=203, maxMBSortInHeap=6.586752071408995, sim=RandomSimilarity(queryNorm=false): {}, locale=en-BB, timezone=Australia/ACT\r\n  2> NOTE: Linux 4.15.0-1006-gcp amd64/Oracle Corporation 10.0.1 (64-bit)/cpus=8,threads=1,free=500541920,total=536870912\r\n  2> NOTE: All tests run in this JVM: [UpgradeClusterClientYamlTestSuiteIT]\r\nCompleted [1/1] in 9.96s, 11 tests, 1 failure <<< FAILURES! \r\n```\r\nFirst failure reproduced 3 times for, the second one only once, and got 3 successful runs as well. \r\n"},"comment":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/comments/393457148","html_url":"https://github.com/elastic/elasticsearch/issues/30456#issuecomment-393457148","issue_url":"https://api.github.com/repos/elastic/elasticsearch/issues/30456","id":393457148,"node_id":"MDEyOklzc3VlQ29tbWVudDM5MzQ1NzE0OA==","user":{"login":"droberts195","id":7405510,"node_id":"MDQ6VXNlcjc0MDU1MTA=","avatar_url":"https://avatars0.githubusercontent.com/u/7405510?v=4","gravatar_id":"","url":"https://api.github.com/users/droberts195","html_url":"https://github.com/droberts195","followers_url":"https://api.github.com/users/droberts195/followers","following_url":"https://api.github.com/users/droberts195/following{/other_user}","gists_url":"https://api.github.com/users/droberts195/gists{/gist_id}","starred_url":"https://api.github.com/users/droberts195/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/droberts195/subscriptions","organizations_url":"https://api.github.com/users/droberts195/orgs","repos_url":"https://api.github.com/users/droberts195/repos","events_url":"https://api.github.com/users/droberts195/events{/privacy}","received_events_url":"https://api.github.com/users/droberts195/received_events","type":"User","site_admin":false},"created_at":"2018-05-31T08:35:36Z","updated_at":"2018-05-31T08:35:36Z","author_association":"CONTRIBUTOR","body":"I think the most recent failures are all due to the `autodetect` process core dumping with a SEGV.  The first failure @imotov reported above contains this (and on Linux signal 11 is SEGV):\r\n\r\n> 19:49:03 [2018-05-30T19:48:57,900][ERROR][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/121564] [CDetachedProcessSpawner.cc@184] Child process with PID 121774 was terminated by signal 11\r\n\r\nThese C++ failures are split out into #30982.  As far as I can see all yesterday's failures are caused by this new problem.\r\n\r\nThe full logs for the first failure reported by @atorok are no longer available, but there a total of three problems that were fixed for this issue.  The first was a wire transport format incompatibility between master and 6.x that was fixed in #30512.  @davidkyle found that very quickly during the original investigation of this issue, so I suspect this was the cause of the first reported failure."}}
{"actor":{"display_login":"droberts195"}}