{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/36928","repository_url":"https://api.github.com/repos/elastic/elasticsearch","labels_url":"https://api.github.com/repos/elastic/elasticsearch/issues/36928/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/issues/36928/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch/issues/36928/events","html_url":"https://github.com/elastic/elasticsearch/issues/36928","id":393366037,"node_id":"MDU6SXNzdWUzOTMzNjYwMzc=","number":36928,"title":"[CI] Rolling upgrade tests (twoThirdsUpgradedTestRunner) fails to form cluster","user":{"login":"dliappis","id":1754575,"node_id":"MDQ6VXNlcjE3NTQ1NzU=","avatar_url":"https://avatars0.githubusercontent.com/u/1754575?v=4","gravatar_id":"","url":"https://api.github.com/users/dliappis","html_url":"https://github.com/dliappis","followers_url":"https://api.github.com/users/dliappis/followers","following_url":"https://api.github.com/users/dliappis/following{/other_user}","gists_url":"https://api.github.com/users/dliappis/gists{/gist_id}","starred_url":"https://api.github.com/users/dliappis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dliappis/subscriptions","organizations_url":"https://api.github.com/users/dliappis/orgs","repos_url":"https://api.github.com/users/dliappis/repos","events_url":"https://api.github.com/users/dliappis/events{/privacy}","received_events_url":"https://api.github.com/users/dliappis/received_events","type":"User","site_admin":false},"labels":[{"id":160579278,"node_id":"MDU6TGFiZWwxNjA1NzkyNzg=","url":"https://api.github.com/repos/elastic/elasticsearch/labels/:Distributed/Discovery-Plugins","name":":Distributed/Discovery-Plugins","color":"0e8a16","default":false},{"id":148612629,"node_id":"MDU6TGFiZWwxNDg2MTI2Mjk=","url":"https://api.github.com/repos/elastic/elasticsearch/labels/%3Etest-failure","name":">test-failure","color":"207de5","default":false}],"state":"open","locked":false,"assignee":{"login":"original-brownbear","id":6490959,"node_id":"MDQ6VXNlcjY0OTA5NTk=","avatar_url":"https://avatars0.githubusercontent.com/u/6490959?v=4","gravatar_id":"","url":"https://api.github.com/users/original-brownbear","html_url":"https://github.com/original-brownbear","followers_url":"https://api.github.com/users/original-brownbear/followers","following_url":"https://api.github.com/users/original-brownbear/following{/other_user}","gists_url":"https://api.github.com/users/original-brownbear/gists{/gist_id}","starred_url":"https://api.github.com/users/original-brownbear/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/original-brownbear/subscriptions","organizations_url":"https://api.github.com/users/original-brownbear/orgs","repos_url":"https://api.github.com/users/original-brownbear/repos","events_url":"https://api.github.com/users/original-brownbear/events{/privacy}","received_events_url":"https://api.github.com/users/original-brownbear/received_events","type":"User","site_admin":false},"assignees":[{"login":"original-brownbear","id":6490959,"node_id":"MDQ6VXNlcjY0OTA5NTk=","avatar_url":"https://avatars0.githubusercontent.com/u/6490959?v=4","gravatar_id":"","url":"https://api.github.com/users/original-brownbear","html_url":"https://github.com/original-brownbear","followers_url":"https://api.github.com/users/original-brownbear/followers","following_url":"https://api.github.com/users/original-brownbear/following{/other_user}","gists_url":"https://api.github.com/users/original-brownbear/gists{/gist_id}","starred_url":"https://api.github.com/users/original-brownbear/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/original-brownbear/subscriptions","organizations_url":"https://api.github.com/users/original-brownbear/orgs","repos_url":"https://api.github.com/users/original-brownbear/repos","events_url":"https://api.github.com/users/original-brownbear/events{/privacy}","received_events_url":"https://api.github.com/users/original-brownbear/received_events","type":"User","site_admin":false}],"milestone":null,"comments":4,"created_at":"2018-12-21T08:30:45Z","updated_at":"2018-12-21T16:16:16Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"Build: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.6+intake/24/console\r\n\r\nSimilar to the closed https://github.com/elastic/elasticsearch/issues/35456, but for the twoThirdsUpgradedTestRunner:\r\n\r\n```\r\n04:31:44 > Task :x-pack:qa:rolling-upgrade:with-system-key:v5.6.15#twoThirdsUpgradedTestRunner FAILED\r\n04:31:44 :x-pack:qa:rolling-upgrade:with-system-key:v5.6.15#twoThirdsUpgradedTestRunner (Thread[Execution worker for ':',5,main]) completed. Took 2 mins 49.999 secs.\r\n04:31:44 :x-pack:qa:rolling-upgrade:with-system-key:v5.6.15#oldClusterTestCluster#node2.stop (Thread[Execution worker for ':',5,main]) started.\r\n04:31:44 \r\n```\r\n\r\nI can reproduce this locally on my workstation using:\r\n\r\n```\r\n./gradlew :x-pack:qa:rolling-upgrade:with-system-key:v5.6.15#twoThirdsUpgradedTestRunner -Dtests.seed=679945B82D7ABBD4 -Dtests.class=org.elasticsearch.upgrades.MlMigrationIT -Dtests.method=\"testConfigMigration\" -Dtests.security.manager=true -Dtests.locale=it -Dtests.timezone=Pacific/Ponape -Dcompiler.java=11 -Druntime.java=8\r\n```\r\n\r\nbut in my reproduction the exact failure is:\r\n\r\n```\r\nTask :x-pack:qa:rolling-upgrade:with-system-key:v5.6.15#twoThirdsUpgradedTestCluster#wait FAILED\r\n```\r\n\r\nThe task appears to fail because a cluster doesn't form and the logs (e.g. on my workstation) show:\r\n\r\n```\r\n|    [2018-12-21T10:00:16,272][WARN ][o.e.d.z.FileBasedUnicastHostsProvider] [upgraded-node-1] expected, but did not find, a dynamic hosts list at [/home/dl/source/elastic/elasticsearch/x-pack/qa/rolling-upgrade/with-system-key/build/cluster/v5.6.15#twoThirdsUpgradedTestCluster node0/elasticsearch-6.6.0-SNAPSHOT/config/unicast_hosts.txt]\r\n...\r\n[2018-12-21T10:00:16,665][DEBUG][o.e.a.a.c.h.TransportClusterHealthAction] [upgraded-node-1] no known master node, scheduling a retry\r\n```\r\n\r\nThe particular file those exists and has the correct list of IPv6 addresses (even though there's no newline at the last line):\r\n\r\n```\r\ncat /home/dl/source/elastic/elasticsearch/x-pack/qa/rolling-upgrade/with-system-key/build/cluster/v5.6.15#twoThirdsUpgradedTestCluster\\ node0/elasticsearch-6.6.0-SNAPSHOT/config/unicast_hosts.txt\r\n[::1]:36463\r\n[::1]:44889\r\n[::1]:45733\r\n```\r\n\r\nThis has been failing occasionally in the past: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.5+intake/216/console\r\n"},"comment":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/comments/449430585","html_url":"https://github.com/elastic/elasticsearch/issues/36928#issuecomment-449430585","issue_url":"https://api.github.com/repos/elastic/elasticsearch/issues/36928","id":449430585,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0OTQzMDU4NQ==","user":{"login":"original-brownbear","id":6490959,"node_id":"MDQ6VXNlcjY0OTA5NTk=","avatar_url":"https://avatars0.githubusercontent.com/u/6490959?v=4","gravatar_id":"","url":"https://api.github.com/users/original-brownbear","html_url":"https://github.com/original-brownbear","followers_url":"https://api.github.com/users/original-brownbear/followers","following_url":"https://api.github.com/users/original-brownbear/following{/other_user}","gists_url":"https://api.github.com/users/original-brownbear/gists{/gist_id}","starred_url":"https://api.github.com/users/original-brownbear/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/original-brownbear/subscriptions","organizations_url":"https://api.github.com/users/original-brownbear/orgs","repos_url":"https://api.github.com/users/original-brownbear/repos","events_url":"https://api.github.com/users/original-brownbear/events{/privacy}","received_events_url":"https://api.github.com/users/original-brownbear/received_events","type":"User","site_admin":false},"created_at":"2018-12-21T16:16:16Z","updated_at":"2018-12-21T16:16:16Z","author_association":"MEMBER","body":"@DaveCTurner \r\n\r\n> Maybe we can fix the REPRODUCE WITH line to say to run the same test on the fully-upgraded cluster?\r\n\r\nYea that would be a good approach imo.\r\n\r\n@atorok you're currently refactoring this logic anyway, wdyt?"}}
{"actor":{"display_login":"original-brownbear"}}