{"action":"closed","number":9251,"pull_request":{"url":"https://api.github.com/repos/elastic/kibana/pulls/9251","id":95749859,"html_url":"https://github.com/elastic/kibana/pull/9251","diff_url":"https://github.com/elastic/kibana/pull/9251.diff","patch_url":"https://github.com/elastic/kibana/pull/9251.patch","issue_url":"https://api.github.com/repos/elastic/kibana/issues/9251","number":9251,"state":"closed","locked":false,"title":"Handle test failures better","user":{"login":"thomasneirynck","id":1833023,"avatar_url":"https://avatars.githubusercontent.com/u/1833023?v=3","gravatar_id":"","url":"https://api.github.com/users/thomasneirynck","html_url":"https://github.com/thomasneirynck","followers_url":"https://api.github.com/users/thomasneirynck/followers","following_url":"https://api.github.com/users/thomasneirynck/following{/other_user}","gists_url":"https://api.github.com/users/thomasneirynck/gists{/gist_id}","starred_url":"https://api.github.com/users/thomasneirynck/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thomasneirynck/subscriptions","organizations_url":"https://api.github.com/users/thomasneirynck/orgs","repos_url":"https://api.github.com/users/thomasneirynck/repos","events_url":"https://api.github.com/users/thomasneirynck/events{/privacy}","received_events_url":"https://api.github.com/users/thomasneirynck/received_events","type":"User","site_admin":false},"body":"When a test used to fail, the error used to be swallowed and the done() method would not be called. This caused the entire suite to fail with time-out errors, when a single test failed, because the other tests would not be triggered to run. Resolved this now by explicitly trapping the errors and calling done().\r\n\r\nAlso improved the error messages for individual assertions.\r\n\r\nThis may not fix the intermittent errors in CI, but\r\n- it will give us a more meaningful error message when a test failure does happen\r\n- a single failing test will not bring down the entire test suite with timeout errors\r\n\r\n----------\r\nUPDATE:\r\n\r\nAlong with tests against 5.1 (#9254) and 5.x (#9255), the tests fail about 1/5 times. They always fail with an identical issue across all branches: one tag is placed at the center, while two other tags are moved away far outside the viewport. The 'wrong' positions are always identical for each test-stub. So the test fails \"correctly\" on an assertion, and not due to an unexpected error.\r\n\r\nhttps://github.com/elastic/kibana/pull/9251/commits/b79ec6048b1593ac7fdb4cd4d2f2c48e29eda986: this reorganizes the test suite. It makes all tests synchronous, isolating each assertion into its own stub, and moves all bootstrapping of the test into the before-handler. The advantage is that we get better labeling of the tests and their failures, and perhaps we get more consistent behavior. The reasoning for this is fairly thin: if there is latency in the stub, it is now moved to the bootstrapping phase, which may have less of an impact. That said, the reorganization also helps readability because the assertions are no longer nested in callback hell. (@spalger suggested further simplifcations)\r\n\r\n7bb150f48becbbf2ccd0cedeb71d0a99fc4cce8b: removes the time limit for a single cloud-layout on each tick. I don't think this is necessary because we truncate the cloud already to have a manageable number of tags on screen. However, perhaps because of this the layout does not complete in time, causing cloud to be stuck mid-way.\r\n\r\n---\r\nUPDATE 2:\r\n\r\nhttps://github.com/elastic/kibana/pull/9251/commits/f6463b86ce11786ff8960ae0d1d572ba0049913f: Skips the tests when they fail in the particular way outlined above. Failures for any other reason will continue to propagate correctly. This will keep the tests from bringing down the entire test-suite in CI. In the long term, we will need to address a different way, by identifying why this random failure occurs.\r\n \r\nThe diff of the test-file looks rather scary. Note that the content has not changed, but right now it uses a cleaner syntax (async/await instead of promises + done-callback).\r\n\r\n","created_at":"2016-11-29T18:36:55Z","updated_at":"2016-12-01T23:18:43Z","closed_at":"2016-12-01T23:18:43Z","merged_at":"2016-12-01T23:18:43Z","merge_commit_sha":"8569bd1fce130b294b1a101fdd9a74f808194f10","assignee":{"login":"spalger","id":1329312,"avatar_url":"https://avatars.githubusercontent.com/u/1329312?v=3","gravatar_id":"","url":"https://api.github.com/users/spalger","html_url":"https://github.com/spalger","followers_url":"https://api.github.com/users/spalger/followers","following_url":"https://api.github.com/users/spalger/following{/other_user}","gists_url":"https://api.github.com/users/spalger/gists{/gist_id}","starred_url":"https://api.github.com/users/spalger/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/spalger/subscriptions","organizations_url":"https://api.github.com/users/spalger/orgs","repos_url":"https://api.github.com/users/spalger/repos","events_url":"https://api.github.com/users/spalger/events{/privacy}","received_events_url":"https://api.github.com/users/spalger/received_events","type":"User","site_admin":false},"assignees":[{"login":"spalger","id":1329312,"avatar_url":"https://avatars.githubusercontent.com/u/1329312?v=3","gravatar_id":"","url":"https://api.github.com/users/spalger","html_url":"https://github.com/spalger","followers_url":"https://api.github.com/users/spalger/followers","following_url":"https://api.github.com/users/spalger/following{/other_user}","gists_url":"https://api.github.com/users/spalger/gists{/gist_id}","starred_url":"https://api.github.com/users/spalger/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/spalger/subscriptions","organizations_url":"https://api.github.com/users/spalger/orgs","repos_url":"https://api.github.com/users/spalger/repos","events_url":"https://api.github.com/users/spalger/events{/privacy}","received_events_url":"https://api.github.com/users/spalger/received_events","type":"User","site_admin":false},{"login":"jbudz","id":3143860,"avatar_url":"https://avatars.githubusercontent.com/u/3143860?v=3","gravatar_id":"","url":"https://api.github.com/users/jbudz","html_url":"https://github.com/jbudz","followers_url":"https://api.github.com/users/jbudz/followers","following_url":"https://api.github.com/users/jbudz/following{/other_user}","gists_url":"https://api.github.com/users/jbudz/gists{/gist_id}","starred_url":"https://api.github.com/users/jbudz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jbudz/subscriptions","organizations_url":"https://api.github.com/users/jbudz/orgs","repos_url":"https://api.github.com/users/jbudz/repos","events_url":"https://api.github.com/users/jbudz/events{/privacy}","received_events_url":"https://api.github.com/users/jbudz/received_events","type":"User","site_admin":false},{"login":"ppisljar","id":13629809,"avatar_url":"https://avatars.githubusercontent.com/u/13629809?v=3","gravatar_id":"","url":"https://api.github.com/users/ppisljar","html_url":"https://github.com/ppisljar","followers_url":"https://api.github.com/users/ppisljar/followers","following_url":"https://api.github.com/users/ppisljar/following{/other_user}","gists_url":"https://api.github.com/users/ppisljar/gists{/gist_id}","starred_url":"https://api.github.com/users/ppisljar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ppisljar/subscriptions","organizations_url":"https://api.github.com/users/ppisljar/orgs","repos_url":"https://api.github.com/users/ppisljar/repos","events_url":"https://api.github.com/users/ppisljar/events{/privacy}","received_events_url":"https://api.github.com/users/ppisljar/received_events","type":"User","site_admin":false}],"milestone":null,"commits_url":"https://api.github.com/repos/elastic/kibana/pulls/9251/commits","review_comments_url":"https://api.github.com/repos/elastic/kibana/pulls/9251/comments","review_comment_url":"https://api.github.com/repos/elastic/kibana/pulls/comments{/number}","comments_url":"https://api.github.com/repos/elastic/kibana/issues/9251/comments","statuses_url":"https://api.github.com/repos/elastic/kibana/statuses/d5b6879eb55be7e3b7b9b5a74e4d6bcb849addee","head":{"label":"thomasneirynck:tagcloud-handle-test-failure","ref":"tagcloud-handle-test-failure","sha":"d5b6879eb55be7e3b7b9b5a74e4d6bcb849addee","user":{"login":"thomasneirynck","id":1833023,"avatar_url":"https://avatars.githubusercontent.com/u/1833023?v=3","gravatar_id":"","url":"https://api.github.com/users/thomasneirynck","html_url":"https://github.com/thomasneirynck","followers_url":"https://api.github.com/users/thomasneirynck/followers","following_url":"https://api.github.com/users/thomasneirynck/following{/other_user}","gists_url":"https://api.github.com/users/thomasneirynck/gists{/gist_id}","starred_url":"https://api.github.com/users/thomasneirynck/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thomasneirynck/subscriptions","organizations_url":"https://api.github.com/users/thomasneirynck/orgs","repos_url":"https://api.github.com/users/thomasneirynck/repos","events_url":"https://api.github.com/users/thomasneirynck/events{/privacy}","received_events_url":"https://api.github.com/users/thomasneirynck/received_events","type":"User","site_admin":false},"repo":{"id":63882657,"name":"kibana","full_name":"thomasneirynck/kibana","owner":{"login":"thomasneirynck","id":1833023,"avatar_url":"https://avatars.githubusercontent.com/u/1833023?v=3","gravatar_id":"","url":"https://api.github.com/users/thomasneirynck","html_url":"https://github.com/thomasneirynck","followers_url":"https://api.github.com/users/thomasneirynck/followers","following_url":"https://api.github.com/users/thomasneirynck/following{/other_user}","gists_url":"https://api.github.com/users/thomasneirynck/gists{/gist_id}","starred_url":"https://api.github.com/users/thomasneirynck/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thomasneirynck/subscriptions","organizations_url":"https://api.github.com/users/thomasneirynck/orgs","repos_url":"https://api.github.com/users/thomasneirynck/repos","events_url":"https://api.github.com/users/thomasneirynck/events{/privacy}","received_events_url":"https://api.github.com/users/thomasneirynck/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/thomasneirynck/kibana","description":":bar_chart: Kibana analytics and search dashboard for Elasticsearch","fork":true,"url":"https://api.github.com/repos/thomasneirynck/kibana","forks_url":"https://api.github.com/repos/thomasneirynck/kibana/forks","keys_url":"https://api.github.com/repos/thomasneirynck/kibana/keys{/key_id}","collaborators_url":"https://api.github.com/repos/thomasneirynck/kibana/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/thomasneirynck/kibana/teams","hooks_url":"https://api.github.com/repos/thomasneirynck/kibana/hooks","issue_events_url":"https://api.github.com/repos/thomasneirynck/kibana/issues/events{/number}","events_url":"https://api.github.com/repos/thomasneirynck/kibana/events","assignees_url":"https://api.github.com/repos/thomasneirynck/kibana/assignees{/user}","branches_url":"https://api.github.com/repos/thomasneirynck/kibana/branches{/branch}","tags_url":"https://api.github.com/repos/thomasneirynck/kibana/tags","blobs_url":"https://api.github.com/repos/thomasneirynck/kibana/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/thomasneirynck/kibana/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/thomasneirynck/kibana/git/refs{/sha}","trees_url":"https://api.github.com/repos/thomasneirynck/kibana/git/trees{/sha}","statuses_url":"https://api.github.com/repos/thomasneirynck/kibana/statuses/{sha}","languages_url":"https://api.github.com/repos/thomasneirynck/kibana/languages","stargazers_url":"https://api.github.com/repos/thomasneirynck/kibana/stargazers","contributors_url":"https://api.github.com/repos/thomasneirynck/kibana/contributors","subscribers_url":"https://api.github.com/repos/thomasneirynck/kibana/subscribers","subscription_url":"https://api.github.com/repos/thomasneirynck/kibana/subscription","commits_url":"https://api.github.com/repos/thomasneirynck/kibana/commits{/sha}","git_commits_url":"https://api.github.com/repos/thomasneirynck/kibana/git/commits{/sha}","comments_url":"https://api.github.com/repos/thomasneirynck/kibana/comments{/number}","issue_comment_url":"https://api.github.com/repos/thomasneirynck/kibana/issues/comments{/number}","contents_url":"https://api.github.com/repos/thomasneirynck/kibana/contents/{+path}","compare_url":"https://api.github.com/repos/thomasneirynck/kibana/compare/{base}...{head}","merges_url":"https://api.github.com/repos/thomasneirynck/kibana/merges","archive_url":"https://api.github.com/repos/thomasneirynck/kibana/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/thomasneirynck/kibana/downloads","issues_url":"https://api.github.com/repos/thomasneirynck/kibana/issues{/number}","pulls_url":"https://api.github.com/repos/thomasneirynck/kibana/pulls{/number}","milestones_url":"https://api.github.com/repos/thomasneirynck/kibana/milestones{/number}","notifications_url":"https://api.github.com/repos/thomasneirynck/kibana/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/thomasneirynck/kibana/labels{/name}","releases_url":"https://api.github.com/repos/thomasneirynck/kibana/releases{/id}","deployments_url":"https://api.github.com/repos/thomasneirynck/kibana/deployments","created_at":"2016-07-21T15:51:25Z","updated_at":"2016-07-21T15:51:31Z","pushed_at":"2016-12-01T23:05:25Z","git_url":"git://github.com/thomasneirynck/kibana.git","ssh_url":"git@github.com:thomasneirynck/kibana.git","clone_url":"https://github.com/thomasneirynck/kibana.git","svn_url":"https://github.com/thomasneirynck/kibana","homepage":"https://www.elastic.co/products/kibana","size":223639,"stargazers_count":0,"watchers_count":0,"language":"JavaScript","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"elastic:master","ref":"master","sha":"c989ee152fc216c85a9b8902a72d951aee3a3479","user":{"login":"elastic","id":6764390,"avatar_url":"https://avatars.githubusercontent.com/u/6764390?v=3","gravatar_id":"","url":"https://api.github.com/users/elastic","html_url":"https://github.com/elastic","followers_url":"https://api.github.com/users/elastic/followers","following_url":"https://api.github.com/users/elastic/following{/other_user}","gists_url":"https://api.github.com/users/elastic/gists{/gist_id}","starred_url":"https://api.github.com/users/elastic/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/elastic/subscriptions","organizations_url":"https://api.github.com/users/elastic/orgs","repos_url":"https://api.github.com/users/elastic/repos","events_url":"https://api.github.com/users/elastic/events{/privacy}","received_events_url":"https://api.github.com/users/elastic/received_events","type":"Organization","site_admin":false},"repo":{"id":7833168,"name":"kibana","full_name":"elastic/kibana","owner":{"login":"elastic","id":6764390,"avatar_url":"https://avatars.githubusercontent.com/u/6764390?v=3","gravatar_id":"","url":"https://api.github.com/users/elastic","html_url":"https://github.com/elastic","followers_url":"https://api.github.com/users/elastic/followers","following_url":"https://api.github.com/users/elastic/following{/other_user}","gists_url":"https://api.github.com/users/elastic/gists{/gist_id}","starred_url":"https://api.github.com/users/elastic/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/elastic/subscriptions","organizations_url":"https://api.github.com/users/elastic/orgs","repos_url":"https://api.github.com/users/elastic/repos","events_url":"https://api.github.com/users/elastic/events{/privacy}","received_events_url":"https://api.github.com/users/elastic/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/elastic/kibana","description":":bar_chart: Kibana analytics and search dashboard for Elasticsearch","fork":false,"url":"https://api.github.com/repos/elastic/kibana","forks_url":"https://api.github.com/repos/elastic/kibana/forks","keys_url":"https://api.github.com/repos/elastic/kibana/keys{/key_id}","collaborators_url":"https://api.github.com/repos/elastic/kibana/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/elastic/kibana/teams","hooks_url":"https://api.github.com/repos/elastic/kibana/hooks","issue_events_url":"https://api.github.com/repos/elastic/kibana/issues/events{/number}","events_url":"https://api.github.com/repos/elastic/kibana/events","assignees_url":"https://api.github.com/repos/elastic/kibana/assignees{/user}","branches_url":"https://api.github.com/repos/elastic/kibana/branches{/branch}","tags_url":"https://api.github.com/repos/elastic/kibana/tags","blobs_url":"https://api.github.com/repos/elastic/kibana/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/elastic/kibana/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/elastic/kibana/git/refs{/sha}","trees_url":"https://api.github.com/repos/elastic/kibana/git/trees{/sha}","statuses_url":"https://api.github.com/repos/elastic/kibana/statuses/{sha}","languages_url":"https://api.github.com/repos/elastic/kibana/languages","stargazers_url":"https://api.github.com/repos/elastic/kibana/stargazers","contributors_url":"https://api.github.com/repos/elastic/kibana/contributors","subscribers_url":"https://api.github.com/repos/elastic/kibana/subscribers","subscription_url":"https://api.github.com/repos/elastic/kibana/subscription","commits_url":"https://api.github.com/repos/elastic/kibana/commits{/sha}","git_commits_url":"https://api.github.com/repos/elastic/kibana/git/commits{/sha}","comments_url":"https://api.github.com/repos/elastic/kibana/comments{/number}","issue_comment_url":"https://api.github.com/repos/elastic/kibana/issues/comments{/number}","contents_url":"https://api.github.com/repos/elastic/kibana/contents/{+path}","compare_url":"https://api.github.com/repos/elastic/kibana/compare/{base}...{head}","merges_url":"https://api.github.com/repos/elastic/kibana/merges","archive_url":"https://api.github.com/repos/elastic/kibana/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/elastic/kibana/downloads","issues_url":"https://api.github.com/repos/elastic/kibana/issues{/number}","pulls_url":"https://api.github.com/repos/elastic/kibana/pulls{/number}","milestones_url":"https://api.github.com/repos/elastic/kibana/milestones{/number}","notifications_url":"https://api.github.com/repos/elastic/kibana/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/elastic/kibana/labels{/name}","releases_url":"https://api.github.com/repos/elastic/kibana/releases{/id}","deployments_url":"https://api.github.com/repos/elastic/kibana/deployments","created_at":"2013-01-26T04:00:59Z","updated_at":"2016-12-01T17:08:13Z","pushed_at":"2016-12-01T23:18:42Z","git_url":"git://github.com/elastic/kibana.git","ssh_url":"git@github.com:elastic/kibana.git","clone_url":"https://github.com/elastic/kibana.git","svn_url":"https://github.com/elastic/kibana","homepage":"https://www.elastic.co/products/kibana","size":272429,"stargazers_count":6408,"watchers_count":6408,"language":"JavaScript","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":2359,"mirror_url":null,"open_issues_count":1339,"forks":2359,"open_issues":1339,"watchers":6408,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/elastic/kibana/pulls/9251"},"html":{"href":"https://github.com/elastic/kibana/pull/9251"},"issue":{"href":"https://api.github.com/repos/elastic/kibana/issues/9251"},"comments":{"href":"https://api.github.com/repos/elastic/kibana/issues/9251/comments"},"review_comments":{"href":"https://api.github.com/repos/elastic/kibana/pulls/9251/comments"},"review_comment":{"href":"https://api.github.com/repos/elastic/kibana/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/elastic/kibana/pulls/9251/commits"},"statuses":{"href":"https://api.github.com/repos/elastic/kibana/statuses/d5b6879eb55be7e3b7b9b5a74e4d6bcb849addee"}},"merged":true,"mergeable":null,"mergeable_state":"unknown","merged_by":{"login":"thomasneirynck","id":1833023,"avatar_url":"https://avatars.githubusercontent.com/u/1833023?v=3","gravatar_id":"","url":"https://api.github.com/users/thomasneirynck","html_url":"https://github.com/thomasneirynck","followers_url":"https://api.github.com/users/thomasneirynck/followers","following_url":"https://api.github.com/users/thomasneirynck/following{/other_user}","gists_url":"https://api.github.com/users/thomasneirynck/gists{/gist_id}","starred_url":"https://api.github.com/users/thomasneirynck/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thomasneirynck/subscriptions","organizations_url":"https://api.github.com/users/thomasneirynck/orgs","repos_url":"https://api.github.com/users/thomasneirynck/repos","events_url":"https://api.github.com/users/thomasneirynck/events{/privacy}","received_events_url":"https://api.github.com/users/thomasneirynck/received_events","type":"User","site_admin":false},"comments":36,"review_comments":4,"commits":1,"additions":309,"deletions":167,"changed_files":2}}
{"actor":{"display_login":"thomasneirynck"}}