{"action":"created","issue":{"url":"https://api.github.com/repos/nodejs/build/issues/248","labels_url":"https://api.github.com/repos/nodejs/build/issues/248/labels{/name}","comments_url":"https://api.github.com/repos/nodejs/build/issues/248/comments","events_url":"https://api.github.com/repos/nodejs/build/issues/248/events","html_url":"https://github.com/nodejs/build/issues/248","id":116236732,"number":248,"title":"Flaky tests","user":{"login":"jbergstroem","id":176984,"avatar_url":"https://avatars.githubusercontent.com/u/176984?v=3","gravatar_id":"","url":"https://api.github.com/users/jbergstroem","html_url":"https://github.com/jbergstroem","followers_url":"https://api.github.com/users/jbergstroem/followers","following_url":"https://api.github.com/users/jbergstroem/following{/other_user}","gists_url":"https://api.github.com/users/jbergstroem/gists{/gist_id}","starred_url":"https://api.github.com/users/jbergstroem/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jbergstroem/subscriptions","organizations_url":"https://api.github.com/users/jbergstroem/orgs","repos_url":"https://api.github.com/users/jbergstroem/repos","events_url":"https://api.github.com/users/jbergstroem/events{/privacy}","received_events_url":"https://api.github.com/users/jbergstroem/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":13,"created_at":"2015-11-11T00:49:43Z","updated_at":"2016-01-27T05:21:53Z","closed_at":"2016-01-27T05:07:02Z","body":"Before the nodejs merge we only has pass/fail. We had a discussion about this during the merge and the conclusion was to introduce the flaky test structure already found in nodejs so we could always have a passing [although flaky] test suite for releases.\r\n\r\nWe [the build group] just brought it up again since the tests has a tendency to rot. Also, reporting tests to github (see #236) will make it even less visible since a flaky pass is still a pass.\r\n\r\nWhat can we do to improve this? Do we add a policy for moving flaky tests to fail after a while? Removing them altogether? I know @trott has been doing a lot of work to improve the status of flaky tests (well, tests in general) -- perhaps you want to chip in?"},"comment":{"url":"https://api.github.com/repos/nodejs/build/issues/comments/175403527","html_url":"https://github.com/nodejs/build/issues/248#issuecomment-175403527","issue_url":"https://api.github.com/repos/nodejs/build/issues/248","id":175403527,"user":{"login":"Trott","id":718899,"avatar_url":"https://avatars.githubusercontent.com/u/718899?v=3","gravatar_id":"","url":"https://api.github.com/users/Trott","html_url":"https://github.com/Trott","followers_url":"https://api.github.com/users/Trott/followers","following_url":"https://api.github.com/users/Trott/following{/other_user}","gists_url":"https://api.github.com/users/Trott/gists{/gist_id}","starred_url":"https://api.github.com/users/Trott/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Trott/subscriptions","organizations_url":"https://api.github.com/users/Trott/orgs","repos_url":"https://api.github.com/users/Trott/repos","events_url":"https://api.github.com/users/Trott/events{/privacy}","received_events_url":"https://api.github.com/users/Trott/received_events","type":"User","site_admin":false},"created_at":"2016-01-27T05:21:53Z","updated_at":"2016-01-27T05:21:53Z","body":"@jbergstroem wrote:\r\n\r\n> For an 'outsider' it might be good to define the scope of the build and testing workgroup somewhere (where do I post what problem).\r\n\r\nThe Testing WG is putting together docs so we can be chartered by the TSC. Because some of our charter is likely to overlap with the existing Build WG charter, the Build WG will probably have to ratify our charter as well.\r\n\r\nIt would be great if as many build folks as were willing would read the very short draft docs for the Testing WG and comment. You can find them here: https://github.com/nodejs/testing/pull/2/files"}}