{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/beats/issues/9496","repository_url":"https://api.github.com/repos/elastic/beats","labels_url":"https://api.github.com/repos/elastic/beats/issues/9496/labels{/name}","comments_url":"https://api.github.com/repos/elastic/beats/issues/9496/comments","events_url":"https://api.github.com/repos/elastic/beats/issues/9496/events","html_url":"https://github.com/elastic/beats/issues/9496","id":389894744,"node_id":"MDU6SXNzdWUzODk4OTQ3NDQ=","number":9496,"title":"Selective testing in beats and test with latest version of the branch.","user":{"login":"ph","id":640,"node_id":"MDQ6VXNlcjY0MA==","avatar_url":"https://avatars0.githubusercontent.com/u/640?v=4","gravatar_id":"","url":"https://api.github.com/users/ph","html_url":"https://github.com/ph","followers_url":"https://api.github.com/users/ph/followers","following_url":"https://api.github.com/users/ph/following{/other_user}","gists_url":"https://api.github.com/users/ph/gists{/gist_id}","starred_url":"https://api.github.com/users/ph/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ph/subscriptions","organizations_url":"https://api.github.com/users/ph/orgs","repos_url":"https://api.github.com/users/ph/repos","events_url":"https://api.github.com/users/ph/events{/privacy}","received_events_url":"https://api.github.com/users/ph/received_events","type":"User","site_admin":false},"labels":[{"id":364269915,"node_id":"MDU6TGFiZWwzNjQyNjk5MTU=","url":"https://api.github.com/repos/elastic/beats/labels/:Testing","name":":Testing","color":"ffffff","default":false},{"id":227508070,"node_id":"MDU6TGFiZWwyMjc1MDgwNzA=","url":"https://api.github.com/repos/elastic/beats/labels/discuss","name":"discuss","color":"ffa500","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2018-12-11T18:26:24Z","updated_at":"2018-12-12T10:19:06Z","closed_at":null,"author_association":"MEMBER","body":"Beats is a mono repo of many projects and has a lot of tests to run. When a PR is created, our CI will run the tests for all the beats even if no code was changed in that project. This lead to longer test time and bigger CI queues, it not uncommon to have to wait more than one or two hours when we are close to the feature freeze.\r\n\r\nI would like to implement selective testings, what I mean by that is:\r\n\r\n1. A change of code in Filebeat should only run the test in Filebeat.\r\n2. A change in Auditbeat should only run the tests in Auditbeat\r\n3. A change in Metricbeat should trigger a tests in Metricbeat and Auditbeat.\r\n4. A change in libbeat should run the tests in all the beats.\r\n5. A new vendor should trigger the tests in dependant beats.\r\n6. A change in a module in Filebeat should only trigger the test_modules for that module.\r\n7. A change in a module in Metricbeat should only run the test for that module.\r\n8. When the docs are changed only run the doc tests.\r\n\r\nTo make the above work we probably need to either use a tool to get the dependency graph and generate what need to be executed or we could probably have a crude implementation purely based on path or maybe something that use both. \r\n\r\nThe other thing I would like to have is instead of only testing the PR code in the branch is to merge the PR of with the latest version of the target branch and then run the tests. When the PR tests are green and the PR is ready to merge it doesn't mean the branch will be green after the merge.\r\n\r\n\r\nThe above is based on discussion that @urso and myself had.\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/elastic/beats/issues/comments/446535787","html_url":"https://github.com/elastic/beats/issues/9496#issuecomment-446535787","issue_url":"https://api.github.com/repos/elastic/beats/issues/9496","id":446535787,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0NjUzNTc4Nw==","user":{"login":"kuisathaverat","id":5400788,"node_id":"MDQ6VXNlcjU0MDA3ODg=","avatar_url":"https://avatars1.githubusercontent.com/u/5400788?v=4","gravatar_id":"","url":"https://api.github.com/users/kuisathaverat","html_url":"https://github.com/kuisathaverat","followers_url":"https://api.github.com/users/kuisathaverat/followers","following_url":"https://api.github.com/users/kuisathaverat/following{/other_user}","gists_url":"https://api.github.com/users/kuisathaverat/gists{/gist_id}","starred_url":"https://api.github.com/users/kuisathaverat/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kuisathaverat/subscriptions","organizations_url":"https://api.github.com/users/kuisathaverat/orgs","repos_url":"https://api.github.com/users/kuisathaverat/repos","events_url":"https://api.github.com/users/kuisathaverat/events{/privacy}","received_events_url":"https://api.github.com/users/kuisathaverat/received_events","type":"User","site_admin":false},"created_at":"2018-12-12T10:19:06Z","updated_at":"2018-12-12T10:19:06Z","author_association":"NONE","body":"Indeed they are, as better granularity, you have to run tests isolated, it makes easier to run then in parallel or filter which test you want to run based on files, path, labels, comments, whatever you want to check before to trigger a test, e.g for the APM UI we have a very similar issue, everytime we push something to kibana we have to wait for all the kibana-ci test to know if everything is OK, it is fine to do that, but we need also for us a quick set of checks/tests that should be run on the CI and are faster than run the whole CI tests, so we will run on our apm-ci a job that will identify our PRs based on the changes made on files and trigger a build that makes our tests."}}
{"actor":{"display_login":"kuisathaverat"}}