{"action":"created","issue":{"url":"https://api.github.com/repos/stryker-mutator/stryker-api/issues/4","repository_url":"https://api.github.com/repos/stryker-mutator/stryker-api","labels_url":"https://api.github.com/repos/stryker-mutator/stryker-api/issues/4/labels{/name}","comments_url":"https://api.github.com/repos/stryker-mutator/stryker-api/issues/4/comments","events_url":"https://api.github.com/repos/stryker-mutator/stryker-api/issues/4/events","html_url":"https://github.com/stryker-mutator/stryker-api/issues/4","id":168196226,"number":4,"title":"A better test runner API","user":{"login":"nicojs","id":1828233,"avatar_url":"https://avatars.githubusercontent.com/u/1828233?v=3","gravatar_id":"","url":"https://api.github.com/users/nicojs","html_url":"https://github.com/nicojs","followers_url":"https://api.github.com/users/nicojs/followers","following_url":"https://api.github.com/users/nicojs/following{/other_user}","gists_url":"https://api.github.com/users/nicojs/gists{/gist_id}","starred_url":"https://api.github.com/users/nicojs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nicojs/subscriptions","organizations_url":"https://api.github.com/users/nicojs/orgs","repos_url":"https://api.github.com/users/nicojs/repos","events_url":"https://api.github.com/users/nicojs/events{/privacy}","received_events_url":"https://api.github.com/users/nicojs/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/stryker-mutator/stryker-api/labels/discussion","name":"discussion","color":"B74934"},{"url":"https://api.github.com/repos/stryker-mutator/stryker-api/labels/help%20wanted","name":"help wanted","color":"128A0C"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2016-07-28T21:03:21Z","updated_at":"2016-08-03T15:45:52Z","closed_at":null,"body":"I want to discus a better test runner API. Right now it looks like this (removed jsdoc for clarity):\r\n\r\n```javascript\r\n // Omitted some details for clarity\r\n\r\nnamespace TestRunnerFactory {\r\n  class TestRunnerFactory extends Factory<RunnerOptions, TestRunner> {\r\n  }\r\n  let testRunnerFactoryInstance = new TestRunnerFactory();\r\n  export function instance() {\r\n    return <Factory<RunnerOptions, TestRunner>>testRunnerFactoryInstance;\r\n  }\r\n}\r\n\r\ninterface TestRunner {\r\n  run(options: RunOptions): Promise<RunResult>;\r\n}\r\n\r\ninterface RunResult {\r\n  result: TestResult;\r\n  testNames?: string[];\r\n  succeeded?: number;\r\n  failed?: number;\r\n  timeSpent?: number;\r\n  coverage?: CoverageCollection;\r\n  errorMessages?: string[];\r\n}\r\n\r\nenum TestResult {\r\n  Complete,\r\n  Error,\r\n  Timeout\r\n}\r\n```\r\n\r\nI have a few problems with this. \r\n\r\n#### 1. We cannot see which tests failed and which tests passed.\r\n\r\nWhen we have failed tests, we cannot see which of them failed and which of them passed. We also cannot see which error message belonged to which test. We could solve this by creating a new entity to hold the result of one single test. Something like this:\r\n\r\n```typescript\r\ninterface RunResult {\r\n  testResults: TestResult[];\r\n  timeSpent: number;\r\n  coverage?: CoverageCollection;\r\n}\r\n\r\ninterface TestResult{\r\n  name: string;\r\n  state: TestState;\r\n  errorMessages?: string[];\r\n}\r\n\r\nenum TestState { Complete, Error, Timeout }\r\n```\r\n\r\nThis can work for karma and mocha, which both know which tests are being ran. Which brings me to my next point.\r\n\r\n#### 2. Implementing a test runner is kind of difficult.\r\n\r\nWhen implementing a test runner you have to provide a lot of information. You need to report code coverage (it will work without it, but there is know way to communicate this to Stryker in advance), you need to report on the tests being ran and the state of each test with possible error. This is too bad, if you for example don't use an out-of-the-box test runner at all, but your custom testing logic.\r\n\r\nWhat would the best course of action be here? We could change the interface of the `TestRunnerFactory` , in that it requires a static method `capabilities(): TestCapabilities` on each test runner. The test capabilities could look like this:\r\n\r\n```typescript\r\ninterface TestCapabilities {\r\n  codeCoverage?: boolean;\r\n  identitfyTests?: boolean;\r\n}\r\n```\r\nStryker could then use this interface to determine the best strategy to run mutations.\r\n\r\nThis brings me to my third point:\r\n\r\n#### 3. Implementing code coverage reporting is a lot of work.\r\n\r\nImplementing code coverage reporting is a lot of work. Just think about it. A test runner needs to know which code it must instrument for code coverage (the once with `mutated: true`), then it needs to instrument the code for code coverage and it has to do that with istanbul (or transform the code coverage report of the coverage tool in question). Finally, it needs to report the code coverage object. \r\n\r\n Could we find a better way to implement code coverage? Maybe let Stryker instrument the code and then leave it up to the test runner to report the code coverage report (and reset it for next run). Maybe not ideal, as the code coverage responsibility gets filled by a tandem of Stryker and the test runner.\r\n\r\nCan anyone think of a solution?"},"comment":{"url":"https://api.github.com/repos/stryker-mutator/stryker-api/issues/comments/237277706","html_url":"https://github.com/stryker-mutator/stryker-api/issues/4#issuecomment-237277706","issue_url":"https://api.github.com/repos/stryker-mutator/stryker-api/issues/4","id":237277706,"user":{"login":"nicojs","id":1828233,"avatar_url":"https://avatars.githubusercontent.com/u/1828233?v=3","gravatar_id":"","url":"https://api.github.com/users/nicojs","html_url":"https://github.com/nicojs","followers_url":"https://api.github.com/users/nicojs/followers","following_url":"https://api.github.com/users/nicojs/following{/other_user}","gists_url":"https://api.github.com/users/nicojs/gists{/gist_id}","starred_url":"https://api.github.com/users/nicojs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nicojs/subscriptions","organizations_url":"https://api.github.com/users/nicojs/orgs","repos_url":"https://api.github.com/users/nicojs/repos","events_url":"https://api.github.com/users/nicojs/events{/privacy}","received_events_url":"https://api.github.com/users/nicojs/received_events","type":"User","site_admin":false},"created_at":"2016-08-03T15:45:52Z","updated_at":"2016-08-03T15:45:52Z","body":"I did some more research on Istanbul and i think we can fairly simple to use the more low-level apis and let stryker be responsible for code coverage.\r\n\r\nFor example:\r\n\r\n```typescript\r\nimport {Instrumenter, Collector} from 'istanbul';\r\nimport * as fs from 'fs';\r\n\r\nlet instrumenter = new Instrumenter({coverageVariable: '__cov__'}); // default\r\nlet collector = new Collector();\r\n\r\nlet instrumented = instrumenter.instrumentSync('function mesaningOfLife() { return 42; }' /*This can even be the AST, this will save some time*/, 'filename.js');\r\nfs.writeFileSync('filename.js', instrumented);\r\nrequire('./filename'); // load the newly created file\r\ncollector.add(__cov__); // `__cov__` now magically exists\r\nconsole.log(collector.getFinalCoverage())\r\n```\r\n\r\nAs you can see, coverage consists of 2 steps. Instrument and Collect. Instrumentation can be done by any process, however the collection *only* works when the tests are ran in the same node process. This means that we can only collect the report inside child process (`IsolatedTestRunnerAdapterRunner`) and we can than only use it for the mocha runner (and other node-based runners). For the karma runner and other browser based runners we could still instrument the code, but the runner should in that case be responsible for retrieving the `__cov__` global object.\r\n\r\nIn other words, i think the test capabilities interface should look like:\r\n\r\n```typescript\r\ninterface TestCapabilities {\r\n  codeCoverageResponsibility?: CodeCoverageResponsibilityKind; // if empty, code coverage is disabled completely\r\n}\r\n\r\nenum CodeCoverageResponsibilityKind {\r\n  runner, // Test runner has full responsibility, stryker does nothing. Like the way stryker-karma-runner works now\r\n  stryker, // Test runner does nothing, stryker takes full responsibility. Should work for the stryker-mocha-runner\r\n  tandem // Stryker instruments the code and the test-runner collects it. This could in theory make the stryker-karma-runner a lot faster as it does not have to re-instrument the code every time. Maybe we'll implement this in the future, not right away. \r\n}\r\n```\r\n@simondel Do you (1) understand and (2) agree with this reasoning?"}}
{"actor":{"display_login":"nicojs"}}