NodeJS Code Coverage of Automated Tests <p>As part of a custom testing framework for a NodeJS REST API, I'd like to automatically detect when my tests are no longer providing proper coverage by comparing all <em>possible</em> outcomes with what the test suite received. </p> <p>What methods exist for doing this? We can assume that it's being used for a REST API with a list of entry functions (API endpoints) that need coverage analysis, and each entry function will end with a known 'exit function' that responds to the requester in a standard way. </p> <p>Here's what I've found so far:</p> <p><strong>1: Basic solution (Currently implemented)</strong></p> <ul> <li>When writing each REST endpoint, manually create a list of all the possible outcome 'codes' [Success, FailureDueToX, FailureDueToY] for example</li> <li>After the tests have run, ensure every code in the list has been seen by the test suite for each endpoint. </li> </ul> <p><em><strong>Pros:</strong></em> Very basic and easy to use; Doesn't change performance testing times</p> <p><em><strong>Cons:</strong></em> Highly prone to error with lots of manual checking; Doesn't flag any issues if there are 5 ways to 'FailDueToX' and you only test 1 of them. Very basic definition of 'coverage'</p> <p><strong>2: Static analysis</strong></p> <ul> <li>Parse the code into some sort of parse tree and look for all the instances of the 'exit function'</li> <li>Traverse up the tree until an API Endpoint is reached, and add that exit instance to the endpoint as an expected output (needs to keep a record of the stack trace to get there via a hash or similar)</li> <li>When tests are run, the endpoints return the stack trace hash or similar and this is compared to the expected output list.</li> </ul> <p><em><strong>Pros:</strong></em> Automatic; Catches the different branches that may result in the same output code</p> <p><em><strong>Cons:</strong></em> Generating the parse tree is non trivial; Doesn't detect dead code that never gets run; test suite needs to be kept in sync</p> <p><strong>3: Profiling</strong></p> <p>I've done this in the past on Embedded Systems with <a href="http://www.ghs.com/products/safety_critical/gcover.html" rel="nofollow">GreenHills Code Coverage Tools</a></p> <ul> <li>Start up a profiler like <a href="http://blog.nodejs.org/2012/04/25/profiling-node-js/" rel="nofollow">dtrace</a> and record the stack log for each test separately</li> <li>Parse the stack log and assign the 'test' to each line of code</li> <li>Manually analyse the code-with-annotations to find gaps.</li> </ul> <p><em><strong>Pros:</strong></em> Semi Automatic; Gives more information about total coverage to a developer; Can see </p> <p><em><strong>Cons:</strong></em> Slows down the tests; Unable to do performance testing in parallel; Doesn't flag when a possible outcome is never made to happen. </p> <p>What else is there, and what tools can help me with my Static Analysis and Profiling goals? </p>