title: Python test discovery with doctests, coverage and parallelism
body: <p>... and a pony! No, seriously. I am looking for a way to organize tests that "just works". Most things do work, but not all pieces fit together. So here is what I want:</p>
  
  <ul>
  <li>Having tests automatically discovered. This includes doctests. Note that the sum of doctests must not appear as a single test. (i.e. not what py.test --doctest-modules does)</li>
  <li>Being able to run tests in parallel. (Something like py.test -n from xdist)</li>
  <li>Generating a coverage report.</li>
  <li>Make <code>python setup.py test</code> just work.</li>
  </ul>
  
  <p>My current approach involves a <code>tests</code> directory and the <a href="http://docs.python.org/2/library/unittest.html#load-tests-protocol" rel="noreferrer">load_tests protocol</a>. All files contained are named like <code>test_*.py</code>. This makes <code>python -m unittest discover</code> just work, if I create a file <code>test_doctests.py</code> with the following content.</p>
  
  <pre><code>import doctest
  import mymodule1, mymodule2
  def load_tests(loader, tests, ignore):
      tests.addTests(doctest.DocTestSuite(mymodule1))
      tests.addTests(doctest.DocTestSuite(mymodule2))
      return tests
  </code></pre>
  
  <p>This approach also has the upside that one can use setuptools and supply <code>setup(test_suite="unittest2.collector")</code>.</p>
  
  <p>However this approach has a few problems.</p>
  
  <ul>
  <li>coverage.py expects to run a script. So I cannot use unittest2 discovery here.</li>
  <li>py.test does not run load_tests functions, so it does not find the doctests and the <code>--doctest-modules</code> option is crap.</li>
  <li>nosetests runs the <code>load_tests</code> functions, but does not supply any parameters. This appears totally broken on the side of nose.</li>
  </ul>
  
  <p>How can I make things work better than this or fix some of the issues above?</p>