title: Randomized testing in Java
body: <p>I write a lot of unit tests. Often, you need to write carefully considered test cases by hand, a form of whitebox testing. If you are lucky enough to work for a company with a separate quality assurance engineers, perhaps someone else writes test cases for you (kind of a mix between white and black box testing).</p>
  
  <p>Many times, however, randomized testing would find many bugs and would serve as a great complement to hand-written cases.</p>
  
  <p>For example, I might have a self-contained class and be able to express the invariants and broad-stroke behavior of the class simply (such as "this method never throws an exception" or "this method always returns a positive value"). I would like a test framework that just bashes on my class and checks the invariants.</p>
  
  <p>A similar case: I often have a class which implements similar functionality to another class (but does it with different performance characteristics or with some added functionality). I would to A vs B test the two classes in a randomized way. For example, if I was implementing <code>TreeMap</code>, I could use <code>HashMap</code> as a comparable implementation (modulo a few differences due to the sorted behavior of <code>TreeMap</code>) and check most of the basic functionality in a randomized way. Simlarly, someone implementing <code>LinkedList</code> could use <code>ArrayList</code> as a comparable implementation and vice-versa.</p>
  
  <p>I've written some basic stuff to do this in the past, but it is painstaking to set to up all the boilerplate to:</p>
  
  <ol>
  <li>Create objects with random initial state</li>
  <li>Apply random mutations</li>
  <li>Create mappings between "like" objects for A vs B testing</li>
  <li>Define invariants and rules such as "when will exceptions be thrown" </li>
  </ol>
  
  <p>I still do it from time to time, but I want to reduce my effort level. For example, are there frameworks that remove or simplify the required boilerplate?</p>
  
  <p>Otherwise, what techniques are used to do randomized testing in Java?</p>
  
  <p>This is related, but not the same as fuzz testing. Fuzz testing seems to focus on random inputs to a single entity, in hope of triggering bad behavior, often with an adaptive input model based on dynamic coverage observations. That's covers a lot of the above, but doesn't cover, stuff like A vs B testing when comparable implementations exist, or invariant checking. In any case, I'm also interested in decent fuzz testing libraries for Java.</p>