title: How do I integration test a Dataflow pipeline writing to Bigtable?
body: <p>According to the <a href="https://beam.apache.org/documentation/pipelines/test-your-pipeline/" rel="nofollow noreferrer">Beam website</a>,</p>
  
  <blockquote>
    <p>Often it is faster and simpler to perform local unit testing on your
    pipeline code than to debug a pipeline’s remote execution.</p>
  </blockquote>
  
  <p>I want to use test-driven development for my Beam/Dataflow app that writes to Bigtable for this reason.</p>
  
  <p>However, following the Beam testing documentation I get to an impasse--PAssert isn't useful because the output PCollection contains org.apache.hadoop.hbase.client.Put objects, which don't override the equals method.</p>
  
  <p>I <a href="https://stackoverflow.com/a/41452218/2430134">can't get the contents</a> of the PCollection to do validation on them either, since</p>
  
  <blockquote>
    <p>It is not possible to get the contents of a PCollection directly - an
    Apache Beam or Dataflow pipeline is more like a query plan of what
    processing should be done, with PCollection being a logical
    intermediate node in the plan, rather than containing the data.</p>
  </blockquote>
  
  <p>So how can I test this pipeline, other than manually running it? I'm using Maven and JUnit (in Java since that's all the <a href="https://cloud.google.com/bigtable/docs/hbase-dataflow-java" rel="nofollow noreferrer">Dataflow Bigtable Connector</a> seems to support).</p>