title: How to debug python dataflow beam.io.WriteToBigQuery
body: <p>I currently have a python dataflow job whose end sink is a PCollection write to BigQuery. It's failing with the following error:</p>
  
  <pre><code>Workflow failed. Causes: S01:XXXX+XXX+Write/WriteToBigQuery/NativeWrite failed., BigQuery import job "dataflow_job_XXXXXX" failed., BigQuery job "dataflow_job_XXXXXX" in project "XXXXXX" finished with error(s): errorResult: Error while reading data, error message: JSON table encountered too many errors, giving up. Rows: 19; errors: 1
  </code></pre>
  
  <p>To get a more detailed error report I'm then running:</p>
  
  <pre><code>bq --format=prettyjson show -j dataflow_job_XXXXXX
  </code></pre>
  
  <p>which displays something like (there are a bunch of errors this is just one of them):</p>
  
  <pre><code>{
  
      "location": "gs://XXXXX/XXXXXX/tmp/XXXXX/10002237702794672370/dax-tmp-2019-02-05_20_14_50-18341731408970037725-S01-0-5144bb700f6a9f0b/-shard--try-00d3c2c24d5b0371-endshard.json",
  
      "message": "Error while reading data, error message: JSON table encountered too many errors, giving up. Rows: 11; errors: 1. Please look into the errors[] collection for more details.",                                          
  
      "reason": "invalid"
  
    },
  </code></pre>
  
  <p>I then go looking for the particular shard to see what PCollection row is in error and what I need to do to either filter these rows or fix a bug of mine:</p>
  
  <pre><code>gsutil ls gs://XXXXX/XXXXXX/tmp/XXXXX/10002237702794672370/dax-tmp-2019-02-05_20_14_50-18341731408970037725-S01-0-5144bb700f6a9f0b/-shard--try-00d3c2c24d5b0371-endshard.json
  </code></pre>
  
  <p>But that command returns:</p>
  
  <pre><code>CommandException: One or more URLs matched no objects.
  </code></pre>
  
  <p>What are the best practices for debugging jobs (which takes multiple hours btw)? My thought right now is to write the PCollection to GCS in a non-temp location in JSON format and try to ingest it myself. </p>