title: How do I get a dataframe or database write from TFX BulkInferrer?
body: <p>I'm very new to TFX, but have an apparently-working ML Pipeline which is to be used via <a href="https://www.tensorflow.org/tfx/api_docs/python/tfx/components/BulkInferrer" rel="noreferrer">BulkInferrer</a>. That seems to produce output exclusively in Protobuf format, but since I'm running bulk inference I want to pipe the results to a database instead. (DB output seems like it should be the default for bulk inference, since both Bulk Inference &amp; DB access take advantage of parallelization... but Protobuf is a per-record, serialized format.)</p>
  <p>I assume I could use something like <a href="https://github.com/rdblue/parquet-avro-protobuf/blob/master/src/main/java/com/example/ProtobufToParquet.java" rel="noreferrer">Parquet-Avro-Protobuf</a> to do the conversion (though that's in Java and the rest of the pipeline's in Python), or I could write something myself to consume all the protobuf messages one-by-one, convert them into JSON, deserialize the JSON into a list of dicts, and load the dict into a Pandas DataFrame, or store it as a bunch of key-value pairs which I treat like a single-use DB... but that sounds like a lot of work and pain involving parallelization and optimization for a very common use case. The top-level Protobuf message definition is Tensorflow's <a href="https://github.com/tensorflow/serving/blob/master/tensorflow_serving/apis/prediction_log.proto" rel="noreferrer">PredictionLog</a>.</p>
  <p>This <em>must</em> be a common use case, because TensorFlowModelAnalytics functions like <a href="https://tensorflow.google.cn/tfx/model_analysis/api_docs/python/tfma/analyze_raw_data?hl=en" rel="noreferrer">this one</a> consume Pandas DataFrames. I'd rather be able to write directly to a DB (preferably Google BigQuery), or a Parquet file (since Parquet / Spark seems to parallelize better than Pandas), and again, those seem like they should be common use cases, but I haven't found any examples.  Maybe I'm using the wrong search terms?</p>
  <p>I also looked at the <a href="https://tensorflow.google.cn/tfx/model_analysis/api_docs/python/tfma/extractors/PredictExtractor?hl=en" rel="noreferrer">PredictExtractor</a>, since &quot;extracting predictions&quot; sounds close to what I want... but the official documentation appears silent on how that class is supposed to be used. I thought <a href="https://tensorflow.google.cn/tfx/transform/api_docs/python/tft/TFTransformOutput?hl=en" rel="noreferrer">TFTransformOutput</a> sounded like a promising verb, but instead it's a noun.</p>
  <p>I'm clearly missing something fundamental here. Is there a reason no one wants to store BulkInferrer results in a database? Is there a configuration option that allows me to write the results to a DB? Maybe I want to add a <a href="https://beam.apache.org/releases/pydoc/2.26.0/apache_beam.io.parquetio.html" rel="noreferrer">ParquetIO</a> or <a href="https://beam.apache.org/releases/pydoc/2.26.0/apache_beam.io.gcp.bigquery.html" rel="noreferrer">BigQueryIO</a> instance to the TFX pipeline? (TFX docs say it uses Beam &quot;<a href="https://www.tensorflow.org/tfx/guide/transform" rel="noreferrer">under the hood</a>&quot; but that doesn't say much about how I should use them together.) But the syntax in those documents looks sufficiently different from my TFX code that I'm not sure if they're compatible?</p>
  <p>Help?</p>