title: Service to periodically load data to BigQuery
body: <p>I noticed that the billing of our project on Google BigQuery is dominated by the cost of the resource <code>StreamingBytes</code> (90% of the total cost). This is due to the fact that we stream one record at a time by using the <code>insertAll()</code> method. In fact, the data inserted per day is in the order of 100GB, but we are charged for about 10 times that (~1TB), because BigQuery considers each row to be minimum 1KB and we push many rows of small size.</p>
  
  <p>Changing the schema of the tables in order to make them more "horizontal" would be a solution to reduce the cost, but it would be not convenient for the way our pipelines are designed.</p>
  
  <p>Instead, we wanted to exploit the fact that loading data to BigQuery from files (POST request) or from Google Cloud Storage is free and that we don't need immediate access to the data. We want to create a service that "buffers" rows coming from different machines (Celery workers in particular) and periodically loads batches of rows to BigQuery via loading from files.</p>
  
  <blockquote>
    <p>First question: is there a service that does the same already?</p>
  </blockquote>
  
  <p>When it comes to the implementation, we were thinking of using a relational database (i.e. PostgreSQL) to store the buffered data so that we can exploit the way it handles concurrent transactions. An in-memory storage (like Redis) would be too expensive for such use case. Can you think of a better solution?</p>
  
  <blockquote>
    <p>Where should I store data in the buffer service?</p>
  </blockquote>