title: BigQueryException: API limit exceeded
body: <p>I'm using Bigquery Java API to run ~1000 copy jobs simultaneously (With scala.concurrent.Future) with WriteDisposition WRITE_APPEND, but I'm getting </p>
  
  <pre><code>com.google.cloud.bigquery.BigQueryException: API limit exceeded: Unable to return a row that exceeds the API limits. To retrieve the row, export the table 
  </code></pre>
  
  <p>I thought this is caused by too much concurrency, then I tried to use Monix's Task to limit the parallelism to at most 20:</p>
  
  <pre><code>def execute(queries: List[Query]): Future[Seq[Boolean]] = {
      val tasks: Iterator[Task[List[Boolean]]] = queries.map(q =&gt; BqApi.copyTable(q, destinationTable))
        .sliding(20, 20)
        .map(Task.gather(_))
  
      val results: Task[List[Boolean]] = Task.sequence(tasks)
        .map(_.flatten.toList)
  
      results.runAsync
    }
  </code></pre>
  
  <p>where BqApi.copyTable executes the query and copy the result to the destination table then returns a Task[Boolean]. </p>
  
  <p>The same exception still happens.</p>
  
  <p>But if I change the WriteDisposition to WRITE_TRUNCATE, the exception goes away.</p>
  
  <p>Can anyone help me to understand what happens under the hood? And why Bigquery API behaves like this?</p>