title: SpannerIO Batch Read Speed Affected by Slow Partition Reader
body: <p>We have been using <code>SpannerIO.readAll</code> to scan large amount of data in google dataflow setting. The <code>ReadOperation</code>s passed to spanner are created <code>withQuery(query)</code> and <code>withBatching(true)</code>. I noticed that though initially the throughput is OK, it dropped to very low throughput in the end probably due to outliers with larger amount work. Looking at <a href="https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/BatchSpannerRead.java" rel="nofollow noreferrer">BatchSpannerRead</a> code, one DoFn is taking care of all the batch scan work for a partition. Although in a perfect world, we should assume the generated partitions should handle this outlier issues, but in practice, will it make sense to re-split the work of those slow workers?</p>