title: Does the current GoLang SDK for Google Dataflow support Autoscaling and Parallel Processing?
body: <p>Currently utilising the Google Dataflow with Python for batch processing. This works fine, however, I'm interested in getting a bit more speed out of my Dataflow Jobs without having to deal with Java.</p>
  
  <p>Using the Go SDK, I've implemented a simple pipeline that reads a series <em>100-500mb</em> files from Google Storage (using <em><code>textio.Read</code></em>), does some aggregation and updates CloudSQL with the results. The number of files being read can range from dozens to hundreds.</p>
  
  <p>When I run the pipeline, I can see from the logs that files are being read serially, instead of in parallel, as a result the job takes much longer. The same process executed with the Python SDK triggers autoscaling and runs multiple reads within minutes.</p>
  
  <p>I've tried specifying the number of workers using <code>--num_workers=</code>, however, Dataflow scales the job down to one instance after a few minutes and from the logs no parallel reads take place in the time the instance was running.</p>
  
  <p>Something similar happens if I remove the <code>textio.Read</code> and implement a custom DoFn for reading from GCS. The read process is still run serially.</p>
  
  <p>I'm aware the current Go SDK is experimental and lacks many features, however, I haven't found a direct reference to limitations with Parallel processing, <a href="https://docs.google.com/document/d/1yj0_hxq2J1iestjFUUrm_BVQLsFxQiiqtcFhgodzIgM/edit#heading=h.mgr8mrx81tnc" rel="nofollow noreferrer">here</a>. Does the current incarnation of the Go SDK support parallel processing on Dataflow?</p>
  
  <p>Thanks in advance</p>