title: Hot keys in dataflow/apache beam pipeline, is turning on shuffle mode the only solution
body: <p><strong>Pipeline Use case:</strong>
  This pipeline reads json files from 30 GCS directories, each day is represented as a directory so reads past 30 days of data (14 billion rows). Does some processing  on that data and outputs to GCS  as json files.</p>
  
  <p><strong>Hotkeys problem:</strong>
  The hot key problem gets introduced because our business logic requires a lot more processing on the latest days data (~400 million rows) relative to past 29 days of data. So if pipeline ran today it will read data from past month, data 30 days back to yesterday requires minimal processing. While data from today requires about 100x more processing. Since we process all the data from past month including today in one step a hot key is generated when data from today is being processed</p>
  
  <p>In pipeline when it initially starts I notice that there is very high throughput (million rows per second) after pipeline has processed past 29 days of data and starts processing todays data throughput declines dramatically (hundered rows per second). Since dataflow is assigning worker nodes per input (30 directories) the input that requires the most processing gets stalled because it is not parallelized to all worker nodes. While most of the worker nodes were used to process 29 days of data.</p>
  
  <p><strong>Solutions attempted:</strong></p>
  
  <ul>
  <li>Grouping and Hashing to create shuffling to mix high processing rows with low processing (2018-05-02_02_24_23-2746349996700640181) <code>GroupBy</code> step in this performed so slow that performance was worst than it was with hotkey problem.</li>
  <li>Adding <code>Reshuffle.createRandomKey</code> step to create shuffling to mix high processing rows with low processing (2018-05-03_04_50_16-6027709731981798472) Reshuffle step in this performed so slow that performance was worst than it was with hotkey problem</li>
  </ul>
  
  <p>One common suggestion here is to enable to shuffle mode. We have tried to do that in the past. While shuffle mode did improve performance of <code>GroupByKey</code> step it dramatically decreased performance when output was being written to GCS. </p>