title: How to activate Dataflow Shuffle service?
body: <p>I am trying to use <a href="https://cloud.google.com/dataflow/service/dataflow-service-desc#cloud-dataflow-shuffle" rel="nofollow noreferrer">Dataflow Shuffle service</a> in python environment, but it seems shuffle service is not working, as you can see below.</p>
  
  <p><a href="https://i.stack.imgur.com/cfaq5m.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/cfaq5m.png" alt="console output"></a></p>
  
  <p>I set the SDK version above 2.1, and the region is <code>us-central1</code>.
  I thought we can activate Dataflow Shuffle service by just adding experiments option, am I missing something?</p>
  
  <p><a href="https://i.stack.imgur.com/GSKevm.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/GSKevm.png" alt=""></a></p>
  
  <p>Following is the code I tested, you can reproduce the phenomenon.</p>
  
  <pre><code>import apache_beam as beam
  
  options = beam.options.pipeline_options.PipelineOptions()
  gcloud_options = options.view_as(
      beam.options.pipeline_options.GoogleCloudOptions)
  gcloud_options.job_name = 'dataflow-shuffle-test'
  gcloud_options.project = 'PROJECTID'
  gcloud_options.staging_location = 'gs://BUCKET/staging'
  gcloud_options.temp_location = 'gs://BUCKET/temp'
  
  # maybe this is the wrong way?
  debug_options = options.view_as(beam.options.pipeline_options.DebugOptions)
  debug_options.experiments = 'shuffle_mode=service'
  
  worker_options = options.view_as(beam.options.pipeline_options.WorkerOptions)
  worker_options.disk_size_gb = 20
  worker_options.max_num_workers = 2
  
  options.view_as(beam.options.pipeline_options.StandardOptions).runner = 'DataflowRunner'
  
  def modify_data2(kvpair):
      return {'name': kvpair[0],
              'sum': sum(kvpair[1])
              }
  
  
  p = beam.Pipeline(options=options)
  
  query = 'SELECT * FROM [bigquery-public-data:usa_names.usa_1910_current]'
  (p | 'read' &gt;&gt; beam.io.Read(beam.io.BigQuerySource(project='PROJECTID', 
                                                     use_standard_sql=False, 
                                                     query=query))
     | 'pair' &gt;&gt; beam.Map(lambda x: (x['name'], x['number']))
     | "groupby" &gt;&gt; beam.GroupByKey()
     | 'modify' &gt;&gt; beam.Map(modify_data2)
     | 'write' &gt;&gt; beam.io.WriteToText('gs://BUCKET/test.txt', num_shards=1)
   )
  
  p.run()
  </code></pre>
  
  <p>The job finishes successfully, without any errors. Any comment would be helpful!</p>
  
  <p><strong>EDIT</strong><br>
  Thanks to Sergei's answer, I found my mistake. What I mistook is the experiments option. Set the experiments option as below.</p>
  
  <pre><code># set as list, instead of string.
  debug_options.experiments = ['shuffle_mode=service']
  </code></pre>
  
  <p>Also I made a gist to run a simple pipeline with shuffle service, the notebook can be runnable on Datalab.
  <a href="https://gist.github.com/hayatoy/f6664f965a2519ec406e11235faf75b6" rel="nofollow noreferrer">https://gist.github.com/hayatoy/f6664f965a2519ec406e11235faf75b6</a></p>