title: Side Input data doesn't get updated - Python Apache Beam
body: <p>I'm building a pipeline with dynamic configuration data that gets updated whenever is triggered.</p>
  <p>There are 2 PubSub topics, topic A for the IoT data, topic B is for the configuration that will be used to transform the IoT data.</p>
  <p>The configuration is kept in Cloud Firestore. When the database gets updated, Cloud Function will read and send the updated configuration to PubSub topic B.</p>
  <p>The problem is that Dataflow jobs only read configuration data at the start of the job and will never be updated.</p>
  <p>How can I do it so that the side input gets updated?</p>
  <pre><code>p = beam.Pipeline(options=options)
  
  class Transform(beam.DoFn):
      def process(self, configuration):
          ...
          yield output
  
  def run():
      ...
      iot_data = (p
          | 'ReadIotData' &gt;&gt; ReadFromPubSub(TOPIC_A)
  
      configuration = (p
          | 'ReadConfig' &gt;&gt; ReadFromPubSub(TOPIC_B)
          | 'WindowUserData' &gt;&gt; beam.WindowInto(
              window.GlobalWindows(),
              trigger=trigger.Repeatedly(trigger.AfterCount(1)),
              accumulation_mode=trigger.AccumulationMode.DISCARDING)
          | 'JsonLoadsUserData' &gt;&gt; beam.Map(lambda x: ('data', x.decode().replace('\\','')))
  
      output = (iot_data
          | 'transform' &gt;&gt; beam.ParDo(Transform(),
              beam.pvalue.AsDict(configuration))
          | 'Output' &gt;&gt; WriteToPubSub(TOPIC_C)
  </code></pre>