title: Second `ParallelRunStep` in pipeline times out at start
body: <p>Im trying to run a sequence of more than one <code>ParallelRunStep</code> in an AzureML pipeline. To do so, I create a step with the following helper:</p>
  
  <pre><code>def create_step(name, script, inp, inp_ds):
      out = pip_core.PipelineData(name=f"{name}_out", datastore=dstore, is_directory=True)
      out_ds = out.as_dataset()
      out_ds_named = out_ds.as_named_input(f"{name}_out")
  
      config = cont_steps.ParallelRunConfig(
          source_directory="src",
          entry_script=script,
          mini_batch_size="1",
          error_threshold=0,
          output_action="summary_only",
          compute_target=compute_target,
          environment=component_env,
          node_count=2,
          logging_level="DEBUG"
      )
  
      step = cont_steps.ParallelRunStep(
          name=name,
          parallel_run_config=config,
          inputs=[inp_ds],
          output=out,
          arguments=[],
          allow_reuse=False,
      )
  
      return step, out, out_ds_named
  </code></pre>
  
  <p>As an example I create two steps like this</p>
  
  <pre><code>step1, out1, out1_ds_named = create_step("step1", "demo_s1.py", input_ds, named_input_ds)
  step2, out2, out2_ds_named = create_step("step2", "demo_s2.py", out1, out1_ds_named)
  </code></pre>
  
  <p>Creating an experiment and submitting it to an existing workspace and Azure ML compute cluster works. Also the first step <code>step1</code> uses the <code>input_ds</code> runs its script <code>demo_s1.py</code> (which produces its output files, and finishes successfully. </p>
  
  <p>However the second step <code>step2</code> never get started. </p>
  
  <p><a href="https://i.stack.imgur.com/JPnPI.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/JPnPI.png" alt="enter image description here"></a></p>
  
  <p>And there is a final exception</p>
  
  <pre><code>The experiment failed. Finalizing run...
  Cleaning up all outstanding Run operations, waiting 300.0 seconds
  2 items cleaning up...
  Cleanup took 0.16968441009521484 seconds
  Starting the daemon thread to refresh tokens in background for process with pid = 394
  Traceback (most recent call last):
    File "driver/amlbi_main.py", line 52, in &lt;module&gt;
      main()
    File "driver/amlbi_main.py", line 44, in main
      JobStarter().start_job()
    File "/mnt/batch/tasks/shared/LS_root/jobs/pipeline/azureml/08a1e1e1-7c3f-4c5a-84ad-ca99b8a6cb31/mounts/workspaceblobstore/azureml/08a1e1e1-7c3f-4c5a-84ad-ca99b8a6cb31/driver/job_starter.py", line 48, in start_job
      job.start()
    File "/mnt/batch/tasks/shared/LS_root/jobs/pipeline/azureml/08a1e1e1-7c3f-4c5a-84ad-ca99b8a6cb31/mounts/workspaceblobstore/azureml/08a1e1e1-7c3f-4c5a-84ad-ca99b8a6cb31/driver/job.py", line 70, in start
      master.start()
    File "/mnt/batch/tasks/shared/LS_root/jobs/pipeline/azureml/08a1e1e1-7c3f-4c5a-84ad-ca99b8a6cb31/mounts/workspaceblobstore/azureml/08a1e1e1-7c3f-4c5a-84ad-ca99b8a6cb31/driver/master.py", line 174, in start
      self._start()
    File "/mnt/batch/tasks/shared/LS_root/jobs/pipeline/azureml/08a1e1e1-7c3f-4c5a-84ad-ca99b8a6cb31/mounts/workspaceblobstore/azureml/08a1e1e1-7c3f-4c5a-84ad-ca99b8a6cb31/driver/master.py", line 149, in _start
      self.wait_for_input_init()
    File "/mnt/batch/tasks/shared/LS_root/jobs/pipeline/azureml/08a1e1e1-7c3f-4c5a-84ad-ca99b8a6cb31/mounts/workspaceblobstore/azureml/08a1e1e1-7c3f-4c5a-84ad-ca99b8a6cb31/driver/master.py", line 124, in wait_for_input_init
      raise exc
  exception.FirstTaskCreationTimeout: Unable to create any task within 600 seconds.
  Load the datasource and read the first row locally to see how long it will take.
  Set the advanced argument '--first_task_creation_timeout' to a larger value in arguments in ParallelRunStep.
  
  </code></pre>
  
  <p>I have the impression, that the second step is waiting for some data. However the first step creates the supplied output directory and also a file. </p>
  
  <pre><code>import argparse
  import os
  
  def init():
      pass
  
  def run(parallel_input):
      print(f"*** Running {os.path.basename(__file__)} with input {parallel_input}")
  
      parser = argparse.ArgumentParser(description="Data Preparation")
      parser.add_argument('--output', type=str, required=True)
      args, unknown_args = parser.parse_known_args()
  
      out_path = os.path.join(args.output, "1.data")
      os.makedirs(args.output, exist_ok=True)
      open(out_path, "a").close()
  
      return [out_path]
  </code></pre>
  
  <p>I have no idea how to debug further. Has anybody an idea?</p>