title: GCP Vertex AI Pipeline fails during build on the endpoint error
body: <p>I have deployed a custom Kubeflow pipeline using a mix of AutoML components and a custom Kubeflow Component.</p>
  <p>When I deploy the pipeline, it fails and I get the following error:</p>
  <pre><code>textPayload: &quot;The replica workerpool0-0 exited with a non-zero status of 1. Termination reason: 
   Error. To find out more about why your job exited please check the logs: 
   https://console.cloud.google.com/logs/viewer? project=205438435937&amp;resource=ml_job%2Fjob_id%XXXXXXXXXXXXXXXX&amp;advancedFilter=resource.type%3D 
   %22ml_job%22%0Aresource.labels.job_id%3D%XXXXXXXXXXXXXXXXXXXX%22&quot;
  insertId: &quot;ibt166bgd&quot;
  resource: {
   type: &quot;ml_job&quot;
    labels: {
     job_id: &quot;XXXXXXXXXXXXXXXXXX&quot;
     task_name: &quot;service&quot;
     project_id: &quot;XXXXXXX-XXXXXX&quot;
    }
   }
   timestamp: &quot;2021-06-10T12:18:53.807150835Z&quot;
   severity: &quot;ERROR&quot;
   labels: {
    ml.googleapis.com/endpoint: &quot;&quot;
   }
   logName: &quot;projects/XXXXXXX-XXXXXX/logs/ml.googleapis.com%XXXXXXXXXXXXXXXXXXXX&quot;
   receiveTimestamp: &quot;2021-06-10T12:18:55.087983509Z&quot;
  }
  </code></pre>
  <p>This is my pipeline configuration:</p>
  <pre><code># Kubeflow pipline defined by a Python function
  @kfp.dsl.pipeline(
      name=&quot;sales-prediction-iowa&quot;,
      pipeline_root=pipeline_root_path)
  def pipeline(project_id: str):
      pre_process = preprocess(
          project_id=project_id,
      )
  
      create_dataset = gcc_aip.TabularDatasetCreateOp(
      project=project_id,
      display_name=display_name,
      # gcs_source=&quot;gs://vertex-ai-pipeline-bucket/iowa-2020_pre-processed.csv&quot;
      gcs_source=pre_process.output
      )
  
  
      training_job_run_op = gcc_aip.AutoMLTabularTrainingJobRunOp(
          project=project_id,
          display_name=&quot;training-iowa-sales&quot;,
          optimization_prediction_type=&quot;regression&quot;,
          dataset=create_dataset.outputs[&quot;dataset&quot;],
          model_display_name=&quot;iowa-sales-model&quot;,
          target_column=&quot;sale_dollars&quot;,
          training_fraction_split=0.8,
          validation_fraction_split=0.1,
          test_fraction_split=0.1,
          budget_milli_node_hours=8000,
      )
  
      endpoint_op = gcc_aip.ModelDeployOp(
          project=project_id, model=training_job_run_op.outputs.model
      )
  
  
  compiler.Compiler().compile(pipeline_func=pipeline,
          package_path='iowa-pipeline-job.json')
  
  api_client = AIPlatformClient(project_id=project_id, region=region)
  
  response = api_client.create_run_from_job_spec(
      'iowa-pipeline-job.json',
      pipeline_root=pipeline_root_path,
      service_account=service_account,
      parameter_values={
          'project_id': project_id,
          # 'region': region,
          # 'pipeline_root_path': pipeline_root_path,
          # 'service_account': service_account,
          # 'display_name': display_name
      }
  )
  </code></pre>
  <p>I have a sneaky suspicion it might be linked to regions, but please let me know if there is something else her.</p>
  <p>Thanks in advance!</p>