<p>I am running Kubeflow pipeline(docker approach) and cluster uses the <strong>endpoint</strong> to navigate to the dashboard. The Clusters is created followed by the instructions mentioned in this link <a href="https://deploy.kubeflow.cloud/#/" rel="nofollow noreferrer">Deploy Kubeflow</a>. Everything is successfully created and the cluster generated the endpoints and its working perfectly. </p>

<p>Endpoint link would be something like this <a href="https://%3Cname%3E.endpoints.%3Cproject%3E.cloud.goog" rel="nofollow noreferrer">https://appname.endpoints.projectname.cloud.goog</a>.</p>

<p>Every workload of the pipeline is working fine except the last one. In the last workload, I am trying to submit a job to the cloud-ml engine. But it logs shows that the application has no access to the project. Here is the full image of the log.</p>

<blockquote>
  <p>ERROR:
  (gcloud.ml-engine.versions.create) PERMISSION_DENIED: Request had
  insufficient authentication scopes. </p>
  
  <p>ERROR:
  (gcloud.ml-engine.jobs.submit.prediction) User
  [clustername@project_name.iam.gserviceaccount.com]
  does not have permission to access project [project_name]
  (or it may not exist): Request had insufficient authentication scopes.</p>
</blockquote>

<p>From the logs, it's clear that this service account doesn't have access to the project itself. However, I tried to give access for Cloud ML Service to this service account but still, it's throwing the same error.</p>

<p>Any other ways to give Cloud ML service credentials to this application. </p>

<p>Check two things:</p>

<p>1) GCP IAM: if clustername-user@projectname.iam.gserviceaccount.com has ML Engine Admin permission.</p>

<p>2) Your pipeline DSL: if the cloud-ml engine step calls apply(gcp.use_gcp_secret('user-gcp-sa')), e.g. <a href="https://github.com/kubeflow/pipelines/blob/ea07b33b8e7173a05138d9dbbd7e1ce20c959db3/samples/tfx/taxi-cab-classification-pipeline.py#L67" rel="nofollow noreferrer">https://github.com/kubeflow/pipelines/blob/ea07b33b8e7173a05138d9dbbd7e1ce20c959db3/samples/tfx/taxi-cab-classification-pipeline.py#L67</a></p>
