title: How to pass environment variables to gcloud beta ai custom-jobs create with custom container (Vertex AI)
body: <p>I'm running custom training jobs in google's Vertex AI. A simple <code>gcloud</code> command to execute a custom job would use something like the following syntax (complete documentation for the command can be seen <a href="https://cloud.google.com/sdk/gcloud/reference/beta/ai/custom-jobs/create#--config" rel="nofollow noreferrer">here</a>):</p>
  <pre><code>gcloud beta ai custom-jobs create --region=us-central1 \
  --display-name=test \
  --config=config.yaml
  </code></pre>
  <p>In the <code>config.yaml</code> file, it is possible to specify the machine and accelerator (GPU) types, etc., and in my case, point to a custom container living in the Google Artifact Registry that executes the training code (specified in the <code>imageUri</code> part of the <code>containerSpec</code>). An example config file may look like this:</p>
  <pre><code># config.yaml
  workerPoolSpecs:
    machineSpec:
      machineType: n1-highmem-2
      acceleratorType: NVIDIA_TESLA_P100
      acceleratorCount: 2
    replicaCount: 1
    containerSpec:
      imageUri: {URI_FOR_CUSTOM_CONATINER}
      args:
      - {ARGS TO PASS TO CONTAINER ENTRYPOINT COMMAND}
  </code></pre>
  <p>The code we're running needs some runtime environment variables (that need to be secure) passed to the container. In the <a href="https://cloud.google.com/vertex-ai/docs/reference/rest/v1/CustomJobSpec#containerspec" rel="nofollow noreferrer">API documentation</a> for the <code>containerSpec</code>, it says it is possible to set environment variables as follows:</p>
  <pre><code># config.yaml
  workerPoolSpecs:
    machineSpec:
      machineType: n1-highmem-2
      acceleratorType: NVIDIA_TESLA_P100
      acceleratorCount: 2
    replicaCount: 1
    containerSpec:
      imageUri: {URI_FOR_CUSTOM_CONATINER}
      args:
      - {ARGS TO PASS TO CONTAINER ENTRYPOINT COMMAND}
      env:
      - name: SECRET_ONE
        value: $SECRET_ONE
      - name: SECRET_TWO
        value: $SECRET_TWO
  </code></pre>
  <p>When I try and add the <code>env</code> flag to the <code>containerSpec</code>, I get an error saying it's not part of the container spec:</p>
  <pre><code>ERROR: (gcloud.beta.ai.custom-jobs.create) INVALID_ARGUMENT: Invalid JSON payload received. Unknown name &quot;env&quot; at 'custom_job.job_spec.worker_pool_specs[0].container_spec': Cannot find field.
  - '@type': type.googleapis.com/google.rpc.BadRequest
    fieldViolations:
    - description: &quot;Invalid JSON payload received. Unknown name \&quot;env\&quot; at 'custom_job.job_spec.worker_pool_specs[0].container_spec':\
        \ Cannot find field.&quot;
      field: custom_job.job_spec.worker_pool_specs[0].container_spec
  </code></pre>
  <p>Any idea how to securely set runtime environment variables in Vertex AI custom jobs using custom containers?</p>