<p>I'm trying to host an application using Google Kubernetes Engine. My docker image works when run locally, but when I put it on Google Cloud and set it up using a kubernetes cluster it fails in a very strange way. </p>

<p>I'm able to connect to the application and it works until I trigger a call of <code>google.cloud.storage.Client()</code>. Then it attempts to read the file I've provided through the <code>GOOGLE_APPLICATION_CREDENTIALS</code> environment variable and something goes wrong. I get this (truncated and redacted) traceback:</p>

<pre><code>self.gcs_client = storage.Client()
  File "/usr/local/lib/python3.6/site-packages/google/cloud/storage/client.py", line 71, in __init__
    _http=_http)
  File "/usr/local/lib/python3.6/site-packages/google/cloud/client.py", line 215, in __init__
    _ClientProjectMixin.__init__(self, project=project)
  File "/usr/local/lib/python3.6/site-packages/google/cloud/client.py", line 169, in __init__
    project = self._determine_default(project)
  File "/usr/local/lib/python3.6/site-packages/google/cloud/client.py", line 182, in _determine_default
    return _determine_default_project(project)
  File "/usr/local/lib/python3.6/site-packages/google/cloud/_helpers.py", line 179, in _determine_default_project
    _, project = google.auth.default()
  File "/usr/local/lib/python3.6/site-packages/google/auth/_default.py", line 294, in default
    credentials, project_id = checker()
  File "/usr/local/lib/python3.6/site-packages/google/auth/_default.py", line 165, in _get_explicit_environ_credentials
    os.environ[environment_vars.CREDENTIALS])
  File "/usr/local/lib/python3.6/site-packages/google/auth/_default.py", line 89, in _load_credentials_from_file
    'File {} was not found.'.format(filename))
google.auth.exceptions.DefaultCredentialsError: File {--redacted--} was not found.
</code></pre>

<p>What I've redacted is JSON containing my service account's private key. I've checked the docker container while it's running on the cloud and <code>GOOGLE_APPLICATION_CREDENTIALS</code> is set to a file name as expected. Somehow when my docker container is run on the cloud, instead of using the environment variable as the file name - it uses the contents of the referenced file as the file name. This error also gets reported in the browser console, so anyone that navigates to the app can get my service account credentials. </p>

<p>Does anyone have any guesses about what's going wrong here? </p>

<p><strong>UPDATE:</strong>
Looks like I've got it working now. My guess is the error occurred because I set <code>GOOGLE_APPLICATION_CREDENTIALS</code> using <code>valueFrom</code> as in the <code>bokeh.yaml</code> file. This seems to have been setting <code>GOOGLE_APPLICATION_CREDENTIALS</code> equal to the contents of <code>bokeh.yaml</code>. This format seems to work for <code>pandas.io.gbq.read_gbq</code> used in the tutorial code but not for instantiating <code>google.cloud.storage.Client()</code> like I was trying to do. Pointing <code>GOOGLE_APPLICATION_CREDENTIALS</code> to a volume mounted file like DazWilkin suggested and Eric Guan showed worked. </p>

<p>I noticed you didn't mention using Kubernetes Secrets. Here's how I did it.</p>

<ol>
<li>Create a Secret with <code>kubectl</code> on your local machine.</li>
</ol>



<pre><code>$ kubectl create secret generic gac-keys --from-file=&lt;PATH_TO_SERVICE_ACCOUNT_FILE&gt; 
</code></pre>

<p>This creates a Secret called <code>gac-keys</code>. It contains your json file found at <code>&lt;PATH_TO_SERVICE_ACCOUNT_FILE&gt;</code>. If you want to rename the json file, you can do </p>

<p><code>--from-file=new-file-name.json=&lt;PATH_TO_SERVICE_ACCOUNT_FILE&gt;</code></p>

<ol start="2">
<li>Configure your Deployment to use the Secret.</li>
</ol>



<pre><code>apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: my-app
spec:
  template:
    spec:
      volumes:
      - name: google-cloud-keys
        secret:
          secretName: gac-keys
      containers:
      - name: my-app
        image: us.gcr.io/my-app
        volumeMounts:
        - name: google-cloud-keys
          mountPath: /var/secrets/google
          readOnly: true
        env:
        - name: GOOGLE_APPLICATION_CREDENTIALS
          value: /var/secrets/google/new-file-name.json
</code></pre>

<p>You specify a volume with an arbitrary name <code>google-cloud-keys</code> to be referenced later. The volume is linked to the secret. Inside the container spec, you mount the <code>google-cloud-keys</code> volume at path <code>/var/secrets/google</code>. This places your file at the path, so <code>/var/secrets/google/new-file-name.json</code> should exist in the container at runtime. Then you specify an env variable named <code>GOOGLE_APPLICATION_CREDENTIALS</code> which points to the path. Now your client library can authenticate with google.</p>

<p>Docs: <a href="https://kubernetes.io/docs/concepts/configuration/secret/" rel="nofollow noreferrer">https://kubernetes.io/docs/concepts/configuration/secret/</a></p>
