<p>I am trying to connect to Firestore from code running on GKE Container.  Simple REST GET api is working fine, but when I access the Firestore from read/write, I am getting Missing or insufficient permissions.</p>
<pre><code>An unhandled exception was thrown by the application.
Info
2021-06-06 21:21:20.283 EDT
      Grpc.Core.RpcException: Status(StatusCode=&quot;PermissionDenied&quot;, Detail=&quot;Missing or insufficient permissions.&quot;, DebugException=&quot;Grpc.Core.Internal.CoreErrorDetailException: {&quot;created&quot;:&quot;@1623028880.278990566&quot;,&quot;description&quot;:&quot;Error received from peer ipv4:172.217.193.95:443&quot;,&quot;file&quot;:&quot;/var/local/git/grpc/src/core/lib/surface/call.cc&quot;,&quot;file_line&quot;:1068,&quot;grpc_message&quot;:&quot;Missing or insufficient permissions.&quot;,&quot;grpc_status&quot;:7}&quot;)
         at Google.Api.Gax.Grpc.ApiCallRetryExtensions.&lt;&gt;c__DisplayClass0_0`2.&lt;&lt;WithRetry&gt;b__0&gt;d.MoveNext()
</code></pre>
<p><strong>Update</strong> I am trying to provide secret to pod with service account credentails.
Here is the k8 file which deploys a pod to cluster with no issues when no secrets are provided and I can do Get Operations which don't hit Firestore, and they work fine.</p>
<pre><code>kind: Deployment
apiVersion: apps/v1
metadata:
  name: foo-worldmanagement-production
spec:
  replicas: 1
  selector:
    matchLabels:
       app: foo
       role: worldmanagement
       env: production
  template:
    metadata:
      name: worldmanagement
      labels:
        app: foo
        role: worldmanagement
        env: production
    spec:
      containers:
      - name: worldmanagement
        image: gcr.io/foodev/foo/master/worldmanagement.21
        resources:
          limits:
            memory: &quot;500Mi&quot;
            cpu: &quot;300m&quot;
        imagePullworld: Always
        readinessProbe:
          httpGet:
            path: /api/worldManagement/policies
            port: 80
        ports:
        - name: worldmgmt
          containerPort: 80
</code></pre>
<p>Now, if I try to mount secret, the pod never gets created fully, and it eventually fails</p>
<pre><code>kind: Deployment
apiVersion: apps/v1
metadata:
  name: foo-worldmanagement-production
spec:
  replicas: 1
  selector:
    matchLabels:
       app: foo
       role: worldmanagement
       env: production
  template:
    metadata:
      name: worldmanagement
      labels:
        app: foo
        role: worldmanagement
        env: production
    spec:
      volumes:
      - name: google-cloud-key
        secret:
          secretName: firestore-key   
      containers:
      - name: worldmanagement
        image: gcr.io/foodev/foo/master/worldmanagement.21
        volumeMounts:
        - name: google-cloud-key
          mountPath: /var/
        env:
        - name: GOOGLE_APPLICATION_CREDENTIALS
          value: /var/key.json        
        resources:
          limits:
            memory: &quot;500Mi&quot;
            cpu: &quot;300m&quot;
        imagePullworld: Always
        readinessProbe:
          httpGet:
            path: /api/worldManagement/earth
            port: 80
        ports:
        - name: worldmgmt
          containerPort: 80
</code></pre>
<p>I tried to deploy the <a href="https://cloud.google.com/kubernetes-engine/docs/tutorials/authenticating-to-cloud-platform" rel="nofollow noreferrer">sample application</a> and it works fine.</p>
<p>If I keep only the following the yaml file, the container gets deployed properly</p>
<pre><code>- name: google-cloud-key
            secret:
              secretName: firestore-key 
</code></pre>
<p>But once I add the following to yaml, it fails</p>
<pre><code>    volumeMounts:
    - name: google-cloud-key
      mountPath: /var/
    env:
    - name: GOOGLE_APPLICATION_CREDENTIALS
      value: /var/key.json      
</code></pre>
<p>And I can see in GCP events that the container is not able to find the google-cloud-key. Any idea how to troubleshoot this issue, i.e why I am not able to mount the secrets, I can bash into the pod if needed.</p>
<p>I am using multi stage docker file made of</p>
<pre><code>From mcr.microsoft.com/dotnet/sdk:5.0 AS build 
FROM mcr.microsoft.com/dotnet/aspnet:5.0 AS runtime
</code></pre>
<p>Thanks</p>

<p>Looks like they key itself might not be correctly visible to the pod. I would start by getting into the pod with <code>kubectl exec --stdin --tty &lt;podname&gt; -- /bin/bash</code> and ensuring that the <code>/var/key.json</code> (per your config) is accessible and has the correct credentials.</p>
<p>The following would be a good way to mount the secret:</p>
<pre><code>volumeMounts:
- name: google-cloud-key
  mountPath: /var/run/secret/cloud.google.com
env:
- name: GOOGLE_APPLICATION_CREDENTIALS
  value: /var/run/secret/cloud.google.com/key.json
</code></pre>
<p>The above assumes your secret was created with a command like:</p>
<pre><code>kubectl --namespace &lt;namespace&gt; create secret generic firestore-key --from-file key.json
</code></pre>
<p>Also it is important to check your Workload Identity setup. The <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity#pod_cant_authenticate_to:%7E:text=Pod%20can%27t%20authenticate%20to%20Google%20Cloud" rel="nofollow noreferrer">Workload Identity | Kubernetes Engine Documentation</a> has a good section on this.</p>
