<p>I have a docker container python app deployed on a kubernetes cluster on Azure (I also tried on a container app). I'm trying to connect this app to Azure key vault to fetch some secrets. I created a managed identity and assigned it to both but the python app always fails to find the managed identity to even attempt connecting to the key vault.</p>
<p>The Managed Identity role assignments:</p>
<p>Key Vault Contributor -&gt; on the key vault</p>
<p>Managed Identity Operator -&gt; Managed Identity</p>
<p>Azure Kubernetes Service Contributor Role,
Azure Kubernetes Service Cluster User Role,
Managed Identity Operator -&gt; on the resource group that includes the cluster</p>
<p>Also on the key vault Access policies I added the Managed Identity and gave it access to all key, secrets, and certs permissions (for now)</p>
<p>Python code:</p>
<pre><code> credential = ManagedIdentityCredential()
 vault_client = SecretClient(vault_url=key_vault_uri, credential=credential)
 retrieved_secret = vault_client.get_secret(secret_name)
</code></pre>
<p>I keep getting the error:</p>
<pre><code>azure.core.exceptions.ClientAuthenticationError: Unexpected content type &quot;text/plain; charset=utf-8&quot;
Content: no azure identity found for request clientID 
</code></pre>
<p>So at some point I attempted to add the managed identity clientID in the cluster secrets and load it from there and still got the same error:</p>
<p>Python code:</p>
<pre><code>    def get_kube_secret(self, secret_name):
        kube_config.load_incluster_config()
        v1_secrets = kube_client.CoreV1Api()
        
        string_secret = str(v1_secrets.read_namespaced_secret(secret_name, &quot;redacted_namespace_name&quot;).data).replace(&quot;'&quot;, &quot;\&quot;&quot;)
        json_secret = json.loads(string_secret)
        return json_secret
    
    def decode_base64_string(self, encoded_string):
        decoded_secret = base64.b64decode(encoded_string.strip())
        decoded_secret = decoded_secret.decode('UTF-8')
        return decoded_secret

    managed_identity_client_id_secret = self.get_kube_secret('managed-identity-credential')['clientId']
    managed_identity_client_id = self.decode_base64_string(managed_identity_client_id_secret)
</code></pre>
<p><strong>Update:</strong></p>
<p>I also attempted to use the secret store CSI driver, but I have a feeling I'm missing a step there. Should the python code be updated to be able to use the secret store CSI driver?</p>
<pre><code># This is a SecretProviderClass using user-assigned identity to access the key vault
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: azure-kvname-user-msi
spec:
  provider: azure
  parameters:
    usePodIdentity: &quot;false&quot;
    useVMManagedIdentity: &quot;true&quot;          # Set to true for using managed identity
    userAssignedIdentityID: &quot;$CLIENT_ID&quot;   # Set the clientID of the user-assigned managed identity to use
    vmmanagedidentityclientid: &quot;$CLIENT_ID&quot;
    keyvaultName: &quot;$KEYVAULT_NAME&quot;        # Set to the name of your key vault
    cloudName: &quot;&quot;                         # [OPTIONAL for Azure] if not provided, the Azure environment defaults to AzurePublicCloud
    objects:  &quot;&quot;
    tenantId: &quot;$AZURE_TENANT_ID&quot; 
</code></pre>
<p>Deployment Yaml</p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: backend
  namespace: redacted_namespace
  labels:
    app: backend
spec:
  replicas: 1
  selector:
    matchLabels:
      app: backend
  template:
    metadata:
      labels:
        app: backend
    spec:
      containers:
        - name: backend
          image: redacted_image
          ports:
            - name: http
              containerPort: 80
            - name: https
              containerPort: 443
          imagePullPolicy: Always
          resources:
            # You must specify requests for CPU to autoscale
            # based on CPU utilization
            requests:
              cpu: &quot;250m&quot;
          env:
            - name:  test-secrets
              valueFrom:
                secretKeyRef:
                  name:  test-secrets
                  key:  test-secrets
          volumeMounts:
            - name: test-secrets
              mountPath: &quot;/mnt/secrets-store&quot;
              readOnly: true
      volumes:
        - name: test-secrets
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: &quot;azure-kvname-user-msi&quot;
      dnsPolicy: ClusterFirst
</code></pre>
<p><strong>Update 16/01/2023</strong></p>
<p>I followed the steps in the answers and the linked docs to the letter, even contacted Azure support and followed it step by step with them on the phone and the result is still the following error:</p>
<p><code>&quot;failed to process mount request&quot; err=&quot;failed to get objectType:secret, objectName:MongoUsername, objectVersion:: azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to https://&lt;RedactedVaultName&gt;.vault.azure.net/secrets/&lt;RedactedSecretName&gt;/?api-version=2016-10-01: StatusCode=400 -- Original Error: adal: Refresh request failed. Status Code = '400'. Response body: {\&quot;error\&quot;:\&quot;invalid_request\&quot;,\&quot;error_description\&quot;:\&quot;Identity not found\&quot;} Endpoint http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&amp;client_id=&lt;RedactedClientId&gt;&amp;resource=https%3A%2F%2Fvault.azure.net&quot;</code></p>

<p>Using the <a href="https://secrets-store-csi-driver.sigs.k8s.io/topics/set-as-env-var.html" rel="nofollow noreferrer">Secrets Store CSI Driver</a>, you can configure the <code>SecretProviderClass</code> to use a <a href="https://learn.microsoft.com/azure/aks/workload-identity-overview" rel="nofollow noreferrer">workload identity</a> by setting the <code>clientID</code> in the <code>SecretProviderClass</code>. You'll need to use the client ID of your <a href="https://learn.microsoft.com/azure/active-directory/managed-identities-azure-resources/overview?source=recommendations#managed-identity-types" rel="nofollow noreferrer">user assigned managed identity</a> and change the <code>usePodIdentity</code> and <code>useVMManagedIdentity</code> setting to <code>false</code>.</p>
<p>With this approach, you don't need to add any additional code in your app to retrieve the secrets. Instead, you can mount a secrets store (using CSI driver) as a volume mount in your pod and have secrets loaded as environment variables which is documented <a href="https://azure.github.io/secrets-store-csi-driver-provider-azure/docs/configurations/sync-with-k8s-secrets/" rel="nofollow noreferrer">here</a>.</p>
<p>This <a href="https://learn.microsoft.com/azure/aks/csi-secrets-store-identity-access#configure-workload-identity" rel="nofollow noreferrer">doc</a> will walk you through setting it up on Azure, but at a high-level here is what you need to do:</p>
<ol>
<li>Register the <code>EnableWorkloadIdentityPreview</code> feature using Azure CLI</li>
<li>Create an AKS cluster using Azure CLI with the <code>azure-keyvault-secrets-provider</code> add-on enabled and <code>--enable-oidc-issuer</code> and <code>--enable-workload-identiy</code> flags set</li>
<li>Create an Azure Key Vault and set your secrets</li>
<li>Create an Azure User Assigned Managed Identity and set an access policy on the key vault for the the managed identity' client ID</li>
<li>Connect to the AKS cluster and create a Kubernetes <code>ServiceAccount</code> with annotations and labels that enable this for Azure workload identity</li>
<li>Create an Azure identity federated credential for the managed identity using the AKS cluster's OIDC issuer URL and Kubernetes ServiceAccount as the subject</li>
<li>Create a Kubernetes <code>SecretProviderClass</code> using <code>clientID</code> to use workload identity and adding a <code>secretObjects</code> block to enable syncing objects as environment variables using Kubernetes secret store.</li>
<li>Create a Kubernetes <code>Deployment</code> with a <code>label</code> to use workload identity, the <code>serviceAccountName</code> set to the service account you created above, volume using CSI and the secret provider class you created above, volumeMount, and finally environment variables in your container using <code>valueFrom</code> and <code>secretKeyRef</code> syntax to mount from your secret object store.</li>
</ol>
<p>Hope that helps.</p>
