<p>A couple of weeks ago i published similar question regarding a Kubernetes deployment that uses Key Vault (with User Assigned Managed identity method). The issue was resolved but when trying to implemente everything from scratch something makes not sense to me.</p>
<p>Basically i am getting this error regarding mounting volume:</p>
<pre><code>Volumes:
  sonar-data-new:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  sonar-data-new
    ReadOnly:   false
  sonar-extensions-new2:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  sonar-extensions-new2
    ReadOnly:   false
  secrets-store-inline:
    Type:              CSI (a Container Storage Interface (CSI) volume source)
    Driver:            secrets-store.csi.k8s.io
    FSType:
    ReadOnly:          true
    VolumeAttributes:      secretProviderClass=azure-kv-provider
  default-token-zwxzg:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-zwxzg
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  &lt;none&gt;
Tolerations:     node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason       Age               From               Message
  ----     ------       ----              ----               -------
  Normal   Scheduled    12s               default-scheduler  Successfully assigned default/sonarqube-d44d498f8-46mpz to aks-agentpool-35716862-vmss000000
  Warning  FailedMount  3s (x5 over 11s)  kubelet            MountVolume.SetUp failed for volume &quot;secrets-store-inline&quot; : rpc error: code = Unknown desc = failed to mount secrets store objects for pod default/sonarqube-d44d498f8-46mpz, err: rpc error: code = Unknown desc = failed to mountobjects, error: failed to get objectType:secret, objectName:username, objectVersion:: azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to https://SonarQubeHelm.vault.azure.net/secrets/username/?api-version=2016-10-01: StatusCode=400 -- Original Error: adal: Refresh request failed. Status Code = '400'. Response body: {&quot;error&quot;:&quot;invalid_request&quot;,&quot;error_description&quot;:&quot;Identity not found&quot;}
</code></pre>
<p>This is my <strong>secret-class.yml</strong> file (name of the keyvault is correct). Also <strong>xxx-xxxx-xxx-xxx-xxxxx4b5ec83</strong> is the <strong>objectID</strong> of the AKS managed identity (<strong>SonarQubeHelm-agentpool</strong>)</p>
<pre><code>apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
  name: azure-kv-provider
spec:
  provider: azure
  secretObjects:
   - data:
      - key: username
        objectName: username
      - key: password
        objectName: password
     secretName: test-secret
     type: Opaque
  parameters:
    usePodIdentity: &quot;false&quot;
    useVMManagedIdentity: &quot;true&quot;
    userAssignedIdentityID: &quot;xxx-xxxx-xxx-xxx-xxxxx4b5ec83&quot;
    keyvaultName: &quot;SonarQubeHelm&quot;
    cloudName: &quot;&quot;
    objects:  |
      array:
        - |
          objectName: username
          objectType: secret
          objectAlias: username
          objectVersion: &quot;&quot;
        - |
          objectName: password
          objectType: secret
          objectAlias: password
          objectVersion: &quot;&quot;
    resourceGroup: &quot;rg-LD-sandbox&quot;
    subscriptionId: &quot;xxxx&quot;
    tenantId: &quot;yyyy&quot;
</code></pre>
<p>and this is my <strong>deployment.yml</strong> file</p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: sonarqube
  name: sonarqube
spec:
  selector:
    matchLabels:
      app: sonarqube
  replicas: 1
  template:
    metadata:
      labels:
        app: sonarqube
    spec:
      containers:
        - name: sonarqube
          image: sonarqube:8.9-developer
          resources:
            requests:
              cpu: 500m
              memory: 1024Mi
            limits:
              cpu: 2000m
              memory: 4096Mi
          volumeMounts:
          - mountPath: &quot;/mnt/&quot;
            name: secrets-store-inline
          - mountPath: &quot;/opt/sonarqube/data/&quot;
            name: sonar-data-new
          - mountPath: &quot;/opt/sonarqube/extensions/plugins/&quot;
            name: sonar-extensions-new2
          env:
          - name: &quot;SONARQUBE_JDBC_USERNAME&quot;
            valueFrom:
              secretKeyRef:
                name: test-secret
                key: username
          - name: &quot;SONARQUBE_JDBC_PASSWORD&quot;
            valueFrom:
              secretKeyRef:
                name: test-secret
                key: password
          - name: &quot;SONARQUBE_JDBC_URL&quot;
            valueFrom:
              configMapKeyRef:
                name: sonar-config
                key: url
          ports:
          - containerPort: 9000
            protocol: TCP
      volumes:
      - name: sonar-data-new
        persistentVolumeClaim:
          claimName: sonar-data-new
      - name: sonar-extensions-new2
        persistentVolumeClaim:
          claimName: sonar-extensions-new2
      - name: secrets-store-inline
        csi:
          driver: secrets-store.csi.k8s.io
          readOnly: true
          volumeAttributes:
           secretProviderClass: &quot;azure-kv-provider&quot;
</code></pre>
<p>I assigned proper permissions to the AKS managed identity to get access to the keyvault (<strong>xxx-xxxx-xxx-xxx-xxxxx4b5ec83</strong> is the <strong>objectID</strong> of the AKS managed identity - <strong>SonarQubeHelm-agentpool</strong>)</p>
<pre><code>xxxx@Azure:~/clouddrive/kubernetes/sonarqubekeyvault$ az role assignment list --assignee xxx-xxxx-xxx-xxx-xxxxx4b5ec83 --all
[
  {
    &quot;canDelegate&quot;: null,
    &quot;condition&quot;: null,
    &quot;conditionVersion&quot;: null,
    &quot;description&quot;: null,
    &quot;id&quot;: &quot;/subscriptions/xxxx-xxx-xxx-xxx-xxxe22e8804e/resourceGroups/rg-LD-sandbox/providers/Microsoft.KeyVault/vaults/SonarQubeHelm/providers/Microsoft.Authorization/roleAssignments/xxxx-x-xx-xx-xx86584218f&quot;,
    &quot;name&quot;: &quot;xxxx-xx-x-x-xx86584218f&quot;,
    &quot;principalId&quot;: &quot;xxx-xxx-x-xxx-xx3a4b5ec83&quot;,
    &quot;principalName&quot;: &quot;xxxx-xxxx-xxx-xxx-xx79a3906b8&quot;,
    &quot;principalType&quot;: &quot;ServicePrincipal&quot;,
    &quot;resourceGroup&quot;: &quot;rg-LD-sandbox&quot;,
    &quot;roleDefinitionId&quot;: &quot;/subscriptions/xxxx-xxxx-xxxx-xxx-0e1e22e8804e/providers/Microsoft.Authorization/roleDefinitions/xxx-xxxx-xxx-xxxx-xxxfe8e74483&quot;,
    &quot;roleDefinitionName&quot;: &quot;Key Vault Administrator&quot;,
    &quot;scope&quot;: &quot;/subscriptions/xxxx-xxx-xxx-xxxx-xxx2e8804e/resourceGroups/rg-LD-sandbox/providers/Microsoft.KeyVault/vaults/SonarQubeHelm&quot;,
    &quot;type&quot;: &quot;Microsoft.Authorization/roleAssignments&quot;
  },
  {
    &quot;canDelegate&quot;: null,
    &quot;condition&quot;: null,
    &quot;conditionVersion&quot;: null,
    &quot;description&quot;: null,
    &quot;id&quot;: &quot;/subscriptions/xxxx-xxxx-xxxx-xxxx-xxxe22e8804e/resourceGroups/rg-LD-sandbox/providers/Microsoft.KeyVault/vaults/SonarQubeHelm/providers/Microsoft.Authorization/roleAssignments/xxxx-xxxx-xxxx-xxxx-xxx5137f480&quot;,
    &quot;name&quot;: &quot;xxxx-xxxx-xxxx-xxxx-xx5137f480&quot;,
    &quot;principalId&quot;: &quot;xxxx-xxxx-xxxx-xxxx-xx3a4b5ec83&quot;,
    &quot;principalName&quot;: &quot;xxxx-xxxx-xxxx-xxxx-xx79a3906b8&quot;,
    &quot;principalType&quot;: &quot;ServicePrincipal&quot;,
    &quot;resourceGroup&quot;: &quot;rg-LD-sandbox&quot;,
    &quot;roleDefinitionId&quot;: &quot;/subscriptions/xxxx-xxxx-xxxx-xxxx-0e1e22e8804e/providers/Microsoft.Authorization/roleDefinitions/xxxx-xxxx-xxxx-xxxx-xx2c155cd7&quot;,
    &quot;roleDefinitionName&quot;: &quot;Key Vault Secrets Officer&quot;,
    &quot;scope&quot;: &quot;/subscriptions/xxxx/resourceGroups/rg-LD-sandbox/providers/Microsoft.KeyVault/vaults/SonarQubeHelm&quot;,
    &quot;type&quot;: &quot;Microsoft.Authorization/roleAssignments&quot;
  }
]
</code></pre>
<p>This is the info about my Key Vault.</p>
<p><strong>az keyvault show --name SonarQubeHelm</strong></p>
<pre><code>{
  &quot;id&quot;: &quot;/subscriptions/xxxx-xxxxx-xxxx-xxxx-xxxxxxxx804e/resourceGroups/rg-LD-sandbox/providers/Microsoft.KeyVault/vaults/SonarQubeHelm&quot;,
  &quot;location&quot;: &quot;xxxxx&quot;,
  &quot;name&quot;: &quot;SonarQubeHelm&quot;,
  &quot;properties&quot;: {
    &quot;accessPolicies&quot;: [
      {
        &quot;applicationId&quot;: null,
        &quot;objectId&quot;: &quot;xxxx-xxx-xxxx-xxxx-xxxxa4b5ec83&quot;,
        &quot;permissions&quot;: {
          &quot;certificates&quot;: [
            &quot;Get&quot;,
            &quot;List&quot;,
            &quot;Update&quot;,
            &quot;Create&quot;,
            &quot;Import&quot;,
            &quot;Delete&quot;,
            &quot;Recover&quot;,
            &quot;Backup&quot;,
            &quot;Restore&quot;,
            &quot;ManageContacts&quot;,
            &quot;ManageIssuers&quot;,
            &quot;GetIssuers&quot;,
            &quot;ListIssuers&quot;,
            &quot;SetIssuers&quot;,
            &quot;DeleteIssuers&quot;,
            &quot;Purge&quot;
          ],
          &quot;keys&quot;: [
            &quot;Get&quot;,
            &quot;List&quot;,
            &quot;Update&quot;,
            &quot;Create&quot;,
            &quot;Import&quot;,
            &quot;Delete&quot;,
            &quot;Recover&quot;,
            &quot;Backup&quot;,
            &quot;Restore&quot;,
            &quot;Decrypt&quot;,
            &quot;Encrypt&quot;,
            &quot;UnwrapKey&quot;,
            &quot;WrapKey&quot;,
            &quot;Verify&quot;,
            &quot;Sign&quot;,
            &quot;Purge&quot;
          ],
          &quot;secrets&quot;: [
            &quot;Get&quot;,
            &quot;List&quot;,
            &quot;Set&quot;,
            &quot;Delete&quot;,
            &quot;Recover&quot;,
            &quot;Backup&quot;,
            &quot;Restore&quot;,
            &quot;Purge&quot;
          ],
          &quot;storage&quot;: null
        },
        &quot;tenantId&quot;: &quot;xxxx-xxxx-xxxx-xxxx-xxxxxdb8c610&quot;
      },
      {
        &quot;applicationId&quot;: null,
        &quot;objectId&quot;: &quot;xxxx-xxxx-xxxx-xxxx-xxxx531f67f8&quot;,
        &quot;permissions&quot;: {
          &quot;certificates&quot;: [
            &quot;Get&quot;,
            &quot;List&quot;,
            &quot;Update&quot;,
            &quot;Create&quot;,
            &quot;Import&quot;,
            &quot;Delete&quot;,
            &quot;Recover&quot;,
            &quot;Backup&quot;,
            &quot;Restore&quot;,
            &quot;ManageContacts&quot;,
            &quot;ManageIssuers&quot;,
            &quot;GetIssuers&quot;,
            &quot;ListIssuers&quot;,
            &quot;SetIssuers&quot;,
            &quot;DeleteIssuers&quot;,
            &quot;Purge&quot;
          ],
          &quot;keys&quot;: [
            &quot;Get&quot;,
            &quot;List&quot;,
            &quot;Update&quot;,
            &quot;Create&quot;,
            &quot;Import&quot;,
            &quot;Delete&quot;,
            &quot;Recover&quot;,
            &quot;Backup&quot;,
            &quot;Restore&quot;,
            &quot;Decrypt&quot;,
            &quot;Encrypt&quot;,
            &quot;UnwrapKey&quot;,
            &quot;WrapKey&quot;,
            &quot;Verify&quot;,
            &quot;Sign&quot;,
            &quot;Purge&quot;
          ],
          &quot;secrets&quot;: [
            &quot;Get&quot;,
            &quot;List&quot;,
            &quot;Set&quot;,
            &quot;Delete&quot;,
            &quot;Recover&quot;,
            &quot;Backup&quot;,
            &quot;Restore&quot;,
            &quot;Purge&quot;
          ],
          &quot;storage&quot;: null
        },
        &quot;tenantId&quot;: &quot;xxxxx-xxxxxx-xxx-xxx8db8c610&quot;
      },
      {
        &quot;applicationId&quot;: null,
        &quot;objectId&quot;: &quot;xxx-xxxx-xxxx-xxx-xxxx0df6af9&quot;,
        &quot;permissions&quot;: {
          &quot;certificates&quot;: [
            &quot;Get&quot;,
            &quot;List&quot;,
            &quot;Update&quot;,
            &quot;Create&quot;,
            &quot;Import&quot;,
            &quot;Delete&quot;,
            &quot;Recover&quot;,
            &quot;Backup&quot;,
            &quot;Restore&quot;,
            &quot;ManageContacts&quot;,
            &quot;ManageIssuers&quot;,
            &quot;GetIssuers&quot;,
            &quot;ListIssuers&quot;,
            &quot;SetIssuers&quot;,
            &quot;DeleteIssuers&quot;
          ],
          &quot;keys&quot;: [
            &quot;Get&quot;,
            &quot;List&quot;,
            &quot;Update&quot;,
            &quot;Create&quot;,
            &quot;Import&quot;,
            &quot;Delete&quot;,
            &quot;Recover&quot;,
            &quot;Backup&quot;,
            &quot;Restore&quot;
          ],
          &quot;secrets&quot;: [
            &quot;Get&quot;,
            &quot;List&quot;,
            &quot;Set&quot;,
            &quot;Delete&quot;,
            &quot;Recover&quot;,
            &quot;Backup&quot;,
            &quot;Restore&quot;
          ],
          &quot;storage&quot;: null
        },
        &quot;tenantId&quot;: &quot;xxx-xxx-xxx-xxx-xxx8db8c610&quot;
      }
    ],
    &quot;createMode&quot;: null,
    &quot;enablePurgeProtection&quot;: null,
    &quot;enableRbacAuthorization&quot;: false,
    &quot;enableSoftDelete&quot;: true,
    &quot;enabledForDeployment&quot;: false,
    &quot;enabledForDiskEncryption&quot;: false,
    &quot;enabledForTemplateDeployment&quot;: false,
    &quot;hsmPoolResourceId&quot;: null,
    &quot;networkAcls&quot;: null,
    &quot;privateEndpointConnections&quot;: null,
    &quot;provisioningState&quot;: &quot;Succeeded&quot;,
    &quot;sku&quot;: {
      &quot;family&quot;: &quot;A&quot;,
      &quot;name&quot;: &quot;Standard&quot;
    },
    &quot;softDeleteRetentionInDays&quot;: 90,
    &quot;tenantId&quot;: &quot;xxx-xxx-xxx-xxx-xxx68db8c610&quot;,
    &quot;vaultUri&quot;: &quot;https://sonarqubehelm.vault.azure.net/&quot;
  },
  &quot;resourceGroup&quot;: &quot;rg-LD-sandbox&quot;,
  &quot;systemData&quot;: null,
  &quot;tags&quot;: {},
  &quot;type&quot;: &quot;Microsoft.KeyVault/vaults&quot;
}
</code></pre>
<p>This is the <strong>CSI pod</strong> running at the moment:</p>
<pre><code>NAME                                                READY   STATUS              RESTARTS   AGE
csi-secrets-store-provider-azure-1632148185-hggl4   1/1     Running             0          5h4m
ingress-nginx-controller-65c4f84996-99pkh           1/1     Running             0          5h49m
secrets-store-csi-driver-xsx2r                      3/3     Running             0          5h4m
sonarqube-d44d498f8-46mpz                           0/1     ContainerCreating   0          26m
</code></pre>
<p>I used this <strong>CSI driver</strong></p>
<p><em>helm install csi-secrets-store-provider-azure/csi-secrets-store-provider-azure --set secrets-store-csi-driver.syncSecret.enabled=true --generate-name</em></p>
<p>To assign proper permissions to the AKS managed identity i followed (just in case i used clientID as well, but it does not work). However, from the previous commands permissions to managed identity seems correct.</p>
<pre><code>export KUBE_ID=$(az aks show -g &lt;resource group&gt; -n &lt;aks cluster name&gt; --query identityProfile.kubeletidentity.objectId -o tsv)
export AKV_ID=$(az keyvault show -g &lt;resource group&gt; -n &lt;akv name&gt; --query id -o tsv)
az role assignment create --assignee $KUBE_ID --role &quot;Key Vault Secrets Officer&quot; --scope $AKV_ID
</code></pre>
<p>In <strong>Access Policies</strong> in the <strong>key vault</strong> &quot;SonarQubeHelm&quot; these are the applications added with all the permission: <strong>SonarQubeHelm-agentpool</strong> and <strong>SonarQubeHelm</strong></p>
<p>The key values in <strong>Secrets</strong> (inside Key Vault) are <strong>username</strong> and <strong>password</strong>.</p>
<p>Everything is on the same region, resource group and namespace (default) and I am working with 1 node cluster.</p>
<p>Any idea about this error?</p>
<p>Thanks in advance!</p>

<p>After doing some tests, it seems that the process that I was following was correct. Most probably, I was using <code>principalId</code> instead of <code>clientId</code> in role assignment for the AKS managed identity.</p>
<p>Key points for someone else that is facing similar issues:</p>
<ol>
<li><p>Check what the managed identity created automatically by AKS is. Check for the <code>clientId</code>; e.g.,</p>
<pre><code>az vmss identity show -g MC_rg-LX-sandbox_SonarQubeHelm_southcentralus -n 
aks-agentpool-xxxxx62-vmss -o yaml
</code></pre>
<blockquote>
<p><em>Remember that <code>MC_**</code> is the resource group that AKS creates automatically to keep AKS resources.</em> VMSS is the Virtual Machine Scale Set; you can get it under the same resource group.</p>
</blockquote>
</li>
<li><p>Check if it has correct permissions to access the Key Vault that you created: e.g., (where <em>xxxx-xxxx-xxx-xxx-xx79a3906b8</em> is the managed identity <code>clientId</code>):</p>
<pre><code>az role assignment list --assignee xxxx-xxxx-xxx-xxx-xx79a3906b8 --all
</code></pre>
<p>It should have:</p>
<pre><code>&quot;roleDefinitionName&quot;: &quot;Key Vault Administrator&quot;
</code></pre>
</li>
<li><p>If it doesn't have correct permissions, assign them:</p>
<pre><code>export KUBE_ID=$(az aks show -g &lt;resource group&gt; -n &lt;aks cluster name&gt; --query identityProfile.kubeletidentity.clientId -o tsv)
export AKV_ID=$(az keyvault show -g &lt;resource group&gt; -n &lt;akv name&gt; --query id -o tsv)
az role assignment create --assignee $KUBE_ID --role &quot;Key Vault Secrets Officer&quot; --scope $AKV_ID
</code></pre>
</li>
</ol>
<p>The role assignments for the managed identity in AKS works for <code>clientId</code>.</p>
