<p>I'm trying to provision an encrypted disk for GKE dynamically.
But I really don't understand below part.</p>

<pre><code>Grant permission to use the key
You must assign the Compute Engine service account used by nodes in your cluster the Cloud KMS CryptoKey Encrypter/Decrypter role. This is required for GKE Persistent Disks to access and use your encryption key.

The Compute Engine service account's name has the following format:

service-[PROJECT_NUMBER]@compute-system.iam.gserviceaccount.com
</code></pre>

<p>Does it really necessary to grant "Cloud KMS CryptoKey Encrypter/Decryter" to Compute Engine Service account? Can I create a new SA and grant this role to it? The description said, the SA used by nodes. So I'm wondering if I can create a new SA and grant Cloud KMS role then use this SA to spin up GKE cluster. Then I think it should be available to provision encrypted disks for GKE.</p>

<p>official document below:</p>

<p><a href="https://cloud.google.com/kubernetes-engine/docs/how-to/using-cmek#dynamically_provision_an_encrypted" rel="nofollow noreferrer">dynamically_provision_an_encrypted</a></p>

Dust break
<p>I tried to follow this <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/dynamic-provisioning-cmek" rel="nofollow noreferrer">documentation</a> step by step: </p>

<ol>
<li><p>create gke cluster (<a href="https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver" rel="nofollow noreferrer">check</a> Kubernetes Compatibility compatibility, I decided to stick with 1.14 this time), key-ring and key</p></li>
<li><p>deploy CSI driver to the cluster</p>

<p>2.1. download driver <code>$git clone https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver /PATH/gcp-compute-persistent-disk-csi-driver</code></p>

<p>2.2. configure variables for your project in <code>/PATH/gcp-compute-persistent-disk-csi-driver/deploy/setup-project.sh</code></p>

<p>2.3. create service account with <code>/PATH/gcp-compute-persistent-disk-csi-driver/deploy/setup-project.sh</code></p></li>
</ol>

<p>2.4. configure variables for driver deployment in <code>/PATH/gcp-compute-persistent-disk-csi-driver/deploy/kubernetes/deploy-driver.sh</code> and  <code>/PATH/gcp-compute-persistent-disk-csi-driver/deploy/kubernetes/deploy-driver.shinstall-kustomize.sh</code></p>

<p>2.5. deploy CSI driver (I stick with stable version)</p>

<pre><code>$./deploy-driver.sh
</code></pre>

<ol start="3">
<li><p>enable the Cloud KMS API</p></li>
<li><p>assign the <code>Cloud KMS CryptoKey Encrypter/Decrypter</code> role (<code>roles/cloudkms.cryptoKeyEncrypterDecrypter</code>) to the <code>Compute Engine Service Agent</code> (<code>service-[PROJECT_NUMBER]@compute-system.iam.gserviceaccount.com</code>)</p></li>
<li><p>create StorageClass</p>

<pre><code>$cat storage.yaml    
apiVersion: storage.k8s.io/v1beta1
kind: StorageClass
metadata:
  name: csi-gce-pd
provisioner: pd.csi.storage.gke.io
parameters:
  type: pd-standard
  disk-encryption-kms-key: projects/test-prj/locations/europe-west3/keyRings/TEST-KEY-RING/cryptoKeys/TEST-KEY


$kubectl describe storageclass csi-gce-pd
Name:            csi-gce-pd
IsDefaultClass:  No
Annotations:     kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"storage.k8s.io/v1beta1","kind":"StorageClass","metadata":{"annotations":{},"name":"csi-gce-pd"},"parameters":{"disk-encryption-kms-key":"projects/test-prj/locations/europe-west3/keyRings/TEST-KEY-RING/cryptoKeys/TEST-KEY","type":"pd-standard"},"provisioner":"pd.csi.storage.gke.io"}

Provisioner:           pd.csi.storage.gke.io
Parameters:            disk-encryption-kms-key=projects/test-prj/locations/europe-west3/keyRings/TEST-KEY-RING/cryptoKeys/TEST-KEY,type=pd-standard
AllowVolumeExpansion:  &lt;unset&gt;
MountOptions:          &lt;none&gt;
ReclaimPolicy:         Delete
VolumeBindingMode:     Immediate
Events:                &lt;none&gt;
</code></pre></li>
<li><p>create persistent volume</p>

<p>$kubectl apply -f pvc.yaml<br>
persistentvolumeclaim/podpvc created</p>

<p>$kubectl describe pvc podpvc
Name:          podpvc
Namespace:     default
StorageClass:  csi-gce-pd
Status:        Bound
Volume:        pvc-b383584a-32c5-11ea-ad6e-42010a9c007d
Labels:        
Annotations:</p>

<pre><code>   kubectl.kubernetes.io/last-applied-configuration:
             {"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"podpvc","namespace":"default"},"spec":{"accessModes...
           pv.kubernetes.io/bind-completed: yes
           pv.kubernetes.io/bound-by-controller: yes
           volume.beta.kubernetes.io/storage-provisioner: pd.csi.storage.gke.io
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      6Gi
Access Modes:  RWO
VolumeMode:    Filesystem
Mounted By:    &lt;none&gt;
Events:
  Type    Reason                 Age                From                                                                                                           Message
  ----    ------                 ----               ----                                                                                                           -------
  Normal  Provisioning           31m                pd.csi.storage.gke.io_gke-test-cluster-default-pool-cd22e088-t1h0_c158f4fc-07ba-411e-8a94-74595f2b2f1d  External provisioner is provisioning volume for claim "default/podpvc"
  Normal  ExternalProvisioning   31m (x2 over 31m)  persistentvolume-controller                                                                                    waiting for a volume to be created, either by external provisioner "pd.csi.storage.gke.io" or manually created by system administrator
  Normal  ProvisioningSucceeded  31m                pd.csi.storage.gke.io_gke-test-cluster-default-pool-cd22e088-t1h0_c158f4fc-07ba-411e-8a94-74595f2b2f1d  Successfully provisioned volume pvc-b383584a-32c5-11ea-ad6e-42010a9c007d
</code></pre></li>
</ol>

<p>And it's successfully provisioned.</p>

<p>Then I removed <code>Cloud KMS CryptoKey Encrypter/Decrypter</code> role from the <code>Compute Engine Service Agent</code> and persistent volume created at step 6 and tried again:</p>

<pre><code>$kubectl apply -f pvc.yaml                                
persistentvolumeclaim/podpvc created

$kubectl describe pvc podpvc
Name:          podpvc
Namespace:     default
StorageClass:  csi-gce-pd
Status:        Pending
Volume:        
Labels:        &lt;none&gt;
Annotations:   kubectl.kubernetes.io/last-applied-configuration:
             {"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"podpvc","namespace":"default"},"spec":{"accessModes...
           volume.beta.kubernetes.io/storage-provisioner: pd.csi.storage.gke.io
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Filesystem
Mounted By:    &lt;none&gt;
Events:
Type     Reason                Age                   From                                                                                                           Message
----     ------                ----                  ----                                                                                                           -------
Normal   Provisioning          2m15s (x10 over 11m)  pd.csi.storage.gke.io_gke-serhii-test-cluster-default-pool-cd22e088-t1h0_c158f4fc-07ba-411e-8a94-74595f2b2f1d  External provisioner is provisioning volume for claim "default/podpvc"
Warning  ProvisioningFailed    2m11s (x10 over 11m)  pd.csi.storage.gke.io_gke-serhii-test-cluster-default-pool-cd22e088-t1h0_c158f4fc-07ba-411e-8a94-74595f2b2f1d  failed to provision volume with StorageClass "csi-gce-pd": rpc error: code = Internal desc = CreateVolume failed to create single zonal disk "pvc-b1a238b5-35fa-11ea-bec8-42010a9c01e6": failed to insert zonal disk: unkown Insert disk error: googleapi: Error 400: Cloud KMS error when using key projects/serhii-test-prj/locations/europe-west3/keyRings/SERHII-TEST-KEY-RING/cryptoKeys/SERHII-TEST-KEY: Permission 'cloudkms.cryptoKeyVersions.useToEncrypt' denied on resource 'projects/serhii-test-prj/locations/europe-west3/keyRings/SERHII-TEST-KEY-RING/cryptoKeys/SERHII-TEST-KEY' (or it may not exist)., kmsPermissionDenied
Normal   ExternalProvisioning  78s (x43 over 11m)    persistentvolume-controller                                                                                    waiting for a volume to be created, either by external provisioner "pd.csi.storage.gke.io" or manually created by system administrator
</code></pre>

<p>and persistent volume stayed in pending status.</p>

<p>And, as you can see, in the <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/dynamic-provisioning-cmek" rel="nofollow noreferrer">documentation</a> it's necessary:</p>

<blockquote>
  <p><strong>Grant permission to use the key</strong></p>
  
  <p>You must assign the Compute Engine service account used by nodes in
  your cluster the Cloud KMS CryptoKey Encrypter/Decrypter role. This is
  required for GKE Persistent Disks to access and use your encryption
  key.</p>
</blockquote>

<p>and it's not enough to create service account with <code>/PATH/gcp-compute-persistent-disk-csi-driver/deploy/setup-project.sh</code> provided by CSI driver. </p>

<p><strong>EDIT</strong> Please notice that:</p>

<blockquote>
  <p>For CMEK-protected node boot disks, this Compute Engine service
  account is the account which requires permissions to do encryption
  using your Cloud KMS key. This is true even if you are using a custom
  service account on your nodes.</p>
</blockquote>

<p>So, there's no way to use only service account in this case without Compute Engine service account, because CMEK-protected persistent volumes are managed by GCE, not by GKE. Meanwhile, you can provide only necessary perdition to your custom service account to improve security of your project. </p>
