<p>I need to resize a bunch of PVCs. It seems the easiest way to do it is through
the ExpandPersistentVolumes feature. I am however having trouble getting the
configuration to cooperate.</p>

<p>The <code>ExpandPersistentVolumes</code> feature gate is set in kubelet on all three
masters, as shown:</p>

<p>(output trimmed to relevant bits for sanity)</p>

<pre><code>$ parallel-ssh -h /tmp/masters -P "ps aux | grep feature"
172.20.53.249: root     15206  7.4  0.5 619888 83952 ?        Ssl  19:52   0:02 /opt/kubernetes/bin/kubelet  --feature-gates=ExpandPersistentVolumes=true,ExperimentalCriticalPodAnnotation=true
[1] 12:53:08 [SUCCESS] 172.20...
172.20.58.111: root     17798  4.5  0.5 636280 87328 ?        Ssl  19:51   0:04 /opt/kubernetes/bin/kubelet --feature-gates=ExpandPersistentVolumes=true,ExperimentalCriticalPodAnnotation=true
[2] 12:53:08 [SUCCESS] 172.20...
172.20.53.240: root      9287  4.0  0.5 645276 90528 ?        Ssl  19:50   0:06 /opt/kubernetes/bin/kubelet --feature-gates=ExpandPersistentVolumes=true,ExperimentalCriticalPodAnnotation=true
[3] 12:53:08 [SUCCESS] 172.20..
</code></pre>

<p>The apiserver has the <code>PersistentVolumeClaimResize</code> admission controller, as shown:</p>

<pre><code>$ kubectl --namespace=kube-system get pod -o yaml | grep -i admission                                                                                                                                                           
    /usr/local/bin/kube-apiserver --admission-control=Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,NodeRestriction,PersistentVolumeClaimResize,ResourceQuota
    /usr/local/bin/kube-apiserver --admission-control=Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,NodeRestriction,PersistentVolumeClaimResize,ResourceQuota
    /usr/local/bin/kube-apiserver --admission-control=Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,NodeRestriction,PersistentVolumeClaimResize,ResourceQuota
</code></pre>

<p>However, when I create or edit a storage class to add <code>allowVolumeExpansion</code>, 
it is removed on save. For example:</p>

<pre><code>$ cat new-sc.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  creationTimestamp: null
  labels:
    k8s-addon: storage-aws.addons.k8s.io
  name: gp2-2
  selfLink: /apis/storage.k8s.io/v1/storageclasses/gp2
parameters:
  encrypted: "true"
  kmsKeyId: arn:aws:kms:us-west-2:&lt;omitted&gt;
  type: gp2
  zone: us-west-2a
provisioner: kubernetes.io/aws-ebs
reclaimPolicy: Delete
allowVolumeExpansion: true

$ kubectl create -f new-sc.yaml
storageclass "gp2-2" created

$ kubectl get sc gp2-2 -o yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  creationTimestamp: 2018-05-22T20:00:17Z
  labels:
    k8s-addon: storage-aws.addons.k8s.io
  name: gp2-2
  resourceVersion: "2546166"
  selfLink: /apis/storage.k8s.io/v1/storageclasses/gp2-2
  uid: &lt;omitted&gt;
parameters:
  encrypted: "true"
  kmsKeyId: arn:aws:kms:us-west-2:&lt;omitted&gt;
  type: gp2
  zone: us-west-2a
provisioner: kubernetes.io/aws-ebs
reclaimPolicy: Delete
</code></pre>

<p>What am I missing? What is erasing this key from my storageclass configuration?</p>

<p><strong>EDIT:</strong> Here is the command used by the kube-apiserver pods. It does not say anything about feature gates. The cluster was launched using Kops.</p>

<pre><code>- /bin/sh
    - -c
    - mkfifo /tmp/pipe; (tee -a /var/log/kube-apiserver.log &lt; /tmp/pipe &amp; ) ; exec
      /usr/local/bin/kube-apiserver --address=127.0.0.1 --admission-control=Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,NodeRestriction,PersistentVolumeClaimResize,ResourceQuota
      --allow-privileged=true --anonymous-auth=false --apiserver-count=3 --authorization-mode=RBAC
      --basic-auth-file=/srv/kubernetes/basic_auth.csv --client-ca-file=/srv/kubernetes/ca.crt
      --cloud-provider=aws --etcd-cafile=/srv/kubernetes/ca.crt --etcd-certfile=/srv/kubernetes/etcd-client.pem
      --etcd-keyfile=/srv/kubernetes/etcd-client-key.pem --etcd-servers-overrides=/events#https://127.0.0.1:4002
      --etcd-servers=https://127.0.0.1:4001 --insecure-port=8080 --kubelet-preferred-address-types=InternalIP,Hostname,ExternalIP
      --proxy-client-cert-file=/srv/kubernetes/apiserver-aggregator.cert --proxy-client-key-file=/srv/kubernetes/apiserver-aggregator.key
      --requestheader-allowed-names=aggregator --requestheader-client-ca-file=/srv/kubernetes/apiserver-aggregator-ca.cert
      --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group
      --requestheader-username-headers=X-Remote-User --secure-port=443 --service-cluster-ip-range=100.64.0.0/13
      --storage-backend=etcd3 --tls-cert-file=/srv/kubernetes/server.cert --tls-private-key-file=/srv/kubernetes/server.key
      --token-auth-file=/srv/kubernetes/known_tokens.csv --v=1 &gt; /tmp/pipe 2&gt;&amp;1
</code></pre>

<p>It could happen if you did not enable alpha feature-gate for the option.</p>

<p>Did you set <code>--feature-gates</code> option for <a href="https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/" rel="nofollow noreferrer">kube-apiserver</a>?</p>

<pre><code>--feature-gates mapStringBool  - A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:
...
ExpandPersistentVolumes=true|false (ALPHA - default=false)
...
</code></pre>

<p>Update: If you don't see this option in the command line arguments, you need to add it (<code>--feature-gates=ExpandPersistentVolumes=true</code>). </p>

<p>In case you run kube-apiserver as a pod, you should edit /<code>etc/kubernetes/manifests/kube-apiserver.yaml</code> and add the feature-gate option to other arguments. kube-apiserver will restart automatically.</p>

<p>In case you run kube-apiserver as a process maintained by systemd, you should edit <code>kube-apiserver.service</code> or service options <code>$KUBE_API_ARGS</code> in a separate file, and append feature-gate option there. Restart the service with <code>systemctl restart kube-apiserver.service</code> command.</p>

<p>After enabling it, you can create a StorageClass object with allowVolumeExpansion option:</p>

<pre><code># kubectl get sc -o yaml --export
apiVersion: v1
items:
- allowVolumeExpansion: true
  apiVersion: storage.k8s.io/v1
  kind: StorageClass
  metadata:
    creationTimestamp: 2018-05-23T14:38:43Z
    labels:
      k8s-addon: storage-aws.addons.k8s.io
    name: gp2-2
    namespace: ""
    resourceVersion: "1385"
    selfLink: /apis/storage.k8s.io/v1/storageclasses/gp2-2
    uid: fe516dcf-5e96-11e8-a86d-42010a9a0002
  parameters:
    encrypted: "true"
    kmsKeyId: arn:aws:kms:us-west-2:&lt;omitted&gt;
    type: gp2
    zone: us-west-2a
  provisioner: kubernetes.io/aws-ebs
  reclaimPolicy: Delete
  volumeBindingMode: Immediate
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""
</code></pre>
