<p>I'm running a kubernetes 1.6.2 cluster across three nodes in different zones in GKE and I'm trying to deploy my statefulset where each pod in the statefulset gets a PV attached to it. The problem is that kubernetes is creating the PVs in the one zone where I don't have a node!</p>

<pre><code>$ kubectl describe node gke-multi-consul-default-pool-747c9378-zls3|grep 'zone=us-central1'
            failure-domain.beta.kubernetes.io/zone=us-central1-a
$ kubectl describe node gke-multi-consul-default-pool-7e987593-qjtt|grep 'zone=us-central1'
            failure-domain.beta.kubernetes.io/zone=us-central1-f
$ kubectl describe node gke-multi-consul-default-pool-8e9199ea-91pj|grep 'zone=us-central1'
            failure-domain.beta.kubernetes.io/zone=us-central1-c

$ kubectl describe pv pvc-3f668058-2c2a-11e7-a7cd-42010a8001e2|grep 'zone=us-central1'
        failure-domain.beta.kubernetes.io/zone=us-central1-b
</code></pre>

<p>I'm using the standard storageclass which has no default zone set:</p>

<pre><code>$ kubectl describe storageclass standard
Name:       standard
IsDefaultClass: Yes
Annotations:    storageclass.beta.kubernetes.io/is-default-class=true
Provisioner:    kubernetes.io/gce-pd
Parameters: type=pd-standard
Events:     &lt;none&gt;
</code></pre>

<p>So I thought that the scheduler would automatically provision the volumes in a zone where a cluster node existed, but it doesn't seem to be doing that.</p>

<p>For reference, here is the yaml for my statefulset:</p>

<pre><code>apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: "{{ template "fullname" . }}"
  labels:
    heritage: {{.Release.Service | quote }}
    release: {{.Release.Name | quote }}
    chart: "{{.Chart.Name}}-{{.Chart.Version}}"
    component: "{{.Release.Name}}-{{.Values.Component}}"
spec:
  serviceName: "{{ template "fullname" . }}"
  replicas: {{default 3 .Values.Replicas}}
  template:
    metadata:
      name: "{{ template "fullname" . }}"
      labels:
        heritage: {{.Release.Service | quote }}
        release: {{.Release.Name | quote }}
        chart: "{{.Chart.Name}}-{{.Chart.Version}}"
        component: "{{.Release.Name}}-{{.Values.Component}}"
        app: "consul"
      annotations:
        pod.alpha.kubernetes.io/initialized: "true"
    spec:
      securityContext:
        fsGroup: 1000
      containers:
      - name: "{{ template "fullname" . }}"
        image: "{{.Values.Image}}:{{.Values.ImageTag}}"
        imagePullPolicy: "{{.Values.ImagePullPolicy}}"
        ports:
        - name: http
          containerPort: {{.Values.HttpPort}}
        - name: rpc
          containerPort: {{.Values.RpcPort}}
        - name: serflan-tcp
          protocol: "TCP"
          containerPort: {{.Values.SerflanPort}}
        - name: serflan-udp
          protocol: "UDP"
          containerPort: {{.Values.SerflanUdpPort}}
        - name: serfwan-tcp
          protocol: "TCP"
          containerPort: {{.Values.SerfwanPort}}
        - name: serfwan-udp
          protocol: "UDP"
          containerPort: {{.Values.SerfwanUdpPort}}
        - name: server
          containerPort: {{.Values.ServerPort}}
        - name: consuldns
          containerPort: {{.Values.ConsulDnsPort}}
        resources:
          requests:
            cpu: "{{.Values.Cpu}}"
            memory: "{{.Values.Memory}}"
        env:
        - name: INITIAL_CLUSTER_SIZE
          value: {{ default 3 .Values.Replicas | quote }}
        - name: STATEFULSET_NAME
          value: "{{ template "fullname" . }}"
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        - name: STATEFULSET_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        volumeMounts:
        - name: datadir
          mountPath: /var/lib/consul
        - name: gossip-key
          mountPath: /etc/secrets
          readOnly: true
        - name: config
          mountPath: /etc/consul
        - name: tls
          mountPath: /etc/tls
        lifecycle:
          preStop:
            exec:
              command:
                - /bin/sh
                - -c
                - consul leave
        livenessProbe:
          exec:
            command:
            - consul
            - members
          initialDelaySeconds: 300
          timeoutSeconds: 5
        command:
          - "/bin/sh"
          - "-ec"
          - "/tmp/consul-start.sh"
      volumes:
      - name: config
        configMap:
          name: consul
      - name: gossip-key
        secret:
          secretName: {{ template "fullname" . }}-gossip-key
      - name: tls
        secret:
          secretName: consul
  volumeClaimTemplates:
  - metadata:
      name: datadir
      annotations:
      {{- if .Values.StorageClass }}
        volume.beta.kubernetes.io/storage-class: {{.Values.StorageClass | quote}}
      {{- else }}
        volume.alpha.kubernetes.io/storage-class: default
      {{- end }}
    spec:
      accessModes:
        - "ReadWriteOnce"
      resources:
        requests:
          # upstream recommended max is 700M
          storage: "{{.Values.Storage}}"
</code></pre>

<p>There is a bug open for this issue <a href="https://github.com/kubernetes/kubernetes/issues/50115" rel="nofollow noreferrer">here</a>.</p>

<p>The workaround in the meantime is to set the <code>zones</code> parameter in your <code>StorageClass</code> to specify the exact zones where your Kubernetes cluster has nodes.  <a href="https://kubernetes.io/docs/concepts/storage/persistent-volumes/#gce" rel="nofollow noreferrer">Here</a> is an example.</p>
