title: kubernetes volume mount timeout
body: <p>I am using PVC to attach the volume to one deployment for grafana, but it times out and container kept in creating stage.
  <strong>Storage class</strong></p>
  
  <pre><code>apiVersion: storage.k8s.io/v1
  kind: StorageClass
  metadata:
    name: grafana-storagetest
  provisioner: kubernetes.io/aws-ebs
  parameters:
    type: gp2
    iopsPerGB: "10"
  reclaimPolicy: Retain
  allowVolumeExpansion: true
  mountOptions:
    - debug
  volumeBindingMode: Immediate
  </code></pre>
  
  <p><strong>PVC</strong></p>
  
  <pre><code>apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    name: grafana-claimtest
  spec:
    accessModes:
      - ReadWriteOnce
    volumeMode: Block
    storageClassName: grafana-storagetest
    resources:
      requests:
        storage: 10G
  </code></pre>
  
  <p><strong>Service</strong></p>
  
  <pre><code>apiVersion: v1
  kind: Service
  metadata:
    annotations:
      getambassador.io/config: |
        ---
        apiVersion: ambassador/v1
        kind: Mapping
        name: grafana-proxytest_mapping
        prefix: /v1/anonymous/grafana-proxytest
        rewrite: /
        service: grafana-proxytest.grafana-proxytest:8080
        timeout_ms: 20000
        connect_timeout_ms: 20000
    labels:
      app: grafana-proxytest
    name: grafana-proxytest
    namespace: grafana-proxytest
  spec:
    ports:
    - name: http
      port: 8080
      protocol: TCP
      targetPort: 8080
    selector:
      app: grafana-proxytest
    type: ClusterIP
  status:
    loadBalancer: {}
  </code></pre>
  
  <p><strong>Deployment</strong></p>
  
  <pre><code>apiVersion: extensions/v1beta1
  kind: Deployment
  metadata:
    annotations:
      deployment.kubernetes.io/revision: "20"
    labels:
      version: v1
    name: grafana-proxytest-v1
    namespace: grafana-proxytest
  spec:
    replicas: 1
    selector:
      matchLabels:
        app: grafana-proxytest
        version: v1
    strategy:
      rollingUpdate:
        maxSurge: 1
        maxUnavailable: 1
      type: RollingUpdate
    template:
      metadata:
        creationTimestamp: null
        labels:
          app: grafana-proxytest
          version: v1
      spec:
        containers:
          image: &lt;aws_ecr&gt;
          imagePullPolicy: Always
          name: grafana-proxytest
          ports:
          - containerPort: 3000
            protocol: TCP
          resources:
            requests:
              cpu: 100m
              memory: 200Mi
          volumeMounts:
            - mountPath: /var/lib/grafana
              name: grafanapdtest
          image: grafana/grafana:latest
          imagePullPolicy: Always
          name: grafanatest
          ports:
          - containerPort: 3000
            protocol: TCP
          resources:
            requests:
              cpu: 100m
              memory: 200Mi
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
        volumes:
         - name: grafanapdtest
           persistentVolumeClaim: 
             claimName: grafana-claimtest
        restartPolicy: Always
        schedulerName: default-scheduler
        securityContext: {}
        terminationGracePeriodSeconds: 30
  </code></pre>
  
  <p><strong>Pod status</strong>
  <code>grafana-proxytest-v1-7cb5b6b6cf-z5zml   0/2     ContainerCreating   0          4m18s</code></p>
  
  <p><strong>Describe Pod</strong></p>
  
  <pre><code>Events:
    Type     Reason                  Age                    From                                                      Message
    ----     ------                  ----                   ----                                                      -------
    Warning  FailedScheduling        4m14s (x2 over 4m14s)  default-scheduler                                         pod has unbound immediate PersistentVolumeClaims (repeated 7 times)
    Normal   Scheduled               4m12s                  default-scheduler                                         Successfully assigned grafana-proxytest/grafana-proxytest-v1-7cb5b6b6cf-z5zml to ip-10-10-107-59.ap-southeast-1.compute.internal
    Normal   SuccessfulAttachVolume  4m10s                  attachdetach-controller                                   AttachVolume.Attach succeeded for volume "pvc-f8ad51be-74c5-11ea-8623-068411799338"
    Warning  FailedMount             2m9s                   kubelet, ip-10-10-107-59.ap-southeast-1.compute.internal  Unable to mount volumes for pod "grafana-proxytest-v1-7cb5b6b6cf-z5zml_grafana-proxytest(fcf38cab-74c5-11ea-8623-068411799338)": timeout expired waiting for volumes to attach or mount for pod "grafana-proxytest"/"grafana-proxytest-v1-7cb5b6b6cf-z5zml". list of unmounted volumes=[grafanapdtest]. list of unattached volumes=[grafanapdtest default-token-pzxdk]
  </code></pre>
  
  <p>Expected result - The volume should be attached properly and the pod should be in running status. </p>
  
  <p>I checked and the storage class, PVC, and PV are created.</p>
  
  <p><strong>Edit 1</strong>
  Added the namespace into PVC as below, but still it fails:</p>
  
  <pre><code>apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    name: grafana-claimtest
    namespace: grafana-proxytest
  spec:
    accessModes:
      - ReadWriteOnce
    volumeMode: Block
    storageClassName: grafana-storagetest
    resources:
      requests:
        storage: 10G
  </code></pre>
  
  <p>Still getting below error:</p>
  
  <pre><code>Events:
    Type     Reason                  Age    From                                                      Message
    ----     ------                  ----   ----                                                      -------
    Normal   Scheduled               2m18s  default-scheduler                                         Successfully assigned grafana-proxytest/grafana-proxytest-v1-7cb5b6b6cf-gpk4l to ip-10-10-107-59.ap-southeast-1.compute.internal
    Normal   SuccessfulAttachVolume  2m16s  attachdetach-controller                                   AttachVolume.Attach succeeded for volume "pvc-5bbc63a3-755f-11ea-920e-0a280935f44c"
    Warning  FailedMount             15s    kubelet, ip-10-10-107-59.ap-southeast-1.compute.internal  Unable to mount volumes for pod "grafana-proxytest-v1-7cb5b6b6cf-gpk4l_grafana-proxytest(62c50717-755f-11ea-8623-068411799338)": timeout expired waiting for volumes to attach or mount for pod "grafana-proxytest"/"grafana-proxytest-v1-7cb5b6b6cf-gpk4l". list of unmounted volumes=[grafanapdtest]. list of unattached volumes=[grafanapdtest default-token-pzxdk]
  </code></pre>
  
  <p>One thing as per the above logs i says that attach succeeded for volume, but then it fails for mounting grafanapdtest, which I am not able to understand.
  PVC details:</p>
  
  <pre><code>get pvc -n grafana-proxytest
  NAME                STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS          AGE
  grafana-claimtest   Bound    pvc-5bbc63a3-755f-11ea-920e-0a280935f44c   10Gi       RWO            grafana-storagetest   15m
  </code></pre>