<p>I'm attempting to follow the instructions at <a href="https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/registry" rel="nofollow">https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/registry</a> to add a private docker registry to Kubernetes, but the pod created by the rc isn't able to mount the persistent volume claim.</p>

<p>First I'm creating a volume on EBS like so:</p>

<pre><code>aws ec2 create-volume --region us-west-1 --availability-zone us-west-1a --size 32 --volume-type gp2
</code></pre>

<p>(<code>us-west-1a</code> is also the availability zone that all of my kube minions are running in.)</p>

<p>Then I create a persistent volume like so:</p>

<pre><code>kind: PersistentVolume
apiVersion: v1
metadata:
  name: kube-system-kube-registry-pv
  labels:
    kubernetes.io/cluster-service: "true"
spec:
  capacity:
    storage: 30Gi
  accessModes:
    - ReadWriteOnce
  awsElasticBlockStore:
    volumeID: vol-XXXXXXXX
    fsType: ext4
</code></pre>

<p>And a claim on the persistent volume like so:</p>

<pre><code>kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: kube-registry-pvc
  namespace: kube-system
  labels:
    kubernetes.io/cluster-service: "true"
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 30Gi
</code></pre>

<p>The replication controller is specified like so:</p>

<pre><code>apiVersion: v1
kind: ReplicationController
metadata:
  name: kube-registry-v0
  namespace: kube-system
  labels:
    k8s-app: kube-registry
    version: v0
    kubernetes.io/cluster-service: "true"
spec:
  replicas: 1
  selector:
    k8s-app: kube-registry
    version: v0
  template:
    metadata:
      labels:
        k8s-app: kube-registry
        version: v0
        kubernetes.io/cluster-service: "true"
    spec:
      containers:
      - name: registry
        image: registry:2
        resources:
          limits:
            cpu: 100m
            memory: 100Mi
        env:
        - name: REGISTRY_HTTP_ADDR
          value: :5000
        - name: REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY
          value: /var/lib/registry
        volumeMounts:
        - name: image-store
          mountPath: /var/lib/registry
        ports:
        - containerPort: 5000
          name: registry
          protocol: TCP
      volumes:
      - name: image-store
        persistentVolumeClaim:
          claimName: kube-registry-pvc
</code></pre>

<p>When I create the rc, It successfully starts a pod, but the pod is unable to mount the volume:</p>

<pre><code>$ kubectl describe po kube-registry --namespace=kube-system
...
Events:
  FirstSeen     LastSeen        Count   From                                                    SubobjectPath   Reason          Message
  ─────────     ────────        ─────   ────                                                    ─────────────   ──────          ───────
  1m            1m              1       {scheduler }                                                            Scheduled       Successfully assigned kube-registry-v0-3jobf to XXXXXXXXXXXXXXX.us-west-1.compute.internal
  22s           22s             1       {kubelet XXXXXXXXXXXXXXX.us-west-1.compute.internal}                    FailedMount     Unable to mount volumes for pod "kube-registry-v0-3jobf_kube-system": Timeout waiting for volume state
  22s           22s             1       {kubelet XXXXXXXXXXXXXXX.us-west-1.compute.internal}                    FailedSync      Error syncing pod, skipping: Timeout waiting for volume state
</code></pre>

<hr>

<p>I'm able to successfully mount EBS volumes if I don't use persistent volumes and persistent volume claims.  The following works without error, for example:</p>

<pre><code>apiVersion: v1
kind: Pod
metadata:
  name: test-ebs
spec:
  containers:
  - image: gcr.io/google_containers/test-webserver
    name: test-container
    volumeMounts:
    - mountPath: /test-ebs
      name: test-volume
  volumes:
  - name: test-volume
    awsElasticBlockStore:
      volumeID: vol-XXXXXXXX
      fsType: ext4
</code></pre>

<hr>

<p>My two questions are:</p>

<ul>
<li>Does anyone know what might be going wrong and how to fix it?</li>
<li>In general, where can I look for more details on errors like these?  I haven't been able to find more detailed log messages anywhere, and "Unable to mount volumes...Timeout waiting for volume state" isn't terribly helpful.</li>
</ul>

<p>I think I was likely running into <a href="https://github.com/kubernetes/kubernetes/issues/15073" rel="nofollow">https://github.com/kubernetes/kubernetes/issues/15073</a> .  (If I create a new EBS volume, I first get a different failure, and then after the pod has been killed if I try to re-create the rc I get the failure I mentioned in my question.)</p>

<p>Also, for anyone else wondering where to look for logs, <code>/var/log/syslog</code> and <code>/var/log/containers/XXX</code> on the kubelet was where I ended up having to look.</p>
