<p>dear Kubernetes guru's!</p>

<p>I have spinned kube 1.4.1 cluster on manually created AWS hosts using 'contrib' Ansible playbook (<a href="https://github.com/kubernetes/contrib/tree/master/ansible" rel="nofollow">https://github.com/kubernetes/contrib/tree/master/ansible</a>). </p>

<p>My problem is that Kube doesn't attach EBS drives to minion hosts. If I define the pod as follows:</p>

<pre><code>apiVersion: extensions/v1beta1 
kind: Deployment 
metadata:
  name: kafka1
spec:
  replicas: 1
  template:
    spec:
      containers:
      - name: kafka1
        image: daniilyar/kafka
        ports:
        - containerPort: 9092 
          name: clientconnct
          protocol: TCP
        volumeMounts:
        - mountPath: /kafka
          name: storage
      volumes:
      - name: storage
        awsElasticBlockStore:
          volumeID: vol-56676d83
          fsType: ext4
</code></pre>

<p>I get the following error in kubelet.log:</p>

<pre><code>Mounting arguments: /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/vol-56676d83 /var/lib/kubelet/pods/db213783-9477-11e6-8aa9-12f3d1cdf81a/volumes/kubernetes.io~aws-ebs/storage  [bind]
Output: mount: special device /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/vol-56676d83 does not exist
</code></pre>

<p>EBS volume keeps being in 'Available' state during that, so I am sure that Kube <strong>doesn't attach</strong> volume to host at all and so, <strong>doesn't mount</strong> it.
I am 100% sure that this is a Kubernetes itself issue and not the permissioning issue because I can mount the same volume manually from within this minion to this minion just fine:</p>

<pre><code>$ aws ec2 --region us-east-1 attach-volume --volume-id vol-56676d83 --instance-id $(wget -q -O - http://instance-data/latest/meta-data/instance-id) --device /dev/sdc
{
    "AttachTime": "2016-10-18T15:02:41.672Z", 
    "InstanceId": "i-603cfb50", 
    "VolumeId": "vol-56676d83", 
    "State": "attaching", 
    "Device": "/dev/sdc"
}
</code></pre>

<p>Googling, hacking and trying older K8 versions didn't help me to solve this. 
Could anyone please point me on what else could I do to understand the problem so I can fix it? Any help is greatly appreciated.</p>

<p>Nobody helped me at K8 Slack channels, so after a day of pulling my hair off I found the solution by myself:</p>

<p>To get the K8 cluster installed by 'contrib' Ansible playbook (<a href="https://github.com/kubernetes/contrib/tree/master/ansible" rel="noreferrer">https://github.com/kubernetes/contrib/tree/master/ansible</a>) mounting EBS volumes properly, except for IAM roles setup, you need to add the <strong>--cloud-provider=aws</strong> flag to your existing cluster: all kubelets, the apiserver, and the controller manager.</p>

<p>Without <strong>--cloud-provider=aws</strong> flag Kubernetes will give you an unfriendly 'mount: special device xxx does not exist' error instead of real cause.</p>
