<p>I would like to add an iSCSI volume to a pod as in this <a href="https://github.com/kubernetes/examples/tree/master/staging/volumes/iscsi" rel="nofollow noreferrer">this example</a>. I have already prepared an iSCSI target on a Debian server and installed <code>open-iscsi</code> on all my worker nodes. I have also confirmed that I can mount the iSCSI target on a worker node with command line tools (i.e. still outside Kubernetes). This works fine. For simplicity, there is no authentication (CHAP) in play yet, and there is already a <code>ext4</code> file system present on the target.</p>

<p>I would now like for Kubernetes 1.14 to mount the same iSCSI target into a pod with the following manifest:</p>

<pre><code>---
apiVersion: v1
kind: Pod
metadata:
  name: iscsipd
spec:
  containers:
  - name: iscsipd-ro
    image: kubernetes/pause
    volumeMounts:
    - mountPath: "/mnt/iscsipd"
      name: iscsivol
  volumes:
  - name: iscsivol
    iscsi:
      targetPortal: 1.2.3.4 # my target
      iqn: iqn.2019-04.my-domain.com:lun1
      lun: 0
      fsType: ext4
      readOnly: true
</code></pre>

<p>According to <code>kubectl describe pod</code> this works in the initial phase (<code>SuccessfulAttachVolume</code>), but then fails (<code>FailedMount</code>). The exact error message reads:</p>

<pre><code>Warning  FailedMount ... Unable to mount volumes for pod "iscsipd_default(...)": timeout expired waiting for volumes to attach or mount for pod "default"/"iscsipd". list of unmounted volumes=[iscsivol]. list of unattached volumes=[iscsivol default-token-7bxnn]
Warning  FailedMount ... MountVolume.WaitForAttach failed for volume "iscsivol" : failed to get any path for iscsi disk, last err seen:
Could not attach disk: Timeout after 10s
</code></pre>

<p>How can I further diagnose and overcome this problem?</p>

<p><strong>UPDATE</strong> In <a href="https://github.com/rancher/rancher/issues/12433" rel="nofollow noreferrer">this</a> related issue the solution consisted of using a numeric IP address for the target. However, this does not help in my case, since I am already using a <code>targetPortal</code> of the form <code>1.2.3.4</code> (have
also tried both with and without port number 3260).</p>

<p><strong>UPDATE</strong> Stopping <code>scsid.service</code> and/or <code>open-iscsi.service</code> (as suggested <a href="https://monach.us/operations/rke-and-iscsi-race-condition/" rel="nofollow noreferrer">here</a>) did not make a difference either.</p>

<p><strong>UPDATE</strong> The error apparently gets triggered in <a href="https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/iscsi/iscsi_util.go" rel="nofollow noreferrer"><code>pkg/volume/iscsi/iscsi_util.go</code></a> if <code>waitForPathToExist(&amp;devicePath, multipathDeviceTimeout, iscsiTransport)</code> fails. However, what is strange is that when it is triggered the file at <code>devicePath</code> (<code>/dev/disk/by-path/ip-...-iscsi-...-lun-...</code>) does actually exist on the node.</p>

<p><strong>UPDATE</strong> I have used this procedure for defining an simple iSCSI target for these test purposes:</p>

<pre><code>pvcreate /dev/sdb
vgcreate iscsi /dev/sdb
lvcreate -L 10G -n iscsi_1 iscsi
apt-get install tgt
cat &gt;/etc/tgt/conf.d/iscsi_1.conf &lt;&lt;EOL
&lt;target iqn.2019-04.my-domain.com:lun1&gt;
  backing-store /dev/mapper/iscsi-iscsi_1
  initiator-address 5.6.7.8 # my cluster node #1
  ... # my cluster node #2, etc.
&lt;/target&gt;
EOL
systemctl restart tgt
tgtadm --mode target --op show
</code></pre>

<p>This is probably because of authentication issue to your iscsi target. </p>

<p>If you don't use CHAP authentication yet, you still have to disable authentication.
For example, if you use <code>targetcli</code>, you can run below commands to disable it.</p>

<pre><code>$ sudo targetcli
/&gt; /iscsi/iqn.2003-01.org.xxxx/tpg1 set attribute authentication=0 # will disable auth
/&gt; /iscsi/iqn.2003-01.org.xxxx/tpg1 set attribute generate_node_acls=1 # will force to use tpg1 auth mode by default
</code></pre>

<p>If this doesn't help you, please share your iscsi target configuration, or guide that you followed.</p>
