title: Why Does Podman Report "Not enough IDs available in namespace" with different UIDs?
body: <p>Facts:</p>
  
  <ol>
  <li>Rootless podman works perfectly for uid <code>1480</code></li>
  <li>Rootless podman fails for uid <code>2088</code></li>
  <li>CentOS 7</li>
  <li>Kernel <code>3.10.0-1062.1.2.el7.x86_64</code></li>
  <li>podman version <code>1.4.4</code></li>
  <li>Almost the entire environment has been removed between the two</li>
  <li>The filesystem for <code>/tmp</code> is <code>xfs</code></li>
  <li>The <code>capsh</code> output of the two users is identical but for uid / username</li>
  <li>Both UIDs have identical entries in <code>/etc/sub{u,g}id</code> files</li>
  <li>The <code>$HOME/.config/containers/storage.conf</code> is the default and is identical between the two with the exception of the uids. The <code>storage.conf</code> is below for reference.</li>
  </ol>
  
  <p>I wrote the following shell script to demonstrate just how similar an environment the two are operating in:</p>
  
  <pre class="lang-sh prettyprint-override"><code>#!/bin/sh
  for i in 1480 2088; do
    sudo chroot --userspec "$i":10 / env -i /bin/sh &lt;&lt;EOF
  echo -------------- $i ----------------
  /usr/sbin/capsh --print
  grep "$i" /etc/subuid /etc/subgid
  mkdir /tmp/"$i"
  HOME=/tmp/"$i"
  export HOME
  podman --root=/tmp/"$i" info &gt; /tmp/podman."$i"
  podman run --rm --root=/tmp/"$i" docker.io/library/busybox printf "\tCOMPLETE\n"
  echo -----------END $i END-------------
  EOF
    sudo rm -rf /tmp/"$i"
  done
  </code></pre>
  
  <p>Here's the output of the script:</p>
  
  <pre><code>$ sh /tmp/podman-fail.sh
  [sudo] password for functional:
  -------------- 1480 ----------------
  Current: =
  Bounding set =cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_module,cap_sys_rawio,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_resource,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap,cap_mac_override,cap_mac_admin,cap_syslog,35,36
  Securebits: 00/0x0/1'b0
   secure-noroot: no (unlocked)
   secure-no-suid-fixup: no (unlocked)
   secure-keep-caps: no (unlocked)
  uid=1480(functional)
  gid=10(wheel)
  groups=0(root)
  /etc/subuid:1480:100000:65536
  /etc/subgid:1480:100000:65536
  Trying to pull docker.io/library/busybox...Getting image source signatures
  Copying blob 7c9d20b9b6cd done
  Copying config 19485c79a9 done
  Writing manifest to image destination
  Storing signatures
  ERRO[0003] could not find slirp4netns, the network namespace won't be configured: exec: "slirp4netns": executable file not found in $PATH
          COMPLETE
  -----------END 1480 END-------------
  -------------- 2088 ----------------
  Current: =
  Bounding set =cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_module,cap_sys_rawio,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_resource,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap,cap_mac_override,cap_mac_admin,cap_syslog,35,36
  Securebits: 00/0x0/1'b0
   secure-noroot: no (unlocked)
   secure-no-suid-fixup: no (unlocked)
   secure-keep-caps: no (unlocked)
  uid=2088(broken)
  gid=10(wheel)
  groups=0(root)
  /etc/subuid:2088:100000:65536
  /etc/subgid:2088:100000:65536
  Trying to pull docker.io/library/busybox...Getting image source signatures
  Copying blob 7c9d20b9b6cd done
  Copying config 19485c79a9 done
  Writing manifest to image destination
  Storing signatures
  ERRO[0003] Error while applying layer: ApplyLayer exit status 1 stdout:  stderr: there might not be enough IDs available in the namespace (requested 65534:65534 for /home): lchown /home: invalid argument
  ERRO[0003] Error pulling image ref //busybox:latest: Error committing the finished image: error adding layer with blob "sha256:7c9d20b9b6cda1c58bc4f9d6c401386786f584437abbe87e58910f8a9a15386b": ApplyLayer exit status 1 stdout:  stderr: there might not be enough IDs available in the namespace (requested 65534:65534 for /home): lchown /home: invalid argument
  Failed
  Error: unable to pull docker.io/library/busybox: unable to pull image: Error committing the finished image: error adding layer with blob "sha256:7c9d20b9b6cda1c58bc4f9d6c401386786f584437abbe87e58910f8a9a15386b": ApplyLayer exit status 1 stdout:  stderr: there might not be enough IDs available in the namespace (requested 65534:65534 for /home): lchown /home: invalid argument
  </code></pre>
  
  <p>Here's the <code>storage.conf</code> for the <code>1480</code> uid. It's identical except <code>s/1480/2088/</code>:</p>
  
  <pre><code>[storage]
    driver = "vfs"
    runroot = "/run/user/1480"
    graphroot = "/tmp/1480/.local/share/containers/storage"
    [storage.options]
      size = ""
      remap-uids = ""
      remap-gids = ""
      remap-user = ""
      remap-group = ""
      ostree_repo = ""
      skip_mount_home = ""
      mount_program = ""
      mountopt = ""
      [storage.options.thinpool]
        autoextend_percent = ""
        autoextend_threshold = ""
        basesize = ""
        blocksize = ""
        directlvm_device = ""
        directlvm_device_force = ""
        fs = ""
        log_level = ""
        min_free_space = ""
        mkfsarg = ""
        mountopt = ""
        use_deferred_deletion = ""
        use_deferred_removal = ""
        xfs_nospace_max_retries = ""
  </code></pre>
  
  <p>You can see there's basically no difference between the two <code>podman info</code> outputs for the users:</p>
  
  <pre><code>$ diff -u /tmp/podman.1480 /tmp/podman.2088
  --- /tmp/podman.1480    2019-10-17 22:41:21.991573733 -0400
  +++ /tmp/podman.2088    2019-10-17 22:41:26.182584536 -0400
  @@ -7,7 +7,7 @@
     Distribution:
       distribution: '"centos"'
       version: "7"
  -  MemFree: 45654056960
  +  MemFree: 45652697088
     MemTotal: 67306323968
     OCIRuntime:
       package: containerd.io-1.2.6-3.3.el7.x86_64
  @@ -24,7 +24,7 @@
     kernel: 3.10.0-1062.1.2.el7.x86_64
     os: linux
     rootless: true
  -  uptime: 30h 17m 50.23s (Approximately 1.25 days)
  +  uptime: 30h 17m 54.42s (Approximately 1.25 days)
   registries:
     blocked: null
     insecure: null
  @@ -35,14 +35,14 @@
     - quay.io
     - registry.centos.org
   store:
  -  ConfigFile: /tmp/1480/.config/containers/storage.conf
  +  ConfigFile: /tmp/2088/.config/containers/storage.conf
     ContainerStore:
       number: 0
     GraphDriverName: vfs
     GraphOptions: null
  -  GraphRoot: /tmp/1480
  +  GraphRoot: /tmp/2088
     GraphStatus: {}
     ImageStore:
       number: 0
  -  RunRoot: /run/user/1480
  -  VolumePath: /tmp/1480/volumes
  +  RunRoot: /run/user/2088
  +  VolumePath: /tmp/2088/volumes
  </code></pre>
  
  <p>I refuse to believe there's an <code>if (2088 == uid) { abort(); }</code> or similar nonsense somewhere in podman's source code. What am I missing?</p>