title: Problems reading slurm configuration file with Singularity
body: <p>I'm trying to run an application in Singularity across nodes (864 MPI tasks) on an HPC system, namely the S4 machine at the University of Wisconsin's Space Science and Engineering Center (SSEC).</p>
  
  <p>I'm using what Singularity describes as the hybrid model <a href="https://sylabs.io/guides/3.5/user-guide/mpi.html" rel="nofollow noreferrer">1</a>, meaning that I'm using the native (system) MPI but I also have MPI installed in the container.  The mpi versions are compatible - I'm using Intel MPI version 17.0.6 outside the container and Intel MPI version 17.0.1 inside the container.  The code in the container is compiled with Intel 17.0.1 compilers (C++, C, and Fortran).</p>
  
  <p>So here's the problem.  When I first ran the code, it complained about not finding the slurm configuration file:</p>
  
  <pre><code>fv3jedi_var.x: error: s_p_parse_file: unable to status file /etc/slurm-llnl/slurm.conf: No such file or directory, retrying in 1sec up to 60sec
  </code></pre>
  
  <p>So I found the system slurm.conf file in /etc/slurm and mounted this directory in the container as /etc/slurm-llnl.  It now finds the configuration file but it does not understand the site-specific configuration:</p>
  
  <pre><code>fv3jedi_var.x: error: "ALL" is not a valid option for "EnforcePartLimits"
  fv3jedi_var.x: error: Parsing error at unrecognized key: Features
  fv3jedi_var.x: error: Parse error in file /etc/slurm-llnl/slurm.conf line 225: " Features=ivy"
  fv3jedi_var.x: error: Parsing error at unrecognized key: Features
  fv3jedi_var.x: error: Parse error in file /etc/slurm-llnl/slurm.conf line 226: " Features=ivy"
  fv3jedi_var.x: error: Parsing error at unrecognized key: Features
  [...]
  </code></pre>
  
  <p>So, I'm stuck.  I'm guessing that this might be a PMI issue?  I currently have slurm libpmi.so installed in the container and that's what I'm specifying with the I_MPI_PMI_LIBRARY variable.  But I wonder if the native (system) PMI (I know it is PMI as opposed to PMI2 or PMIx) is somehow configured to properly process the system slurm.conf file?  I have tried to use the native PMI library by mounting (binding) the appropriate directory in the container and changing my I_MPI_PMI_LIBRARY variable.  But, the native PMI library is in the same directory as the glibc library and when I mount that there is a conflict between the glibc libraries inside and outside the container:</p>
  
  <pre><code>/bin/sh: relocation error: /usr/lib64/libc.so.6: symbol _dl_starting_up, version GLIBC_PRIVATE not defined in file ld-linux-x86-64.so.2 with link time reference
  </code></pre>
  
  <p>Any ideas on how to proceed?  My slurm batch script is below.  Thanks!</p>
  
  <pre><code>#!/usr/bin/bash
  # --mem-per-cpu=8192M
  #SBATCH --job-name=bm_con14
  #SBATCH --partition=ivy
  #SBATCH --ntasks=864
  #SBATCH --cpus-per-task=1
  #SBATCH --time=2:00:00
  #SBATCH --mail-user=miesch@ucar.edu
  
  source /etc/bashrc
  module purge
  module load license_intel
  module load intel/17.0.6
  ulimit -s unlimited
  
  cd /data/users/mmiesch/runs/con-benchmark/con
  
  JEDICON=/data/users/mmiesch
  JEDIBUILD=/data/users/mmiesch/jedi/fv3-bundle/build-con
  JEDIBIN=/data/users/mmiesch/jedi/fv3-bundle/build-con/bin
  
  export SINGULARITY_BINDPATH="$JEDIBUILD,/etc/slurm:/etc/slurm-llnl"
  
  srun --ntasks=864 --cpu_bind=cores --distribution=block:block --verbose singularity exec --home=$PWD $JEDICON/jedi-intel17-impi-hpc-dev.sif ${JEDIBIN
  }/fv3jedi_var.x Config/3dvar_bump.yaml
  
  exit 0
  </code></pre>