{"action":"created","issue":{"url":"https://api.github.com/repos/open-mpi/ompi/issues/5593","repository_url":"https://api.github.com/repos/open-mpi/ompi","labels_url":"https://api.github.com/repos/open-mpi/ompi/issues/5593/labels{/name}","comments_url":"https://api.github.com/repos/open-mpi/ompi/issues/5593/comments","events_url":"https://api.github.com/repos/open-mpi/ompi/issues/5593/events","html_url":"https://github.com/open-mpi/ompi/issues/5593","id":353569903,"node_id":"MDU6SXNzdWUzNTM1Njk5MDM=","number":5593,"title":"OPAL ERROR: Error in file %s/opal/mca/pmix/s1/pmix_s1.c at line 578","user":{"login":"morrisonlevi","id":253316,"node_id":"MDQ6VXNlcjI1MzMxNg==","avatar_url":"https://avatars2.githubusercontent.com/u/253316?v=4","gravatar_id":"","url":"https://api.github.com/users/morrisonlevi","html_url":"https://github.com/morrisonlevi","followers_url":"https://api.github.com/users/morrisonlevi/followers","following_url":"https://api.github.com/users/morrisonlevi/following{/other_user}","gists_url":"https://api.github.com/users/morrisonlevi/gists{/gist_id}","starred_url":"https://api.github.com/users/morrisonlevi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/morrisonlevi/subscriptions","organizations_url":"https://api.github.com/users/morrisonlevi/orgs","repos_url":"https://api.github.com/users/morrisonlevi/repos","events_url":"https://api.github.com/users/morrisonlevi/events{/privacy}","received_events_url":"https://api.github.com/users/morrisonlevi/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":14,"created_at":"2018-08-23T22:01:09Z","updated_at":"2018-08-28T17:07:48Z","closed_at":null,"author_association":"NONE","body":"## Background information\r\n\r\nI am trying to get OpenMPI and Slurm working working with PMIx, but have been hitting issues in various places.\r\n\r\n### What versions of software are you using?\r\n\r\n  - OpenMPI 3.1.2\r\n  - Slurm 17.11.9\r\n  - PMIx 2.1.3\r\n\r\nAll three of these were built from source.\r\n\r\nPMIx is installed to `/usr/local/pmix/2.1` and there is an `ld.so.conf.d` file that includes its libdir in the ld.so.cache.\r\n\r\nSlurm is installed to `/usr/include/slurm`, `/usr/lib64/slurm`, etc.\r\n\r\nOpenMPI is configured with:\r\n\r\n    '--prefix=/apps/openmpi/3.1.2/gcc-7.3.0_cuda-9.2.88_pmix-2.1.3'\r\n    '--enable-wrapper-rpath=no'\r\n    '--enable-wrapper-runpath=no' '--with-slurm'\r\n    '--with-pmix=/usr/local/pmix/2.1'\r\n    '--with-pmi=/usr/local/pmix/2.1'\r\n    '--with-pmi-libdir=/usr/local/pmix/2.1/lib'\r\n    '--with-cuda=/apps/cuda/9.2.88'\r\n    '--with-libevent=/usr'\r\n    'CPPFLAGS=-I/usr/local/pmix/2.1/include'\r\n    'LDFLAGS=-L/usr/local/pmix/2.1/lib'\r\n\r\nSide note: I was unsure what to do for `--with-pmi` -- I have tried various other settings and this has been the version that has worked the best so far.\r\n\r\nSide note 2: I got errors if I did not specific `-I` and `-L` in configure flags -- that shouldn't be necessary should it?\r\n\r\n### Please describe the system on which you are running\r\n\r\n* Operating system/version: RHEL 7\r\n* Computer hardware: Probably N/A\r\n* Network type: Probably N/A\r\n\r\n-----------------------------\r\n\r\n## Details of the problem\r\n\r\nIf I launch an MPI program with `srun --mpi=openmpi ...` I get this error:\r\n\r\n> OPAL ERROR: Error in file %s/opal/mca/pmix/s1/pmix_s1.c at line 578\r\n\r\nWhich can be seen [here](https://github.com/open-mpi/ompi/blob/0a9c358ef8adc7225bdbe97c5fa7741ef0e26297/opal/mca/pmix/s1/pmix_s1.c#L578). Then, instead of the program using *n* ranks there are *n* independent runs of the program. In this example there are 4 tasks all on one node; the issue manifests in other configurations too:\r\n\r\n```bash\r\n$ srun -n 4 --mpi=openmpi $prog\r\n[m7-1-2:08013] OPAL ERROR: Error in file /apps/src/openmpi/src/openmpi-3.1.2/opal/mca/pmix/s1/pmix_s1.c at line 578\r\n[m7-1-2:08014] OPAL ERROR: Error in file /apps/src/openmpi/src/openmpi-3.1.2/opal/mca/pmix/s1/pmix_s1.c at line 578\r\n[m7-1-2:08012] OPAL ERROR: Error in file /apps/src/openmpi/src/openmpi-3.1.2/opal/mca/pmix/s1/pmix_s1.c at line 578\r\n[m7-1-2:08015] OPAL ERROR: Error in file /apps/src/openmpi/src/openmpi-3.1.2/opal/mca/pmix/s1/pmix_s1.c at line 578\r\n6.52178\r\n8.49445\r\n8.49241\r\n8.49539\r\n```\r\n\r\nThe last four numbers are the outputs from each of the four runs.\r\n\r\nDoing `srun --mpi=pmix` does not exhibit this issue:\r\n\r\n```bash\r\n$ srun -n 4 --mpi=pmix $prog\r\n2.94764\r\n```\r\n\r\nI am unsure how to debug this; what further information should I provide?"},"comment":{"url":"https://api.github.com/repos/open-mpi/ompi/issues/comments/416665611","html_url":"https://github.com/open-mpi/ompi/issues/5593#issuecomment-416665611","issue_url":"https://api.github.com/repos/open-mpi/ompi/issues/5593","id":416665611,"node_id":"MDEyOklzc3VlQ29tbWVudDQxNjY2NTYxMQ==","user":{"login":"morrisonlevi","id":253316,"node_id":"MDQ6VXNlcjI1MzMxNg==","avatar_url":"https://avatars2.githubusercontent.com/u/253316?v=4","gravatar_id":"","url":"https://api.github.com/users/morrisonlevi","html_url":"https://github.com/morrisonlevi","followers_url":"https://api.github.com/users/morrisonlevi/followers","following_url":"https://api.github.com/users/morrisonlevi/following{/other_user}","gists_url":"https://api.github.com/users/morrisonlevi/gists{/gist_id}","starred_url":"https://api.github.com/users/morrisonlevi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/morrisonlevi/subscriptions","organizations_url":"https://api.github.com/users/morrisonlevi/orgs","repos_url":"https://api.github.com/users/morrisonlevi/repos","events_url":"https://api.github.com/users/morrisonlevi/events{/privacy}","received_events_url":"https://api.github.com/users/morrisonlevi/received_events","type":"User","site_admin":false},"created_at":"2018-08-28T17:07:48Z","updated_at":"2018-08-28T17:07:48Z","author_association":"NONE","body":"I have tried to thoroughly clean and then rebuild Slurm and OpenMPI. It seems that it was picking up artifacts from a previous build as suspected. Now both `srun --mpi=none ...` and `srun --mpi=pmi ...` with `OMPI_MCA_pmix_base_verbose=10` set show that `libpmi.so.0` could not be found:\r\n\r\n> [m7-1-2:07433] mca_base_component_repository_open: unable to open mca_pmix_s1: libpmi.so.0: cannot open shared object file: No such file or directory (ignored)\r\n\r\nSo I rebuilt with `LDFLAGS=\"-Wl,-rpath,/usr/local/lib` (which is where the Slurm's PMI libraries are at) and it all seems to be working.\r\n\r\nI guess my only remaining question is: should OpenMPI add the `--with-pmi-libdir` path to the RPATH/RUNPATH for the things that need PMI? My guess is that if I had to specify the libdir then I probably need the rpath set."}}
{"actor":{"display_login":"morrisonlevi"}}