{"action":"created","issue":{"url":"https://api.github.com/repos/open-mpi/ompi/issues/6068","repository_url":"https://api.github.com/repos/open-mpi/ompi","labels_url":"https://api.github.com/repos/open-mpi/ompi/issues/6068/labels{/name}","comments_url":"https://api.github.com/repos/open-mpi/ompi/issues/6068/comments","events_url":"https://api.github.com/repos/open-mpi/ompi/issues/6068/events","html_url":"https://github.com/open-mpi/ompi/issues/6068","id":378999764,"node_id":"MDU6SXNzdWUzNzg5OTk3NjQ=","number":6068,"title":"3.1.3: vsetenv PMIX_SERVER_TMPDIR failed","user":{"login":"benmenadue","id":18689148,"node_id":"MDQ6VXNlcjE4Njg5MTQ4","avatar_url":"https://avatars1.githubusercontent.com/u/18689148?v=4","gravatar_id":"","url":"https://api.github.com/users/benmenadue","html_url":"https://github.com/benmenadue","followers_url":"https://api.github.com/users/benmenadue/followers","following_url":"https://api.github.com/users/benmenadue/following{/other_user}","gists_url":"https://api.github.com/users/benmenadue/gists{/gist_id}","starred_url":"https://api.github.com/users/benmenadue/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/benmenadue/subscriptions","organizations_url":"https://api.github.com/users/benmenadue/orgs","repos_url":"https://api.github.com/users/benmenadue/repos","events_url":"https://api.github.com/users/benmenadue/events{/privacy}","received_events_url":"https://api.github.com/users/benmenadue/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2018-11-09T02:39:02Z","updated_at":"2018-11-09T04:52:55Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"## Background information\r\n\r\n### What version of Open MPI are you using? (e.g., v1.10.3, v2.1.0, git branch name and hash, etc.)\r\n\r\nOpenMPI 3.1.3 + external PMIx 3.0.2\r\n\r\n### Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)\r\n\r\nrelease tar-balls\r\n\r\n### Please describe the system on which you are running\r\n\r\n* Operating system/version: CentOS 6.10\r\n* Computer hardware: various\r\n* Network type: IB\r\n\r\n-----------------------------\r\n\r\n## Details of the problem\r\n\r\nOpenMPI library is configured against an external PMIx build. Launching an MPI program produces the message\r\n```\r\nvsetenv PMIX_SERVER_TMPDIR failed\r\n```\r\nseveral times from each MPI process on calling `MPI_Init`.\r\n\r\nA bit of `strace`-ing suggests the message is being generated by ibverbs -- all of the messages are generated follow this trace (or very similar, e.g. an extra `brk` call or three along the way):\r\n```\r\n[pid 20935] open(\"/dev/infiniband/uverbs0\", O_RDWR) = 15\r\n[pid 20935] write(15, \"\\0\\0\\0\\0\\4\\0\\6\\0\\200\\3452W\\377\\177\\0\\0\", 16) = 16\r\n[pid 20935] mmap(NULL, 4096, PROT_WRITE, MAP_SHARED, 15, 0) = 0x2aebf1cea000\r\n[pid 20935] mmap(NULL, 4096, PROT_WRITE, MAP_SHARED, 15, 0x1000) = 0x2aebf1ceb000\r\n[pid 20935] write(15, \"D\\0\\0\\0\\1\\0=\\0\\220\\3372W\\377\\177\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\16\\20\\0\\0\\0\\0\\0\\0\", 32) = 32\r\n[pid 20935] ioctl(15, MGSL_IOCGPARAMS or MMTIMER_GETRES or MTIOCTOP or SNDCTL_MIDI_MPUMODE, 0x7fff5732e5dc) = 0\r\n[pid 20935] mmap(NULL, 3928, PROT_READ, MAP_SHARED, 15, 0x3000) = 0x2aebf1cec000\r\n[pid 20935] write(2, \"vsetenv PMIX_SERVER_TMPDIR failed\\n\", 34vsetenv PMIX_SERVER_TMPDIR failed\r\n) = 34\r\n```\r\nI suspect it's due to there being 2 copies of that environment variable (!) in the environment somehow:\r\n```\r\n> mpirun -np 1 strings /proc/self/environ | grep PMIX_SERVER_TMPDIR\r\nPMIX_SERVER_TMPDIR=/jobfs/local/2185069.r-man2/ompi.r41.5800/pid.17479/0/0\r\nPMIX_SERVER_TMPDIR=/jobfs/local/2185069.r-man2/ompi.r41.5800/pid.17479\r\n```"},"comment":{"url":"https://api.github.com/repos/open-mpi/ompi/issues/comments/437249849","html_url":"https://github.com/open-mpi/ompi/issues/6068#issuecomment-437249849","issue_url":"https://api.github.com/repos/open-mpi/ompi/issues/6068","id":437249849,"node_id":"MDEyOklzc3VlQ29tbWVudDQzNzI0OTg0OQ==","user":{"login":"benmenadue","id":18689148,"node_id":"MDQ6VXNlcjE4Njg5MTQ4","avatar_url":"https://avatars1.githubusercontent.com/u/18689148?v=4","gravatar_id":"","url":"https://api.github.com/users/benmenadue","html_url":"https://github.com/benmenadue","followers_url":"https://api.github.com/users/benmenadue/followers","following_url":"https://api.github.com/users/benmenadue/following{/other_user}","gists_url":"https://api.github.com/users/benmenadue/gists{/gist_id}","starred_url":"https://api.github.com/users/benmenadue/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/benmenadue/subscriptions","organizations_url":"https://api.github.com/users/benmenadue/orgs","repos_url":"https://api.github.com/users/benmenadue/repos","events_url":"https://api.github.com/users/benmenadue/events{/privacy}","received_events_url":"https://api.github.com/users/benmenadue/received_events","type":"User","site_admin":false},"created_at":"2018-11-09T04:52:55Z","updated_at":"2018-11-09T04:52:55Z","author_association":"CONTRIBUTOR","body":"Found it, I think. There's this inside `orted` in the v3.1 branch: https://github.com/open-mpi/ompi/blob/17040631627b760efe3da95b4fc75eaf1d1aca4c/orte/orted/pmix/pmix_server.c#L251-L252\r\n\r\nIt was taken out of master by fa18ba395d72b7d584a65e69788c0af9d850cf4c."}}
{"actor":{"display_login":"benmenadue"}}