{"action":"created","issue":{"url":"https://api.github.com/repos/flux-framework/flux-core/issues/864","repository_url":"https://api.github.com/repos/flux-framework/flux-core","labels_url":"https://api.github.com/repos/flux-framework/flux-core/issues/864/labels{/name}","comments_url":"https://api.github.com/repos/flux-framework/flux-core/issues/864/comments","events_url":"https://api.github.com/repos/flux-framework/flux-core/issues/864/events","html_url":"https://github.com/flux-framework/flux-core/issues/864","id":184512371,"number":864,"title":"flux exec vs wreckrun behavior forwarding environment variables","user":{"login":"chu11","id":274859,"avatar_url":"https://avatars.githubusercontent.com/u/274859?v=3","gravatar_id":"","url":"https://api.github.com/users/chu11","html_url":"https://github.com/chu11","followers_url":"https://api.github.com/users/chu11/followers","following_url":"https://api.github.com/users/chu11/following{/other_user}","gists_url":"https://api.github.com/users/chu11/gists{/gist_id}","starred_url":"https://api.github.com/users/chu11/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chu11/subscriptions","organizations_url":"https://api.github.com/users/chu11/orgs","repos_url":"https://api.github.com/users/chu11/repos","events_url":"https://api.github.com/users/chu11/events{/privacy}","received_events_url":"https://api.github.com/users/chu11/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2016-10-21T15:20:46Z","updated_at":"2016-10-21T15:31:00Z","closed_at":null,"body":"Was messing around trying to get Magpie to work under Slurm & flux and got a very trivial case to work.\r\n\r\nI noticed that the initial environment variables created by Slurm ```srun``` don't get \"forwarded\" to tasks executed by ```wreckrun``` but they do for ```exec```.\r\n\r\nHere's what I mean.\r\n\r\n```\r\n$ cat do_some_stuff.sh\r\n#!/bin/sh\r\nhostname\r\nenv | grep SLURM_JOB_ID\r\nenv | grep FOO\r\nexit 0\r\n\r\n$ cat fluxexec.sh\r\n#!/bin/sh\r\nflux exec do_some_stuff.sh\r\nexit 0\r\n\r\n$ srun -N4 -n4 src/cmd/flux start fluxexec.sh\r\nsrun: Job is in held state, pending scheduler release\r\nsrun: job 1159741 queued and waiting for resources\r\nsrun: job 1159741 has been allocated resources\r\nhype254\r\nSLURM_JOB_ID=1159741\r\nFOO=bar\r\nhype256\r\nhype255\r\nSLURM_JOB_ID=1159741\r\nSLURM_JOB_ID=1159741\r\nFOO=bar\r\nhype257\r\nFOO=bar\r\nSLURM_JOB_ID=1159741\r\nFOO=bar\r\n```\r\n\r\nAs expected, Slurm starts up flux, flux runs ```fluxexec.sh```, which runs ```do_some_stuff.sh``` on all nodes via ```flux exec```.  ```do_some_stuff.sh``` outputs the hostname then looks to see if environment variables ```SLURM_JOB_ID``` and ```FOO``` are there.  ```FOO``` is something I created locally before running.  As you can see in the output, ```SLURM_JOB_ID``` and ```FOO``` are forwarded everywhere and available on each node.\r\n\r\nHere's the case w/ ```wreckrun```\r\n\r\n```\r\n$ cat fluxwreckrun.sh\r\n#!/bin/sh\r\nflux wreckrun -N4 do_some_stuff.sh\r\nexit 0\r\n\r\n$ srun -N4 -n4 src/cmd/flux start fluxwreckrun.sh\r\nsrun: Job is in held state, pending scheduler release\r\nsrun: job 1159742 queued and waiting for resources\r\nsrun: job 1159742 has been allocated resources\r\nhype254\r\nhype255\r\nhype256\r\nhype257\r\nFOO=bar\r\nFOO=bar\r\nFOO=bar\r\nFOO=bar\r\n```\r\n\r\nThe SLURM environment variable ```SLURM_JOB_ID``` doesn't get \"forwarded\" but ```FOO``` does.\r\n\r\nIs this intended in some way?  Or am I not understanding something?  The SLURM environment variables were missing in the ```kvs``` too, but ```FOO``` was there.\r\n\r\nI use the environment variables b/c I use to the jobname and jobid as unique identifiers for creation of various info and files in Magpie.  Obviously in the future with a bootstrapped flux these would perhaps be available in other ways, but I believe it's what I have to use at the moment.  Here's the section of a Magpie script I updated for flux, with some \"HARDCODED\".\r\n\r\n```bash\r\nexport MAGPIE_CLUSTER_NODERANK=\"${FLUX_NODE_ID}\"\r\nexport MAGPIE_NODE_COUNT=\"${FLUX_JOB_NNODES}\"\r\nMAGPIE_NODELIST=\"\"\r\nfor i in `seq 0 $((${FLUX_JOB_NNODES} - 1))`\r\ndo\r\n  fluxhostname=`flux kvs get resource.hwloc.xml.${i} | grep \"HostName\" | sed -e 's/<info name=\"HostName\" value=\"\\(.*\\)\"\\/>/\\1/' | tr -d '[[:space:]]'`\r\n  MAGPIE_NODELIST=${MAGPIE_NODELIST}${MAGPIE_NODELIST:+\",\"}${fluxhostname}\r\ndone\r\nexport MAGPIE_NODELIST\r\n# HARDCODED\r\nexport MAGPIE_JOB_NAME=\"testingflux\"\r\n# HARDCODED\r\nexport MAGPIE_JOB_ID=\"1234\"\r\n```\r\n\r\nAs an aside, that's some hackery I have to do in the middle there to get the nodes I'm running on.  I can add it as a todo to create a FLUX_NODELIST environment variable or some equivalent.\r\n"},"comment":{"url":"https://api.github.com/repos/flux-framework/flux-core/issues/comments/255408942","html_url":"https://github.com/flux-framework/flux-core/issues/864#issuecomment-255408942","issue_url":"https://api.github.com/repos/flux-framework/flux-core/issues/864","id":255408942,"user":{"login":"SteVwonder","id":874707,"avatar_url":"https://avatars.githubusercontent.com/u/874707?v=3","gravatar_id":"","url":"https://api.github.com/users/SteVwonder","html_url":"https://github.com/SteVwonder","followers_url":"https://api.github.com/users/SteVwonder/followers","following_url":"https://api.github.com/users/SteVwonder/following{/other_user}","gists_url":"https://api.github.com/users/SteVwonder/gists{/gist_id}","starred_url":"https://api.github.com/users/SteVwonder/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SteVwonder/subscriptions","organizations_url":"https://api.github.com/users/SteVwonder/orgs","repos_url":"https://api.github.com/users/SteVwonder/repos","events_url":"https://api.github.com/users/SteVwonder/events{/privacy}","received_events_url":"https://api.github.com/users/SteVwonder/received_events","type":"User","site_admin":false},"created_at":"2016-10-21T15:30:59Z","updated_at":"2016-10-21T15:30:59Z","body":"I believe this was intentional due to some issues with MPI.  IIRC, MPI applications would wire up in the SLURM allocation instead of the Flux allocation upon detection of certain SLURM environment variables (not 100% sure though).  The wreck code that is sanitizing the environment is https://github.com/flux-framework/flux-core/blob/18ae69704b5b2e2ee7f221e336bf6f253fc07607/src/bindings/lua/wreck.lua#L133"}}
{"actor":{"display_login":"SteVwonder"}}