Take the 2-minute tour ×

I am writing a program (I'll call it the "launcher") in C++ using MPI to "Spawn" a second executable (the "slave"). Depending on how many nodes a cluster has available for the launcher, it will launch slaves on each node and the slave will communicate back with the launcher also through MPI. When the slave is done with its math, it tells the launcher that the node is now available and the launcher Spawns another slave to the free node. The point is to run 1000 independent calculations, that depend on a second executable, on an heterogeneous group of machines.

This is working in my own computer, where I create a "fake" machinefile (or hostfile) giving two nodes to the program: localhost and localhost. The launcher Spawns two slaves and when one of them ends another slave is launched. This tells me that the Spawning process is working correctly.

When I move it to the cluster at my lab (whe use torque/maui to manage it), it also works if I ask for 1 (one) node. If I ask for more, I get a missing library error (libimf.so, to be precise. A library from the intel compilers). The lib is there and the node can see it, since the program runs if i ask for just one node.

My PBS that works looks like this:

#PBS -q small 
#PBS -l nodes=1:ppn=8:xeon
#PBS -l walltime=1:00:00
#PBS -N MyJob
#PBS -V 

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/mpich2.shared.exec/lib/:/opt/intel/composerxe-2011.3.174/compiler/lib/intel64/:/usr/local/boost/lib/



echo "Beginning PBS script." > $log_file

echo "Executing on hosts ($PBS_NODEFILE): " >> $log_file
cat $PBS_NODEFILE >> $log_file

echo "Running your stuff now!" >> $log_file

# mpiexec is needed in order to let "launcher" call MPI_Comm_spawn.
/usr/local/mpich2.shared.exec/bin/mpiexec -hostfile $PBS_NODEFILE -n 1 /home/user/launhcer --hostfile $PBS_NODEFILE -r 1 >> $log_file 2>&1

echo "Fim do pbs." >> $log_file

When I try two or more nodes, the launcher doesn't Spawn any executables. I get an output like this:

Beginning PBS script.
Executing on hosts (/var/spool/torque/aux//2742.cluster): 
Running your stuff now!

(Bla bla bla from launcher initialization)

/usr/local/mpich2.shared.exec/bin/hydra_pmi_proxy: error while loading shared libraries: libimf.so: cannot open shared object file: No such file or directory

I found one other person with a problem such as mine in a mailing list, but no solution. (http://lists.mcs.anl.gov/pipermail/mpich-discuss/2011-July/010442.html). The only answer suggested trying to find if the node can see the lib (if the directory where the lib was stored was mounted on the node), so I tried an

ssh node2 ls /opt/intel/composerxe-2011.3.174/compiler/lib/intel64/libimf.so >> $log_file

inside my PBS script and the lib exists in a folder that the node can see.

In my opinion, it seems that torque/maui is not exporting the environment variables to all nodes (even though I don't know why it wouldn't), so when I try to use MPI_Spawn to run another executable in another node, it can't find the lib. Does that make any sense? If so, could you suggest a solution? Can anyone offer any other ideas? Thanks in advance, Marcelo


Following the suggestion in one of the answers, I installed OpenMPI to test the option "-x VARNAME" with mpiexec. In the PBS script I changed the execution line to the following:

/usr/local/openmpi144/bin/mpiexec -x LD_LIBRARY_PATH -hostfile $PBS_NODEFILE -n 1 /var/dipro/melomcr/GSAFold_2/gsafold --hostfile $PBS_NODEFILE -r 1 >> $log_file 2>&1

but got the following error messages:

[node5:02982] [[3837,1],0] ORTE_ERROR_LOG: A message is attempting to be sent to a process whose contact information is unknown in file rml_oob_send.c at line 105
[node5:02982] [[3837,1],0] could not get route to [[INVALID],INVALID]
[node5:02982] [[3837,1],0] ORTE_ERROR_LOG: A message is attempting to be sent to a process whose contact information is unknown in file base/plm_base_proxy.c at line 86

From the internet I could gather that this error usually comes from executing mpiexec more than onece, like in /path/to/mpiexec mpiexec -n 2 my_program which is not my case.

I believe I should add that the spawned "slave" program communicates with the "launcher" program using a port. The launcher opens a port with MPI_Open_port and MPI_Comm_accept, then it waits for the slave program to connect when the slave runs MPI_Comm_connect.

Like I said above, all of this works (with MPICH2) when I ask for just one node. With OpenMPI I get the above error even when I ask for only one node.

share|improve this question
add comment

2 Answers

You're correct. The remoting calls far below the clustering software do not transfer environment variables.

share|improve this answer
Any ideas on how could I export the environment variables to the spawned processes? –  MeloMCR Sep 3 '12 at 23:29
spawn as: /bin/sh -c 'ENV=value programname' –  Joshua Sep 4 '12 at 16:02
I avoided doing this because I was looking for a portable solution. This software should be used in different environments. But thank you for the answer. –  MeloMCR Sep 5 '12 at 10:25
It's more portable than your bash script that launches it. –  Joshua Sep 5 '12 at 16:15
That is just the PBS script, it would change depending on the queuing system your cluster uses so it can't be portable, differently from the launcher program, that depends only on common libraries such as boost and MPI. –  MeloMCR Sep 5 '12 at 19:51
show 3 more comments

You could use the -x option to mpiexec to pass environment variables to other nodes.

share|improve this answer
Someone please check this. This has a significant chance of being right; however I'd be paranoid of it copying certain environment variables in mixed environments. –  Joshua Sep 6 '12 at 15:53
@Joshua This option is not available in MPICH2, which I was using in my cluster. I installed OpenMPI to test the solution but I got another error instead. I edited the question to add the information. –  MeloMCR Sep 6 '12 at 17:20
With mpich the option is -env instead of -x –  lrm29 Sep 6 '12 at 18:31
That one I tried but didn't solve the issue. Gives the same output. –  MeloMCR Sep 6 '12 at 19:01
add comment

Your Answer


