title: MPI not using all CPUs allocated
body: <p>I am trying to run some code across multiple CPUs using MPI.</p>
  
  <p>I run using:</p>
  
  <pre><code>$ mpirun -np 24 python mycode.py
  </code></pre>
  
  <p>I'm running on a cluster with 8 nodes, each with 12 CPUs.  My 24 processes get scattered across all nodes.</p>
  
  <p>Let's call the nodes node1, node2, ..., node8 and assume that the master process is on node1 and my job is the only one running.  So node1 has the master process and a few slave processes, the rest of the nodes have only slave processes.</p>
  
  <p>Only the node with the master process (ie node1) is being used.  I can tell because nodes2-8 have load ~0 and node1 has load ~24 (whereas I would expect the load on each node to be approximately equal to the number of CPUs allocated to my job from that node).  Also, each time a function is evaluated, I get it to print out the name of the host on which its running, and it prints out "node1" every time.  I don't know whether the master process is the only one doing anything or if the slave processes on the same node as the master are also being used.</p>
  
  <p>The cluster I'm running on was recently upgraded.  Before the upgrade, I was using the same code and it behaved entirely as expected (i.e. when I asked for 24 CPUs, it gave me 24 CPUs and then used all 24 CPUs).  This problem has only arisen since the upgrade, so I assume a setting somewhere got changed or reset.  Has anyone seen this problem before and know how I might fix it?</p>
  
  <p>Edit: This is submitted as a job to a scheduler using:</p>
  
  <pre><code>#!/bin/bash
  #
  #$ -cwd
  #$ -pe * 24
  #$ -o $JOB_ID.out
  #$ -e $JOB_ID.err
  #$ -r no
  #$ -m n
  #$ -l h_rt=24:00:00
  
  echo job_id $JOB_ID
  echo hostname $HOSTNAME
  
  mpirun -np $NSLOTS python mycode.py
  </code></pre>
  
  <p>The cluster is running SGE and I submit this job using:</p>
  
  <pre><code>qsub myjob
  </code></pre>