title: Memory-intense jobs scaling poorly on multi-core cloud instances (ec2, gce, rackspace)?
body: <p>Has anyone else noticed terrible performance when scaling up to use all the cores on a cloud instance with somewhat memory intense jobs (2.5GB in my case)?</p>
  
  <p>When I run jobs locally on my quad xeon chip, the difference between using 1 core and all 4 cores is about a 25% slowdown with all cores.  This is to be expected from what I understand; a drop in clock rate as the cores get used up is part of the multi-core chip design.</p>
  
  <p>But when I run the jobs on a multicore virtual instance, I am seeing a <strong>slowdown of like 2x - 4x in processing time</strong> between using 1 core and all cores. I've seen this on GCE, EC2, and Rackspace instances. And I have tested many difference instance types, mostly the fastest offered.</p>
  
  <p>So has this behavior been seen by others with jobs about the same size in memory usage?  </p>
  
  <p>The jobs I am running are written in fortran. I did not write them, and I'm not really a fortran guy so my knowledge of them is limited. I know they have low I/O needs.  They appear to be CPU-bound when I watch <i>top</i> as they run. They run without the need to communicate with each other, ie., embarrasingly parallel.  They each take about 2.5GB in memory.</p>
  
  <p>So my best guess so far is that jobs that use up this much memory take a big hit by the virtualization layer's memory management.  It could also be that my jobs are competing for an I/O resource, but this seems highly unlikely according to an expert.</p>
  
  <p>My workaround for now is to use GCE because they have single-core instance that actually runs the jobs as fast as my laptop's chip, and are priced almost proportionally by core.  </p>