title: Poor performance due to hyper-threading with OpenMP: how to bind threads to cores
body: <p>I am developing large dense matrix multiplication code.  When I profile the code it sometimes gets about 75% of the peak flops of my four core system and other times gets about 36%.  The efficiency does not change between executions of the code.  It either starts at 75% and continues with that efficiency or starts at 36% and continues with that efficiency.</p>
  
  <p>I have traced the problem down to hyper-threading and the fact that I set the number of threads to four instead of the default eight. <strong>When I disable hyper-threading in the BIOS I get about 75% efficiency consistently (or at least I never see the drastic drop to 36%).</strong>  </p>
  
  <p>Before I call any parallel code I do <code>omp_set_num_threads(4)</code>.  I have also tried <code>export OMP_NUM_THREADS=4</code> before I run my code but it seems to be equivalent.  </p>
  
  <p>I don't want to disable hyper-threading in the BIOS.  I think I need to bind the four threads to the four cores.  I have tested some different cases of <code>GOMP_CPU_AFFINITY</code> but so far I still have the problem that the efficiency is 36% sometimes.  <strong>What is the mapping with hyper-threading and cores?</strong>  E.g. do thread 0 and thread 1 correspond to the the same core and thread 2 and thread 3 another core?</p>
  
  <p><strong>How can I bind the threads to each core without thread migration so that I don't have to disable hyper-threading in the BIOS?</strong>  Maybe I need to look into using <a href="http://man7.org/linux/man-pages/man2/sched_setaffinity.2.html">sched_setaffinity</a>?</p>
  
  <p>Some details of my current system: Linux kernel 3.13, GCC 4.8,Intel Xeon E5-1620 (four physical cores, eight hyper-threads).</p>
  
  <p>Edit:
  This seems to be working well so far</p>
  
  <pre><code>export GOMP_CPU_AFFINITY="0 1 2 3 4 5 6 7"
  </code></pre>
  
  <p>or </p>
  
  <pre><code>export GOMP_CPU_AFFINITY="0-7"
  </code></pre>
  
  <p>Edit:
  This seems also to work well</p>
  
  <pre><code>export OMP_PROC_BIND=true
  </code></pre>
  
  <p>Edit:
  <a href="http://www.glennklockwood.com/comp/affinity.php">These options</a> also work well (gemm is the name of my executable)</p>
  
  <pre><code>numactl -C 0,1,2,3 ./gemm
  </code></pre>
  
  <p>and</p>
  
  <pre><code>taskset -c 0,1,2,3 ./gemm
  </code></pre>