title: Undestanding timings on a NUMA multiprocessor machine
body: <p>I am new to the topic of NUMA. I also need to say that I am a programmer and don't have a deep knowledge of hardware.</p>
  
  <p>I am working on a Quad Operton 6272 server. The motherboard is <a href="http://www.supermicro.com/Aplus/motherboard/Opteron6100/SR56x0/H8QGi_-F.cfm" rel="nofollow">SuperMicro H8QGi+-F</a>, there is a total of 132GB of memory (8 <a href="http://www.eio.com/p-45887-samsung-m393b2g70bh0-yh9-135v-ddr3-1333-16gb2gx72-eccreg-samsung-chip-server-memory.aspx" rel="nofollow">16GB sticks</a>).</p>
  
  <p>The memory sticks are installed into motherboard slots 1A and 2A - two per Operton "package". <a href="http://developer.amd.com/wordpress/media/2012/10/51803A_OpteronLinuxTuningGuide_SCREEN.pdf" rel="nofollow">This document</a> explains that an Operton "CPU" is a hierarchical thing: package->die->module->core. With this setup 'numactl --hardware' reports 4 NUMA nodes, 16 CPUs and 32GB memory each. I am not sure putting the memory sticks into slots 1A and 2A is the best thing to do, but this is what I am experimenting with ATM.</p>
  
  <p>I wrote a test C++ program to help me understand the properties of NUMA memory access</p>
  
  <pre><code>#include &lt;iostream&gt;
  #include &lt;numa.h&gt;
  #include &lt;pthread.h&gt;
  #include &lt;time.h&gt;
  #include &lt;omp.h&gt;
  #include &lt;cassert&gt;
  
  using namespace std;
  
  const unsigned bufferSize = 50000000;
  
  void pin_to_core(size_t core)
  {
      cpu_set_t cpuset;
      CPU_ZERO(&amp;cpuset);
      CPU_SET(core, &amp;cpuset);
      pthread_setaffinity_np(pthread_self(), sizeof(cpu_set_t), &amp;cpuset);
  }
  
  int main()
  {
      srand(0);
  
      int num_cpus = numa_num_task_cpus();
  
      unsigned* buffers[64] = {0};
  
      for( unsigned whoAllocates = 0; whoAllocates &lt; 64; whoAllocates += 8 )
      {
          cout &lt;&lt; "BUFFERS ARE ALLOCATED BY CORE " &lt;&lt; whoAllocates &lt;&lt; std::endl;
  
          for( unsigned whichProc = 0; whichProc &lt; 4; ++whichProc )
          {
              double firstIter1 = 0.0; // The first iterations of cores 0-7 will be summed here
              double firstIter2 = 0.0; // for cores 8-15
              double allIter1 = 0.0; // all iter cores 0-7
              double allIter2 = 0.0; // all iter cores 8-15
  #pragma omp parallel
              {
                  assert(omp_get_num_threads() == num_cpus);
                  int tid = omp_get_thread_num();
                  pin_to_core( tid );
  
  #pragma omp barrier
                  if( tid == whoAllocates )
                  {
                      for( unsigned i = 0; i &lt; 64; ++i )
                      {
                          if( !( i &gt;= 16*whichProc &amp;&amp; i &lt; 16 * (whichProc + 1) ) )
                              continue;
                          buffers[i] = static_cast&lt;unsigned*&gt;( numa_alloc_local( bufferSize * sizeof(unsigned) ) );
                          for( unsigned j = 0; j &lt; bufferSize; ++j )
                              buffers[i][j] = rand();
                     }
                  }
  
  #pragma omp barrier
  
                  if( tid &gt;= 16*whichProc &amp;&amp; tid &lt; 16 * (whichProc + 1) )
                  {
                      timespec t1;
                      clock_gettime( CLOCK_MONOTONIC, &amp;t1 );
  
                      unsigned* b = buffers[tid];
  
                      unsigned tmp = 0;
                      unsigned iCur = 0;
                      double dt = 0.0;
                      for( unsigned cnt = 0; cnt &lt; 20; ++cnt )
                      {
                          for( unsigned j = 0; j &lt; bufferSize/10; ++j )
                          {
                              b[iCur] = ( b[iCur] + 13567 ) / 2;
                              tmp += b[iCur];
                              iCur = (iCur + 7919) % bufferSize;
                          }
                          if( cnt == 0 )
                          {
                              timespec t2;
                              clock_gettime( CLOCK_MONOTONIC, &amp;t2 );
                              dt = t2.tv_sec - t1.tv_sec + t2.tv_nsec * 0.000000001 - t1.tv_nsec * 0.000000001;
                          }
                      }
  
  
  #pragma omp critical
                      {
                          timespec t3;
                          clock_gettime( CLOCK_MONOTONIC, &amp;t3 );
                          double totaldt = t3.tv_sec - t1.tv_sec + t3.tv_nsec * 0.000000001 - t1.tv_nsec * 0.000000001;
                          if( (tid % 16) &lt; 8 )
                          {
                              firstIter1 += dt;
                              allIter1 += totaldt;
                          }
                          else
                          {
                              firstIter2 += dt;
                              allIter2 += totaldt;
                          }
                      }
                  }
  
  #pragma omp barrier
  
                  if( tid == whoAllocates )
                  {
                      for( unsigned i = 0; i &lt; 64; ++i )
                      {
                          if( i &gt;= 16*whichProc &amp;&amp; i &lt; 16 * (whichProc + 1) )
                              numa_free( buffers[i], bufferSize * sizeof(unsigned) );
                      }
                  }
              }
              cout &lt;&lt; firstIter1 / 8.0 &lt;&lt; "|" &lt;&lt; allIter1 / 8.0 &lt;&lt; " / " &lt;&lt; firstIter2 / 8.0 &lt;&lt; "|" &lt;&lt; allIter2 / 8.0 &lt;&lt; std::endl;
          }
          cout &lt;&lt; std::endl;
      }
  
      return 0;
  }
  </code></pre>
  
  <p>This program allocates buffers, fills them with random integer numbers and does some meaningless computations on them. With loop iterations we vary the thread/core number that allocates the buffers and the core/thread numbers that do the work. The memory allocations are done on threads 0,8,16,...,56. At one time only 16 threads are doing calculations, they are threads <em>16i</em> through <em>16(i+1)</em>.</p>
  
  <p>I am calculating times required to do one unit of work and to do 20 units of work. This is done to see the change of speed when some of the threads finish execution.</p>
  
  <p>From my earlier experiments I have noticed that memory access times for threads <em>8i</em> through <em>8i+7</em> is identical. So I'm just outputting the average timings over 8 samples.</p>
  
  <p>Let me describe the structure of the output produced by my program. On the outermost level there are blocks, each corresponding to one thread doing memory allocation/initialization. Each such block contains 4 lines, each line corresponding to one of the Operton "packages" doing the calculation (if the allocating core belongs to the current Operton "package" then we expect the work to be done quickly). Each line consists of 2 parts: the first part corresponding to cores 0-7 of the package and the 2nd part corresponding to cores 8-15.</p>
  
  <p>Here is the output:</p>
  
  <pre><code>BUFFERS ARE ALLOCATED BY CORE 0
  0.500514|9.9542 / 1.51007|16.5094
  2.2603|45.1606 / 2.2775|45.3465
  1.68496|28.2412 / 1.08619|21.6404
  1.77763|28.9919 / 1.10469|22.1162
  
  BUFFERS ARE ALLOCATED BY CORE 8
  0.493291|9.9364 / 1.56316|16.5003
  2.26248|45.1783 / 2.27799|45.3355
  1.68429|28.25 / 1.08653|21.6459
  1.74917|29.0526 / 1.10497|22.1448
  
  BUFFERS ARE ALLOCATED BY CORE 16
  1.7351|28.0653 / 1.07199|21.462
  0.492752|9.8367 / 1.56163|16.5719
  2.24607|44.8697 / 2.27301|45.1844
  3.1222|45.1603 / 1.91962|37.9283
  
  BUFFERS ARE ALLOCATED BY CORE 24
  1.68059|28.0659 / 1.07882|21.4894
  0.492256|9.83806 / 1.56651|16.5694
  2.24318|44.9446 / 2.30389|45.1441
  3.12939|45.1632 / 1.90041|37.9657
  
  BUFFERS ARE ALLOCATED BY CORE 32
  2.2715|45.1583 / 2.2762|45.3947
  1.6862|28.1196 / 1.07878|21.561
  0.491057|9.82909 / 1.55539|16.5337
  3.13294|45.1643 / 1.89497|37.8627
  
  BUFFERS ARE ALLOCATED BY CORE 40
  2.26877|45.1215 / 2.28221|45.3919
  1.68416|28.1208 / 1.07998|21.5642
  0.491796|9.81286 / 1.56934|16.5408
  3.12412|45.1824 / 1.91072|37.8004
  
  BUFFERS ARE ALLOCATED BY CORE 48
  2.36897|46.8026 / 2.35386|47.0751
  3.16056|45.265 / 1.89596|38.0117
  3.14169|45.1464 / 1.89043|37.8944
  0.493718|9.84713 / 1.56139|16.5472
  
  BUFFERS ARE ALLOCATED BY CORE 56
  2.35647|46.823 / 2.36314|47.0848
  3.12441|45.2807 / 1.90549|38.0006
  3.12573|45.1325 / 1.89693|37.8699
  0.491999|9.83378 / 1.56538|16.5302
  </code></pre>
  
  <p>For example, the fourth line in the block corresponding to allocation by core #16 is "3.1222|45.1603 / 1.91962|37.9283". This means that on average it took cores 48-55 3.1222s to do the first unit of work and 45.1603s to do all 20 units of work (it is not 20 times more because apparently there is a speedup when cores 56-63 have finished). The second half of the line tells us that on average it took cores 56-63 1.91962s to complete the first iteration and 37.9283s to do all 20 iterations.</p>
  
  <p>Things I cannot understand:</p>
  
  <ol>
  <li>When allocation is done, for example, on thread 8 threads 0-7 still finish the job before threads 8-15. I would expect the thread that does the allocation and initialization finish at least not later than all other threads.</li>
  <li>There is some asymmetry between the four Operton packages. For example, on average access to package1's memory (when allocated by cores 0 or 8) is faster than to package4's memory (allocated by cores 48 or 56).</li>
  </ol>
  
  <p>Can anyone give any insights to why this is happening?</p>