title: Poor memcpy Performance on Linux
body: <p>We have recently purchased some new servers and are experiencing poor memcpy performance. The memcpy performance is 3x slower on the servers compared to our laptops.</p>
  
  <p><strong>Server Specs</strong></p>
  
  <ul>
  <li>Chassis and Mobo: SUPER MICRO 1027GR-TRF</li>
  <li>CPU: 2x Intel Xeon E5-2680 @ 2.70 Ghz</li>
  <li>Memory: 8x 16GB DDR3 1600MHz</li>
  </ul>
  
  <p>Edit: I am also testing on another server with slightly higher specs and seeing the same results as the above server</p>
  
  <p><strong>Server 2 Specs</strong></p>
  
  <ul>
  <li>Chassis and Mobo: SUPER MICRO 10227GR-TRFT</li>
  <li>CPU: 2x Intel Xeon E5-2650 v2 @ 2.6 Ghz</li>
  <li>Memory: 8x 16GB DDR3 1866MHz</li>
  </ul>
  
  <p><strong>Laptop Specs</strong></p>
  
  <ul>
  <li>Chassis: Lenovo W530</li>
  <li>CPU: 1x Intel Core i7 i7-3720QM @ 2.6Ghz</li>
  <li>Memory: 4x 4GB DDR3 1600MHz</li>
  </ul>
  
  <p><strong>Operating System</strong></p>
  
  <pre><code>$ cat /etc/redhat-release
  Scientific Linux release 6.5 (Carbon) 
  $ uname -a                      
  Linux r113 2.6.32-431.1.2.el6.x86_64 #1 SMP Thu Dec 12 13:59:19 CST 2013 x86_64 x86_64 x86_64 GNU/Linux
  </code></pre>
  
  <p><strong>Compiler (on all systems)</strong></p>
  
  <pre><code>$ gcc --version
  gcc (GCC) 4.6.1
  </code></pre>
  
  <p>Also tested with gcc 4.8.2 based on a suggestion from @stefan. There was no performance difference between compilers.</p>
  
  <p><strong>Test Code</strong>
  The test code below is a canned test to duplicate the problem i am seeing in our production code. I know this benchmark is simplistic but it was able to exploit and identify our problem. The code creates two 1GB buffers and memcpys between them, timing the memcpy call. You can specify alternate buffer sizes on the command line using: ./big_memcpy_test [SIZE_BYTES]</p>
  
  <pre><code>#include &lt;chrono&gt;
  #include &lt;cstring&gt;
  #include &lt;iostream&gt;
  #include &lt;cstdint&gt;
  
  class Timer
  {
   public:
    Timer()
        : mStart(),
          mStop()
    {
      update();
    }
  
    void update()
    {
      mStart = std::chrono::high_resolution_clock::now();
      mStop  = mStart;
    }
  
    double elapsedMs()
    {
      mStop = std::chrono::high_resolution_clock::now();
      std::chrono::milliseconds elapsed_ms =
          std::chrono::duration_cast&lt;std::chrono::milliseconds&gt;(mStop - mStart);
      return elapsed_ms.count();
    }
  
   private:
    std::chrono::high_resolution_clock::time_point mStart;
    std::chrono::high_resolution_clock::time_point mStop;
  };
  
  std::string formatBytes(std::uint64_t bytes)
  {
    static const int num_suffix = 5;
    static const char* suffix[num_suffix] = { "B", "KB", "MB", "GB", "TB" };
    double dbl_s_byte = bytes;
    int i = 0;
    for (; (int)(bytes / 1024.) &gt; 0 &amp;&amp; i &lt; num_suffix;
         ++i, bytes /= 1024.)
    {
      dbl_s_byte = bytes / 1024.0;
    }
  
    const int buf_len = 64;
    char buf[buf_len];
  
    // use snprintf so there is no buffer overrun
    int res = snprintf(buf, buf_len,"%0.2f%s", dbl_s_byte, suffix[i]);
  
    // snprintf returns number of characters that would have been written if n had
    //       been sufficiently large, not counting the terminating null character.
    //       if an encoding error occurs, a negative number is returned.
    if (res &gt;= 0)
    {
      return std::string(buf);
    }
    return std::string();
  }
  
  void doMemmove(void* pDest, const void* pSource, std::size_t sizeBytes)
  {
    memmove(pDest, pSource, sizeBytes);
  }
  
  int main(int argc, char* argv[])
  {
    std::uint64_t SIZE_BYTES = 1073741824; // 1GB
  
    if (argc &gt; 1)
    {
      SIZE_BYTES = std::stoull(argv[1]);
      std::cout &lt;&lt; "Using buffer size from command line: " &lt;&lt; formatBytes(SIZE_BYTES)
                &lt;&lt; std::endl;
    }
    else
    {
      std::cout &lt;&lt; "To specify a custom buffer size: big_memcpy_test [SIZE_BYTES] \n"
                &lt;&lt; "Using built in buffer size: " &lt;&lt; formatBytes(SIZE_BYTES)
                &lt;&lt; std::endl;
    }
  
  
    // big array to use for testing
    char* p_big_array = NULL;
  
    /////////////
    // malloc 
    {
      Timer timer;
  
      p_big_array = (char*)malloc(SIZE_BYTES * sizeof(char));
      if (p_big_array == NULL)
      {
        std::cerr &lt;&lt; "ERROR: malloc of " &lt;&lt; SIZE_BYTES &lt;&lt; " returned NULL!"
                  &lt;&lt; std::endl;
        return 1;
      }
  
      std::cout &lt;&lt; "malloc for " &lt;&lt; formatBytes(SIZE_BYTES) &lt;&lt; " took "
                &lt;&lt; timer.elapsedMs() &lt;&lt; "ms"
                &lt;&lt; std::endl;
    }
  
    /////////////
    // memset
    {
      Timer timer;
  
      // set all data in p_big_array to 0
      memset(p_big_array, 0xF, SIZE_BYTES * sizeof(char));
  
      double elapsed_ms = timer.elapsedMs();
      std::cout &lt;&lt; "memset for " &lt;&lt; formatBytes(SIZE_BYTES) &lt;&lt; " took "
                &lt;&lt; elapsed_ms &lt;&lt; "ms "
                &lt;&lt; "(" &lt;&lt; formatBytes(SIZE_BYTES / (elapsed_ms / 1.0e3)) &lt;&lt; " bytes/sec)"
                &lt;&lt; std::endl;
    }
  
    /////////////
    // memcpy 
    {
      char* p_dest_array = (char*)malloc(SIZE_BYTES);
      if (p_dest_array == NULL)
      {
        std::cerr &lt;&lt; "ERROR: malloc of " &lt;&lt; SIZE_BYTES &lt;&lt; " for memcpy test"
                  &lt;&lt; " returned NULL!"
                  &lt;&lt; std::endl;
        return 1;
      }
      memset(p_dest_array, 0xF, SIZE_BYTES * sizeof(char));
  
      // time only the memcpy FROM p_big_array TO p_dest_array
      Timer timer;
  
      memcpy(p_dest_array, p_big_array, SIZE_BYTES * sizeof(char));
  
      double elapsed_ms = timer.elapsedMs();
      std::cout &lt;&lt; "memcpy for " &lt;&lt; formatBytes(SIZE_BYTES) &lt;&lt; " took "
                &lt;&lt; elapsed_ms &lt;&lt; "ms "
                &lt;&lt; "(" &lt;&lt; formatBytes(SIZE_BYTES / (elapsed_ms / 1.0e3)) &lt;&lt; " bytes/sec)"
                &lt;&lt; std::endl;
  
      // cleanup p_dest_array
      free(p_dest_array);
      p_dest_array = NULL;
    }
  
    /////////////
    // memmove
    {
      char* p_dest_array = (char*)malloc(SIZE_BYTES);
      if (p_dest_array == NULL)
      {
        std::cerr &lt;&lt; "ERROR: malloc of " &lt;&lt; SIZE_BYTES &lt;&lt; " for memmove test"
                  &lt;&lt; " returned NULL!"
                  &lt;&lt; std::endl;
        return 1;
      }
      memset(p_dest_array, 0xF, SIZE_BYTES * sizeof(char));
  
      // time only the memmove FROM p_big_array TO p_dest_array
      Timer timer;
  
      // memmove(p_dest_array, p_big_array, SIZE_BYTES * sizeof(char));
      doMemmove(p_dest_array, p_big_array, SIZE_BYTES * sizeof(char));
  
      double elapsed_ms = timer.elapsedMs();
      std::cout &lt;&lt; "memmove for " &lt;&lt; formatBytes(SIZE_BYTES) &lt;&lt; " took "
                &lt;&lt; elapsed_ms &lt;&lt; "ms "
                &lt;&lt; "(" &lt;&lt; formatBytes(SIZE_BYTES / (elapsed_ms / 1.0e3)) &lt;&lt; " bytes/sec)"
                &lt;&lt; std::endl;
  
      // cleanup p_dest_array
      free(p_dest_array);
      p_dest_array = NULL;
    }
  
  
    // cleanup
    free(p_big_array);
    p_big_array = NULL;
  
    return 0;
  }
  </code></pre>
  
  <p><strong>CMake File to Build</strong></p>
  
  <pre><code>project(big_memcpy_test)
  cmake_minimum_required(VERSION 2.4.0)
  
  include_directories(${CMAKE_CURRENT_SOURCE_DIR})
  
  # create verbose makefiles that show each command line as it is issued
  set( CMAKE_VERBOSE_MAKEFILE ON CACHE BOOL "Verbose" FORCE )
  # release mode
  set( CMAKE_BUILD_TYPE Release )
  # grab in CXXFLAGS environment variable and append C++11 and -Wall options
  set( CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++0x -Wall -march=native -mtune=native" )
  message( INFO "CMAKE_CXX_FLAGS = ${CMAKE_CXX_FLAGS}" )
  
  # sources to build
  set(big_memcpy_test_SRCS
    main.cpp
  )
  
  # create an executable file named "big_memcpy_test" from
  # the source files in the variable "big_memcpy_test_SRCS".
  add_executable(big_memcpy_test ${big_memcpy_test_SRCS})
  </code></pre>
  
  <p><strong>Test Results</strong></p>
  
  <pre><code>Buffer Size: 1GB | malloc (ms) | memset (ms) | memcpy (ms) | NUMA nodes (numactl --hardware)
  ---------------------------------------------------------------------------------------------
  Laptop 1         | 0           | 127         | 113         | 1
  Laptop 2         | 0           | 180         | 120         | 1
  Server 1         | 0           | 306         | 301         | 2
  Server 2         | 0           | 352         | 325         | 2
  </code></pre>
  
  <p>As you can see the memcpys and memsets on our servers is much slower than the memcpys and memsets on our laptops.</p>
  
  <p><strong>Varying buffer sizes</strong></p>
  
  <p>I have tried buffers from 100MB to 5GB all with similar results (servers slower than laptop)</p>
  
  <p><strong>NUMA Affinity</strong></p>
  
  <p>I read about people having performance issues with NUMA so i tried setting CPU and memory affinity using numactl but the results remained the same.</p>
  
  <p>Server NUMA Hardware</p>
  
  <pre><code>$ numactl --hardware                                                            
  available: 2 nodes (0-1)                                                                     
  node 0 cpus: 0 1 2 3 4 5 6 7 16 17 18 19 20 21 22 23                                         
  node 0 size: 65501 MB                                                                        
  node 0 free: 62608 MB                                                                        
  node 1 cpus: 8 9 10 11 12 13 14 15 24 25 26 27 28 29 30 31                                   
  node 1 size: 65536 MB                                                                        
  node 1 free: 63837 MB                                                                        
  node distances:                                                                              
  node   0   1                                                                                 
    0:  10  21                                                                                 
    1:  21  10 
  </code></pre>
  
  <p>Laptop NUMA Hardware</p>
  
  <pre><code>$ numactl --hardware
  available: 1 nodes (0)
  node 0 cpus: 0 1 2 3 4 5 6 7
  node 0 size: 16018 MB
  node 0 free: 6622 MB
  node distances:
  node   0 
    0:  10
  </code></pre>
  
  <p>Setting NUMA Affinity</p>
  
  <pre><code>$ numactl --cpunodebind=0 --membind=0 ./big_memcpy_test
  </code></pre>
  
  <p>Any help resolving this is greatly appreciated.</p>
  
  <p><strong>Edit: GCC Options</strong></p>
  
  <p>Based on comments i have tried compiling with different GCC options:</p>
  
  <p>Compiling with -march and -mtune set to native</p>
  
  <pre><code>g++ -std=c++0x -Wall -march=native -mtune=native -O3 -DNDEBUG -o big_memcpy_test main.cpp 
  </code></pre>
  
  <p>Result: Exact same performance (no improvement)</p>
  
  <p>Compiling with -O2 instead of -O3</p>
  
  <pre><code>g++ -std=c++0x -Wall -march=native -mtune=native -O2 -DNDEBUG -o big_memcpy_test main.cpp
  </code></pre>
  
  <p>Result: Exact same performance (no improvement)</p>
  
  <p><strong>Edit: Changed memset to write 0xF instead of 0 to avoid NULL page (@SteveCox)</strong></p>
  
  <p>No improvement when memsetting with a value other than 0 (used 0xF in this case).</p>
  
  <p><strong>Edit: Cachebench results</strong></p>
  
  <p>In order to rule out that my test program is too simplistic i downloaded a real benchmarking program LLCacheBench (<a href="http://icl.cs.utk.edu/projects/llcbench/cachebench.html">http://icl.cs.utk.edu/projects/llcbench/cachebench.html</a>)</p>
  
  <p>I built the benchmark on each machine separately to avoid architecture issues. Below are my results.</p>
  
  <p><img src="https://i.stack.imgur.com/PTqZJ.png" alt="laptop vs server memcpy performance"></p>
  
  <p>Notice the VERY large difference is performance on the larger buffer sizes. The last size tested (16777216) performed at 18849.29 MB/sec on the laptop and 6710.40 on the server. That's about a 3x difference in performance. You can also notice that the performance dropoff of the server is much steeper than on the laptop.</p>
  
  <p><strong>Edit: memmove() is 2x FASTER than memcpy() on the server</strong></p>
  
  <p>Based on some experimentation i have tried using memmove() instead of memcpy() in my test case and have found a 2x improvement on the server. Memmove() on the laptop runs slower than memcpy() but oddly enough runs at the same speed as the memmove() on the server. This begs the question, why is memcpy so slow?</p>
  
  <p>Updated Code to test memmove along with memcpy. I had to wrap the memmove() inside a function because if i left it inline GCC optimized it and performed the exact same as memcpy() (i assume gcc optimized it to memcpy because it knew the locations didn't overlap).</p>
  
  <p>Updated Results</p>
  
  <pre><code>Buffer Size: 1GB | malloc (ms) | memset (ms) | memcpy (ms) | memmove() | NUMA nodes (numactl --hardware)
  ---------------------------------------------------------------------------------------------------------
  Laptop 1         | 0           | 127         | 113         | 161       | 1
  Laptop 2         | 0           | 180         | 120         | 160       | 1
  Server 1         | 0           | 306         | 301         | 159       | 2
  Server 2         | 0           | 352         | 325         | 159       | 2
  </code></pre>
  
  <p><strong>Edit: Naive Memcpy</strong></p>
  
  <p>Based on suggestion from @Salgar i have implemented my own naive memcpy function and tested it.</p>
  
  <p>Naive Memcpy Source</p>
  
  <pre><code>void naiveMemcpy(void* pDest, const void* pSource, std::size_t sizeBytes)
  {
    char* p_dest = (char*)pDest;
    const char* p_source = (const char*)pSource;
    for (std::size_t i = 0; i &lt; sizeBytes; ++i)
    {
      *p_dest++ = *p_source++;
    }
  }
  </code></pre>
  
  <p>Naive Memcpy Results Compared to memcpy()</p>
  
  <pre><code>Buffer Size: 1GB | memcpy (ms) | memmove(ms) | naiveMemcpy()
  ------------------------------------------------------------
  Laptop 1         | 113         | 161         | 160
  Server 1         | 301         | 159         | 159
  Server 2         | 325         | 159         | 159
  </code></pre>
  
  <p><strong>Edit: Assembly Output</strong></p>
  
  <p>Simple memcpy source</p>
  
  <pre><code>#include &lt;cstring&gt;
  #include &lt;cstdlib&gt;
  
  int main(int argc, char* argv[])
  {
    size_t SIZE_BYTES = 1073741824; // 1GB
  
    char* p_big_array  = (char*)malloc(SIZE_BYTES * sizeof(char));
    char* p_dest_array = (char*)malloc(SIZE_BYTES * sizeof(char));
  
    memset(p_big_array,  0xA, SIZE_BYTES * sizeof(char));
    memset(p_dest_array, 0xF, SIZE_BYTES * sizeof(char));
  
    memcpy(p_dest_array, p_big_array, SIZE_BYTES * sizeof(char));
  
    free(p_dest_array);
    free(p_big_array);
  
    return 0;
  }
  </code></pre>
  
  <p>Assembly Output: This is the exact same on both the server and the laptop. I'm saving space and not pasting both.</p>
  
  <pre><code>        .file   "main_memcpy.cpp"
          .section        .text.startup,"ax",@progbits
          .p2align 4,,15
          .globl  main
          .type   main, @function
  main:
  .LFB25:
          .cfi_startproc
          pushq   %rbp
          .cfi_def_cfa_offset 16
          .cfi_offset 6, -16
          movl    $1073741824, %edi
          pushq   %rbx
          .cfi_def_cfa_offset 24
          .cfi_offset 3, -24
          subq    $8, %rsp
          .cfi_def_cfa_offset 32
          call    malloc
          movl    $1073741824, %edi
          movq    %rax, %rbx
          call    malloc
          movl    $1073741824, %edx
          movq    %rax, %rbp
          movl    $10, %esi
          movq    %rbx, %rdi
          call    memset
          movl    $1073741824, %edx
          movl    $15, %esi
          movq    %rbp, %rdi
          call    memset
          movl    $1073741824, %edx
          movq    %rbx, %rsi
          movq    %rbp, %rdi
          call    memcpy
          movq    %rbp, %rdi
          call    free
          movq    %rbx, %rdi
          call    free
          addq    $8, %rsp
          .cfi_def_cfa_offset 24
          xorl    %eax, %eax
          popq    %rbx
          .cfi_def_cfa_offset 16
          popq    %rbp
          .cfi_def_cfa_offset 8
          ret
          .cfi_endproc
  .LFE25:
          .size   main, .-main
          .ident  "GCC: (GNU) 4.6.1"
          .section        .note.GNU-stack,"",@progbits
  </code></pre>
  
  <p><strong>PROGRESS!!!! asmlib</strong></p>
  
  <p>Based on suggestion from @tbenson  i tried running with the <a href="http://www.agner.org/optimize/asmlib.zip">asmlib</a> version of memcpy. My results initially were poor but after changing SetMemcpyCacheLimit() to 1GB (size of my buffer) i was running at speed on par with my naive for loop!</p>
  
  <p>Bad news is that the asmlib version of memmove is slower than the glibc version, it is now running at the 300ms mark (on par with the glibc version of memcpy). Weird thing is that on the laptop when i SetMemcpyCacheLimit() to a large number it hurts performance...</p>
  
  <p>In the results below the lines marked with SetCache have SetMemcpyCacheLimit set to 1073741824. The results without SetCache do not call SetMemcpyCacheLimit()</p>
  
  <p>Results using functions from asmlib:</p>
  
  <pre><code>Buffer Size: 1GB  | memcpy (ms) | memmove(ms) | naiveMemcpy()
  ------------------------------------------------------------
  Laptop            | 136         | 132         | 161
  Laptop SetCache   | 182         | 137         | 161
  Server 1          | 305         | 302         | 164
  Server 1 SetCache | 162         | 303         | 164
  Server 2          | 300         | 299         | 166
  Server 2 SetCache | 166         | 301         | 166
  </code></pre>
  
  <p>Starting to lean towards cache issue, but what would cause this?</p>