title: Enhanced REP MOVSB for memcpy
body: <p>I would like to use enhanced REP MOVSB (ERMSB) to get a high bandwidth for a custom <code>memcpy</code>.  </p>
  
  <p>ERMSB was introduced with the Ivy Bridge microarchitecture. See the section "Enhanced REP MOVSB and STOSB operation (ERMSB)" in the <a href="https://www-ssl.intel.com/content/www/us/en/architecture-and-technology/64-ia-32-architectures-optimization-manual.html" rel="noreferrer">Intel optimization manual</a> if you don't know what ERMSB is.</p>
  
  <p>The only way I know to do this directly is with inline assembly.  I got the following function from <a href="https://groups.google.com/forum/#!topic/gnu.gcc.help/-Bmlm_EG_fE" rel="noreferrer">https://groups.google.com/forum/#!topic/gnu.gcc.help/-Bmlm_EG_fE</a></p>
  
  <pre><code>static inline void *__movsb(void *d, const void *s, size_t n) {
    asm volatile ("rep movsb"
                  : "=D" (d),
                    "=S" (s),
                    "=c" (n)
                  : "0" (d),
                    "1" (s),
                    "2" (n)
                  : "memory");
    return d;
  }
  </code></pre>
  
  <p>When I use this however, the bandwidth is much less than with <code>memcpy</code>.
  <code>__movsb</code> gets 15 GB/s and <code>memcpy</code> get 26 GB/s with my i7-6700HQ (Skylake) system, Ubuntu 16.10, DDR4@2400 MHz dual channel 32 GB, GCC 6.2.</p>
  
  <p><strong>Why is the bandwidth so much lower with <code>REP MOVSB</code>? What can I do to improve it?</strong></p>
  
  <p>Here is the code I used to test this.    </p>
  
  <pre><code>//gcc -O3 -march=native -fopenmp foo.c
  #include &lt;stdlib.h&gt;
  #include &lt;string.h&gt;
  #include &lt;stdio.h&gt;
  #include &lt;stddef.h&gt;
  #include &lt;omp.h&gt;
  #include &lt;x86intrin.h&gt;
  
  static inline void *__movsb(void *d, const void *s, size_t n) {
    asm volatile ("rep movsb"
                  : "=D" (d),
                    "=S" (s),
                    "=c" (n)
                  : "0" (d),
                    "1" (s),
                    "2" (n)
                  : "memory");
    return d;
  }
  
  int main(void) {
    int n = 1&lt;&lt;30;
  
    //char *a = malloc(n), *b = malloc(n);
  
    char *a = _mm_malloc(n,4096), *b = _mm_malloc(n,4096);
    memset(a,2,n), memset(b,1,n);
  
    __movsb(b,a,n);
    printf("%d\n", memcmp(b,a,n));
  
    double dtime;
  
    dtime = -omp_get_wtime();
    for(int i=0; i&lt;10; i++) __movsb(b,a,n);
    dtime += omp_get_wtime();
    printf("dtime %f, %.2f GB/s\n", dtime, 2.0*10*1E-9*n/dtime);
  
    dtime = -omp_get_wtime();
    for(int i=0; i&lt;10; i++) memcpy(b,a,n);
    dtime += omp_get_wtime();
    printf("dtime %f, %.2f GB/s\n", dtime, 2.0*10*1E-9*n/dtime);  
  }
  </code></pre>
  
  <hr>
  
  <p>The reason I am interested in <code>rep movsb</code> is based off these comments</p>
  
  <blockquote>
    <p>Note that on Ivybridge and Haswell, with buffers to large to fit in MLC you can beat movntdqa using rep movsb; movntdqa incurs a RFO into LLC, rep movsb does not...
    rep movsb is significantly faster than movntdqa when streaming to memory on Ivybridge and Haswell (but be aware that pre-Ivybridge it is slow!)</p>
  </blockquote>
  
  <p><a href="https://stackoverflow.com/questions/26246040/whats-missing-sub-optimal-in-this-memcpy-implementation/26256216#comment41203823_26256216">What&#39;s missing/sub-optimal in this memcpy implementation?</a></p>
  
  <hr>
  
  <p>Here are my results on the same system from <a href="https://github.com/ssvb/tinymembench" rel="noreferrer">tinymembnech</a>.</p>
  
  <pre><code> C copy backwards                                     :   7910.6 MB/s (1.4%)
   C copy backwards (32 byte blocks)                    :   7696.6 MB/s (0.9%)
   C copy backwards (64 byte blocks)                    :   7679.5 MB/s (0.7%)
   C copy                                               :   8811.0 MB/s (1.2%)
   C copy prefetched (32 bytes step)                    :   9328.4 MB/s (0.5%)
   C copy prefetched (64 bytes step)                    :   9355.1 MB/s (0.6%)
   C 2-pass copy                                        :   6474.3 MB/s (1.3%)
   C 2-pass copy prefetched (32 bytes step)             :   7072.9 MB/s (1.2%)
   C 2-pass copy prefetched (64 bytes step)             :   7065.2 MB/s (0.8%)
   C fill                                               :  14426.0 MB/s (1.5%)
   C fill (shuffle within 16 byte blocks)               :  14198.0 MB/s (1.1%)
   C fill (shuffle within 32 byte blocks)               :  14422.0 MB/s (1.7%)
   C fill (shuffle within 64 byte blocks)               :  14178.3 MB/s (1.0%)
   ---
   standard memcpy                                      :  12784.4 MB/s (1.9%)
   standard memset                                      :  30630.3 MB/s (1.1%)
   ---
   MOVSB copy                                           :   8712.0 MB/s (2.0%)
   MOVSD copy                                           :   8712.7 MB/s (1.9%)
   SSE2 copy                                            :   8952.2 MB/s (0.7%)
   SSE2 nontemporal copy                                :  12538.2 MB/s (0.8%)
   SSE2 copy prefetched (32 bytes step)                 :   9553.6 MB/s (0.8%)
   SSE2 copy prefetched (64 bytes step)                 :   9458.5 MB/s (0.5%)
   SSE2 nontemporal copy prefetched (32 bytes step)     :  13103.2 MB/s (0.7%)
   SSE2 nontemporal copy prefetched (64 bytes step)     :  13179.1 MB/s (0.9%)
   SSE2 2-pass copy                                     :   7250.6 MB/s (0.7%)
   SSE2 2-pass copy prefetched (32 bytes step)          :   7437.8 MB/s (0.6%)
   SSE2 2-pass copy prefetched (64 bytes step)          :   7498.2 MB/s (0.9%)
   SSE2 2-pass nontemporal copy                         :   3776.6 MB/s (1.4%)
   SSE2 fill                                            :  14701.3 MB/s (1.6%)
   SSE2 nontemporal fill                                :  34188.3 MB/s (0.8%)
  </code></pre>
  
  <p>Note that on my system <code>SSE2 copy prefetched</code> is also faster than <code>MOVSB copy</code>.</p>
  
  <hr>
  
  <p>In my original tests I did not disable turbo. I disabled turbo and tested again and it does not appear to make much of a difference.  However, changing the power management does make a big difference.</p>
  
  <p>When I do </p>
  
  <pre><code>sudo cpufreq-set -r -g performance
  </code></pre>
  
  <p>I sometimes see over 20 GB/s with <code>rep movsb</code>.</p>
  
  <p>with </p>
  
  <pre><code>sudo cpufreq-set -r -g powersave
  </code></pre>
  
  <p>the best I see is about 17 GB/s.  But <code>memcpy</code> does not seem to be sensitive to the power management. </p>
  
  <hr>
  
  <p>I checked the frequency (using <code>turbostat</code>) <a href="https://askubuntu.com/a/620114/259597">with and without SpeedStep enabled</a>, with <code>performance</code> and with <code>powersave</code> for idle, a 1 core load and a 4 core load.  I ran Intel's MKL dense matrix multiplication to create a load and set the number of threads using <code>OMP_SET_NUM_THREADS</code>.  Here is a table of the results (numbers in GHz).</p>
  
  <pre><code>              SpeedStep     idle      1 core    4 core
  powersave     OFF           0.8       2.6       2.6
  performance   OFF           2.6       2.6       2.6
  powersave     ON            0.8       3.5       3.1
  performance   ON            3.5       3.5       3.1
  </code></pre>
  
  <p>This shows that with <code>powersave</code> even with SpeedStep disabled the CPU
  still clocks down to the idle frequency of <code>0.8 GHz</code>.  It's only with <code>performance</code> without SpeedStep that the CPU runs at a constant frequency.</p>
  
  <p>I used e.g <code>sudo cpufreq-set -r performance</code> (because <code>cpufreq-set</code> was giving strange results) to change the power settings. This turns turbo back on so I had to disable turbo after. </p>