title: ARM/neon memcpy optimized for *uncached* memory?
body: <p>I'm using a Xilinx Zynq 7000 ARM-based SoC.  I'm struggling with DMA buffers (<a href="https://stackoverflow.com/questions/34884313/need-help-mapping-pre-reserved-cacheable-dma-buffer-on-xilinx-arm-soc-zynq">Need help mapping pre-reserved **cacheable** DMA buffer on Xilinx/ARM SoC (Zynq 7000)</a>), so one thing I pursued was faster memcpy.  </p>
  
  <p>I've been looking at writing a faster memcpy for ARM using Neon instructions and inline asm.  Whatever glibc has, it's terrible, especially if we're copying from an ucached DMA buffer.</p>
  
  <p>I've put together my own copy function from various sources, including:</p>
  
  <ul>
  <li><a href="https://stackoverflow.com/questions/11161237/fast-arm-neon-memcpy">Fast ARM NEON memcpy</a></li>
  <li><a href="https://stackoverflow.com/questions/9302575/arm-inline-assembly-in-gcc">arm Inline assembly in gcc</a></li>
  <li><a href="http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.faqs/ka13544.html" rel="nofollow noreferrer">http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.faqs/ka13544.html</a></li>
  </ul>
  
  <p>The main difference for me is that I'm trying to copy from an <strong>uncached</strong> buffer because it's a DMA buffer, and ARM support for cached DMA buffers is nonexistent.</p>
  
  <p>So here's what I wrote:</p>
  
  <pre><code>void my_copy(volatile unsigned char *dst, volatile unsigned char *src, int sz)
  {
      if (sz &amp; 63) {
          sz = (sz &amp; -64) + 64;
      }
      asm volatile (
          "NEONCopyPLD:                          \n"
          "    VLDM %[src]!,{d0-d7}                 \n"
          "    VSTM %[dst]!,{d0-d7}                 \n"
          "    SUBS %[sz],%[sz],#0x40                 \n"
          "    BGT NEONCopyPLD                  \n"
          : [dst]"+r"(dst), [src]"+r"(src), [sz]"+r"(sz) : : "d0", "d1", "d2", "d3", "d4", "d5", "d6", "d7", "cc", "memory");
  }
  </code></pre>
  
  <p>The main thing I did was leave out the prefetch instruction because I figured it would be worthless on uncached memory.</p>
  
  <p>Doing this resulted in a speedup of 4.7x over the glibc memcpy.  Speed went from about 70MB/sec to about 330MB/sec.</p>
  
  <p>Unfortunately, this isn't nearly as fast as memcpy from cached memory, which runs at around 720MB/sec for system memcpy and 620MB/sec for the Neon version (probably slower because my memcpy doesn't do prefetching, perhaps).</p>
  
  <p>Can anyone help me figure out what I can do make up for this performance gap?</p>
  
  <p>I've tried a number of things like copying more at once, two loads followed by two stores.  I could try prefetch just to prove that it's useless.  Any other ideas?</p>