title: Sending data with PACKET_MMAP and PACKET_TX_RING is slower than "normal" (without)
body: <p>I’m writing a traffic generator in C using the PACKET_MMAP socket option to create a ring buffer to send data over a raw socket. The ring buffer is filled with Ethernet frames to send and <code>sendto</code> is called. The entire contents of the ring buffer is sent over the socket which should give higher performance than having a buffer in memory and calling <code>sendto</code> repeatedly for every frame in the buffer that needs sending.</p>
  
  <p>When not using PACKET_MMAP, upon calling <code>sendto</code> a single frame is copied from the buffer in the user-land memory to an SK buf in kernel memory, then the kernel must copy the packet to memory accessed by the NIC for DMA and signal the NIC to DMA the frame into it's own hardware buffers and queue it for transmission. When using the PACKET_MMAP socket option mmapped memory is allocated by the application and linked to the raw socket. The application places packets into the mmapped buffer, calls <code>sendto</code> and instead of the Kernel having to copy the packets into an SK buf it can read them from the mmapped buffer directly. Also "blocks" of packets can be read from the ring buffer instead of individual packets/frames. So the performance increase is one sys-call to copy multiple frames and one less copy action for each frame to get it into the NIC hardware buffers.</p>
  
  <p><strong>When I am comparing the performance of a socket using PACKET_MMAP to a “normal” socket (a char buffer with a single packet in it) there is no performance benefit at all. Why is this?</strong> When using PACKET_MMAP in Tx mode, only one frame can be put into each ring block (rather than multiple frames per ring block as with Rx mode) however I am creating 256 blocks so we should be sending 256 frames in a single <code>sendto</code> call right?</p>
  
  <p>Performance with PACKET_MMAP, <code>main()</code> calls <code>packet_tx_mmap()</code>:</p>
  
  <pre><code>bensley@ubuntu-laptop:~/C/etherate10+$ sudo taskset -c 1 ./etherate_mt -I 1
  Using inteface lo (1)
  Running in Tx mode
  1. Rx Gbps 0.00 (0) pps 0   Tx Gbps 17.65 (2206128128) pps 1457152
  2. Rx Gbps 0.00 (0) pps 0   Tx Gbps 19.08 (2385579520) pps 1575680
  3. Rx Gbps 0.00 (0) pps 0   Tx Gbps 19.28 (2409609728) pps 1591552
  4. Rx Gbps 0.00 (0) pps 0   Tx Gbps 19.31 (2414260736) pps 1594624
  5. Rx Gbps 0.00 (0) pps 0   Tx Gbps 19.30 (2411935232) pps 1593088
  </code></pre>
  
  <p>Performance without PACKET_MMAP, <code>main()</code> calls <code>packet_tx()</code>:</p>
  
  <pre><code>bensley@ubuntu-laptop:~/C/etherate10+$ sudo taskset -c 1 ./etherate_mt -I 1
  Using inteface lo (1)
  Running in Tx mode
  1. Rx Gbps 0.00 (0) pps 0   Tx Gbps 18.44 (2305001412) pps 1522458
  2. Rx Gbps 0.00 (0) pps 0   Tx Gbps 20.30 (2537520018) pps 1676037
  3. Rx Gbps 0.00 (0) pps 0   Tx Gbps 20.29 (2535744096) pps 1674864
  4. Rx Gbps 0.00 (0) pps 0   Tx Gbps 20.26 (2533014354) pps 1673061
  5. Rx Gbps 0.00 (0) pps 0   Tx Gbps 20.32 (2539476106) pps 1677329
  </code></pre>
  
  <p>The <code>packet_tx()</code> function is slightly faster than the <code>packet_tx_mmap()</code> function it seems but it is also slightly shorter so I think that minimal performance increase is simply the slightly fewer lines of code of present in <code>packet_tx</code>. So it looks to me like both the functions have practically the same performance, why is that? Why isn't PACKET_MMAP much faster, as I understand it there should be far less sys-calls and copies?</p>
  
  <pre><code>void *packet_tx_mmap(void* thd_opt_p) {
  
      struct thd_opt *thd_opt = thd_opt_p;
      int32_t sock_fd = setup_socket_mmap(thd_opt_p);
      if (sock_fd == EXIT_FAILURE) exit(EXIT_FAILURE);
  
      struct tpacket2_hdr *hdr;
      uint8_t *data;
      int32_t send_ret = 0;
      uint16_t i;
  
      while(1) {
  
          for (i = 0; i &lt; thd_opt-&gt;tpacket_req.tp_frame_nr; i += 1) {
  
              hdr = (void*)(thd_opt-&gt;mmap_buf + (thd_opt-&gt;tpacket_req.tp_frame_size * i));
              data = (uint8_t*)(hdr + TPACKET_ALIGN(TPACKET2_HDRLEN));
  
              memcpy(data, thd_opt-&gt;tx_buffer, thd_opt-&gt;frame_size);
              hdr-&gt;tp_len = thd_opt-&gt;frame_size;
              hdr-&gt;tp_status = TP_STATUS_SEND_REQUEST;
  
          }
  
          send_ret = sendto(sock_fd, NULL, 0, 0, NULL, 0);
          if (send_ret == -1) {
              perror("sendto error");
              exit(EXIT_FAILURE);
          }
  
          thd_opt-&gt;tx_pkts  += thd_opt-&gt;tpacket_req.tp_frame_nr;
          thd_opt-&gt;tx_bytes += send_ret;
  
      }
  
      return NULL;
  
  }
  </code></pre>
  
  <p>Note that the function below calls <code>setup_socket()</code> and not <code>setup_socket_mmap()</code>:</p>
  
  <pre><code>void *packet_tx(void* thd_opt_p) {
  
      struct thd_opt *thd_opt = thd_opt_p;
  
      int32_t sock_fd = setup_socket(thd_opt_p); 
  
      if (sock_fd == EXIT_FAILURE) {
          printf("Can't create socket!\n");
          exit(EXIT_FAILURE);
      }
  
      while(1) {
  
          thd_opt-&gt;tx_bytes += sendto(sock_fd, thd_opt-&gt;tx_buffer,
                                      thd_opt-&gt;frame_size, 0,
                                      (struct sockaddr*)&amp;thd_opt-&gt;bind_addr,
                                      sizeof(thd_opt-&gt;bind_addr));
          thd_opt-&gt;tx_pkts += 1;
  
      }
  
  }
  </code></pre>
  
  <p>The only difference in the socket setup functions is pasted below, but essentially its the requirements to set up a SOCKET_RX_RING or SOCKET_TX_RING:</p>
  
  <pre><code>// Set the TPACKET version, v2 for Tx and v3 for Rx
  // (v2 supports packet level send(), v3 supports block level read())
  int32_t sock_pkt_ver = -1;
  
  if(thd_opt-&gt;sk_mode == SKT_TX) {
      static const int32_t sock_ver = TPACKET_V2;
      sock_pkt_ver = setsockopt(sock_fd, SOL_PACKET, PACKET_VERSION, &amp;sock_ver, sizeof(sock_ver));
  } else {
      static const int32_t sock_ver = TPACKET_V3;
      sock_pkt_ver = setsockopt(sock_fd, SOL_PACKET, PACKET_VERSION, &amp;sock_ver, sizeof(sock_ver));
  }
  
  if (sock_pkt_ver &lt; 0) {
      perror("Can't set socket packet version");
      return EXIT_FAILURE;
  }
  
  
  memset(&amp;thd_opt-&gt;tpacket_req, 0, sizeof(struct tpacket_req));
  memset(&amp;thd_opt-&gt;tpacket_req3, 0, sizeof(struct tpacket_req3));
  
  //thd_opt-&gt;block_sz = 4096; // These are set else where
  //thd_opt-&gt;block_nr = 256;
  //thd_opt-&gt;block_frame_sz = 4096;
  
  int32_t sock_mmap_ring = -1;
  if (thd_opt-&gt;sk_mode == SKT_TX) {
  
      thd_opt-&gt;tpacket_req.tp_block_size = thd_opt-&gt;block_sz;
      thd_opt-&gt;tpacket_req.tp_frame_size = thd_opt-&gt;block_sz;
      thd_opt-&gt;tpacket_req.tp_block_nr = thd_opt-&gt;block_nr;
      // Allocate per-frame blocks in Tx mode (TPACKET_V2)
      thd_opt-&gt;tpacket_req.tp_frame_nr = thd_opt-&gt;block_nr;
  
      sock_mmap_ring = setsockopt(sock_fd, SOL_PACKET , PACKET_TX_RING , (void*)&amp;thd_opt-&gt;tpacket_req , sizeof(struct tpacket_req));
  
  } else {
  
      thd_opt-&gt;tpacket_req3.tp_block_size = thd_opt-&gt;block_sz;
      thd_opt-&gt;tpacket_req3.tp_frame_size = thd_opt-&gt;block_frame_sz;
      thd_opt-&gt;tpacket_req3.tp_block_nr = thd_opt-&gt;block_nr;
      thd_opt-&gt;tpacket_req3.tp_frame_nr = (thd_opt-&gt;block_sz * thd_opt-&gt;block_nr) / thd_opt-&gt;block_frame_sz;
      thd_opt-&gt;tpacket_req3.tp_retire_blk_tov   = 1;
      thd_opt-&gt;tpacket_req3.tp_feature_req_word = 0;
  
      sock_mmap_ring = setsockopt(sock_fd, SOL_PACKET , PACKET_RX_RING , (void*)&amp;thd_opt-&gt;tpacket_req3 , sizeof(thd_opt-&gt;tpacket_req3));
  }
  
  if (sock_mmap_ring == -1) {
      perror("Can't enable Tx/Rx ring for socket");
      return EXIT_FAILURE;
  }
  
  
  thd_opt-&gt;mmap_buf = NULL;
  thd_opt-&gt;rd = NULL;
  
  if (thd_opt-&gt;sk_mode == SKT_TX) {
  
      thd_opt-&gt;mmap_buf = mmap(NULL, (thd_opt-&gt;block_sz * thd_opt-&gt;block_nr), PROT_READ | PROT_WRITE, MAP_SHARED | MAP_LOCKED | MAP_POPULATE, sock_fd, 0);
  
      if (thd_opt-&gt;mmap_buf == MAP_FAILED) {
          perror("mmap failed");
          return EXIT_FAILURE;
      }
  
  
  } else {
  
      thd_opt-&gt;mmap_buf = mmap(NULL, (thd_opt-&gt;block_sz * thd_opt-&gt;block_nr), PROT_READ | PROT_WRITE, MAP_SHARED | MAP_LOCKED | MAP_POPULATE, sock_fd, 0);
  
      if (thd_opt-&gt;mmap_buf == MAP_FAILED) {
          perror("mmap failed");
          return EXIT_FAILURE;
      }
  
      // Per bock rings in Rx mode (TPACKET_V3)
      thd_opt-&gt;rd = (struct iovec*)calloc(thd_opt-&gt;tpacket_req3.tp_block_nr * sizeof(struct iovec), 1);
  
      for (uint16_t i = 0; i &lt; thd_opt-&gt;tpacket_req3.tp_block_nr; ++i) {
          thd_opt-&gt;rd[i].iov_base = thd_opt-&gt;mmap_buf + (i * thd_opt-&gt;tpacket_req3.tp_block_size);
          thd_opt-&gt;rd[i].iov_len  = thd_opt-&gt;tpacket_req3.tp_block_size;
      }
  
  
  }
  </code></pre>
  
  <p><strong>Update 1: Result against physical interface(s)</strong>
  It was mentioned that one reason I might not be seeing a performance difference when using PACKET_MMAP was because I was sending traffic to the loopback interface (which, for one thing, doesn't have a QDISC). Since running either of the <code>packet_tx_mmap()</code> or <code>packet_tx()</code> routines can generate more than 10Gbps and I only have 10Gbps interfaces at my disposal I have bonded two together and these are the results, which show pretty much the same as above, there is minimal speed difference between the two functions:</p>
  
  <p><code>packet_tx()</code> to 20G bond0</p>
  
  <ul>
  <li>1 thread: Average 10.77Gbps~ / 889kfps~</li>
  <li>2 threads: Average 19.19Gbps~ / 1.58Mfps~</li>
  <li>3 threads: Average 19.67Gbps~ / 1.62Mfps~ (this is as
  fast as the bond will go)</li>
  </ul>
  
  <p><code>packet_tx_mmap()</code> to 20G bond0:</p>
  
  <ul>
  <li>1 thread: Average 11.08Gbps~ / 913kfps~</li>
  <li>2 threads: Average 19.0Gbps~ / 1.57Mfps~</li>
  <li>3 threads: Average 19.66Gbps~ / 1.62Mfps~ (this is as
  fast as the bond will go)</li>
  </ul>
  
  <p>This was with frames 1514 bytes in size (to keep it the same as the original loopback tests above).</p>
  
  <p>In all of the above tests the number of soft IRQs was roughly the same (measured using <a href="https://gist.github.com/jwbensley/6e1f0c0f8133a723351fbce56cd8b78d" rel="noreferrer">this script</a>). With one thread running <code>packet_tx()</code> there was circa 40k interrupts per second on a CPU core. With 2 and 3 threads running there 40k on 2 and 3 core respectively. The results when using <code>packet_tx_mmap()</code> where the same. Circa 40k soft IRQs for a single thread on one CPU core. 40k per core when running 2 and 3 threads.</p>
  
  <p><strong>Update 2: Full Source Code</strong></p>
  
  <p>I have uploaded the full source code now, I'm still writing this application so it probably has plenty of flaws but they are outside the scope of this question: <a href="https://github.com/jwbensley/EtherateMT" rel="noreferrer">https://github.com/jwbensley/EtherateMT</a></p>