title: linux socket recv buffer packet drops even for "pktcount * pktsize" < SO_RCVBUF
body: <p>I am sending icmp echo to 250 nodes in C program.
  socket is created like this</p>
  
  <pre><code>sfd = socket(PF_INET, SOCK_RAW, IPPROTO_ICMP);
  </code></pre>
  
  <p>I have getsockopt SO_RCVBUF :262142 .i.e     262KB
  as 84 bytes(including IP, 100bytes on wire)  are required for one packet, the rcv buffer should be able to hold
  262142/84 = 3120 packets</p>
  
  <pre><code>sysctl parameters  (defaults)
  net.core.rmem_max     = 131071
  net.core.rmem_default = 113664
  </code></pre>
  
  <p>But there were 10+ drops.</p>
  
  <p>I am sending all echo requests, and then use recvfrom() to get packets.
  It is obvious that reply packets get accumulated in rcv buffer of socket,
  but the rcv buffer is big enough to hold 3120 packets.</p>
  
  <p>Packets show properly in ehtereal on receiving host.</p>
  
  <p>When I set:</p>
  
  <pre><code>sysctl -w net.core.rmem_max=1048576
  and SO_RCVBUF to 2MB, the drops were 0.
  </code></pre>
  
  <p>Why is this so?</p>
  
  <p>There are few queues in action.</p>
  
  <ol>
  <li>the network card ring buffer.</li>
  <li>nic to kernel que</li>
  <li>per socket rcv buffer</li>
  </ol>
  
  <p>I guess net.core.rmem_max will change only per socket rcv buffer.</p>
  
  <p>Any links in right direction.</p>
  
  <pre><code>Platform Linux suse10/x86
  </code></pre>
  
  <h2>    NIC : Intel Corporation PRO/Wireless 2200BG</h2>
  
  <p>-- adding more </p>
  
  <p>I disabled the said wireless interface and started using wired interface
  Ethernet controller: Broadcom Corporation NetXtreme BCM5705M_2 Gigabit Ethernet (rev 03)
  The situation has changed significantly.</p>
  
  <pre><code>net.core.netdev_budget = 300
  net.core.netdev_max_backlog = 1000
  net.core.rmem_max = 131071
  net.core.wmem_max = 131071
  
  getsockopt SO_RCVBUF :262142
  getsockopt SO_SNDBUF :262142
  # ethtool -g eth1
  Current hardware settings:
  RX:             200
  TX:             200
  #
  
  eth1 txqueuelen:1000
  </code></pre>
  
  <p>now it is like 0 drops per 250 packets and around 170 per 1000
  changed rx,tx with ethtool -G from default of 200 but that had no effect.</p>
  
  <p>then I changed  per socket rcv buffer max</p>
  
  <pre><code>sysctl -w net.core.rmem_max=1024000
  </code></pre>
  
  <p>this allowed 0 drops per 6000
  and 500 drops per 7000</p>
  
  <p>further increasing per-socket-rcv-buffer-max</p>
  
  <pre><code>sysctl -w net.core.rmem_max=2048000
  </code></pre>
  
  <p>0 drops per 7000 packets</p>
  
  <p>wired net gave more consistent and cooperative results.</p>
  
  <p>But question remains open
  when 262142 bytes buffer can hold 3000 packets of size 84(with IP header), why drops happened at 1000 packets.
  Even the on wire packet is 100 byte max and 262142 can hold 2600 packets.</p>
  
  <p>ethereal is able to get the packets most of times, and ifconfig counters show no drops so it is something after the nic driver has given the packets to the kernel.</p>
  
  <p>ethereal also missed few packets but that is less occuring.</p>
  
  <p>by changing only these of above </p>
  
  <pre><code>sysctl -w net.core.rmem_max=1024000
  </code></pre>
  
  <p>drops 96/1000
  drops 0/6000</p>
  
  <h2>drops 500/7000</h2>
  
  <pre><code>sysctl -w net.core.rmem_max=2048000
  </code></pre>
  
  <h2>drops 0/7000</h2>
  
  <pre><code>sysctl -w net.core.rmem_max=512000
  sysctl -w net.core.netdev_max_backlog=3000
  </code></pre>
  
  <h2>drops 0/3000</h2>
  
  <pre><code>sysctl -w net.core.rmem_max=256000
  sysctl -w net.core.netdev_max_backlog=3000
  </code></pre>
  
  <h2>drops 1400/3000</h2>
  
  <pre><code>hold = 2 * 1024 * 1024;
  setsockopt(sfd, SOL_SOCKET, SO_RCVBUF, (char *)&amp;hold, sizeof(hold));
  
  for(;;) 
  ...
   if(recvfrom(sfd,(char *)packet, packlen, 0, (struct sockaddr *)&amp;from, &amp;fromlen)) &lt; 0) {
      rcvcount++;
      process_packet(packet,packlen);
   }
  </code></pre>