title: Should we use multiple acceptor sockets to accept a large number of connections?
body: <p>As known, SO_REUSEPORT allows <strong>multiple sockets to listen on the same IP address and port</strong> combination, it increases requests per second by <strong>2 to 3 times</strong>, and reduces both latency <strong>(~30%)</strong> and the standard deviation for latency <strong>(8 times)</strong>: <a href="https://www.nginx.com/blog/socket-sharding-nginx-release-1-9-1/" rel="noreferrer">https://www.nginx.com/blog/socket-sharding-nginx-release-1-9-1/</a></p>
  
  <blockquote>
    <p>NGINX release 1.9.1 introduces a new feature that enables use of the
    <strong>SO_REUSEPORT</strong> socket option, which is available in newer versions of many operating systems, including DragonFly BSD and <strong>Linux (kernel
    version 3.9 and later)</strong>. This socket option allows <strong>multiple sockets
    to listen on the same IP address and port combination</strong>. The kernel
    then load balances incoming connections across the sockets.
    ...</p>
    
    <p>As shown in the figure, <strong>reuseport increases requests per second by 2
    to 3 times, and reduces both latency</strong> and the standard deviation for
    latency.</p>
  </blockquote>
  
  <p><a href="https://i.stack.imgur.com/hSngR.png" rel="noreferrer"><img src="https://i.stack.imgur.com/hSngR.png" alt="enter image description here"></a></p>
  
  <p><a href="https://i.stack.imgur.com/BVVAv.jpg" rel="noreferrer"><img src="https://i.stack.imgur.com/BVVAv.jpg" alt="enter image description here"></a></p>
  
  <p><a href="https://i.stack.imgur.com/twDR5.png" rel="noreferrer"><img src="https://i.stack.imgur.com/twDR5.png" alt="enter image description here"></a></p>
  
  <hr>
  
  <p><strong><code>SO_REUSEPORT</code> is available on most modern OS</strong>: Linux (<a href="https://kernelnewbies.org/Linux_3.9" rel="noreferrer">kernel >= 3.9</a> since <a href="https://github.com/torvalds/linux/releases/tag/v3.9" rel="noreferrer">29 Apr 2013</a>), Free/Open/NetBSD, MacOS, iOS/watchOS/tvOS, <a href="https://www.ibm.com/support/knowledgecenter/en/ssw_aix_72/com.ibm.aix.commtrf2/getsockopt.htm" rel="noreferrer">IBM AIX 7.2</a>, <a href="https://docs.oracle.com/cd/E26502_01/html/E29044/tcp-7p.html" rel="noreferrer">Oracle Solaris 11.1</a>, Windows (is only <code>SO_REUSEPORT</code> that behaves as 2 flags together <code>SO_REUSEPORT</code>+<code>SO_REUSEADDR</code> in BSD), and may be on <a href="https://android.googlesource.com/platform/development/+/73a5a3b/ndk/platforms/android-20/arch-mips64/include/asm/socket.h#32" rel="noreferrer">Android</a> :   <a href="https://stackoverflow.com/a/14388707/1558037">https://stackoverflow.com/a/14388707/1558037</a></p>
  
  <blockquote>
    <p>Linux >= 3.9</p>
    
    <ol start="2">
    <li>Additionally the kernel performs some "special magic" for <code>SO_REUSEPORT</code> sockets that isn't found in other operating systems:
    For UDP sockets, it tries to distribute datagrams evenly, for <strong>TCP
    listening sockets, it tries to distribute incoming connect requests</strong>
    (those accepted by calling <code>accept()</code>)  evenly across all the sockets
    that share the same address and port combination. Thus <strong>an application
    can easily open the same port in multiple child processes and then use
    <code>SO_REUSEPORT</code> to get a very inexpensive load balancing</strong>.</li>
    </ol>
  </blockquote>
  
  <hr>
  
  <p>Also known, to avoid locks of spin-lock and achive high performance there shouldn't be sockets which read more than 1 thread. I.e. each thread should processes its own sockets for read/write.</p>
  
  <ul>
  <li><code>accept()</code> is thread-safe function for the same socket descriptor, so it should be guarded by lock - <strong>so lock contention reduces performance</strong>: <a href="http://unix.derkeiler.com/Newsgroups/comp.unix.programmer/2007-06/msg00246.html" rel="noreferrer">http://unix.derkeiler.com/Newsgroups/comp.unix.programmer/2007-06/msg00246.html</a></li>
  </ul>
  
  <blockquote>
    <p>POSIX.1-2001/SUSv3 <strong>requires accept(), bind(), connect(), listen(),
    socket(), send(), recv(), etc. to be thread-safe functions.</strong> It's
    possible that there are some ambiguities in the standard regarding
    their interaction with threads, but the intention is that their
    behaviour in multithreaded programs is governed by the standard.</p>
  </blockquote>
  
  <ul>
  <li>If we use the same one socket from many threads, then performance will be low because socket defended by lock for thread-safe accessing from many threads: <a href="https://blog.cloudflare.com/how-to-receive-a-million-packets/" rel="noreferrer">https://blog.cloudflare.com/how-to-receive-a-million-packets/</a></li>
  </ul>
  
  <blockquote>
    <p>The receiving performance is down compared to a single threaded
    program. That's <strong>caused by a lock contention on the UDP receive buffer</strong>
    side. Since both threads are using the same socket descriptor, they
    spend a disproportionate amount of time fighting for a lock around the
    UDP receive buffer. This paper describes the problem in more detail.</p>
  </blockquote>
  
  <ul>
  <li>More details about spin-lock when  the  application  tries  to  read data  from  the  socket - "Analysis of Linux UDP Sockets Concurrent Performance": <a href="http://www.jcc2014.ucm.cl/jornadas/WORKSHOP/WSDP%202014/WSDP-4.pdf" rel="noreferrer">http://www.jcc2014.ucm.cl/jornadas/WORKSHOP/WSDP%202014/WSDP-4.pdf</a></li>
  </ul>
  
  <blockquote>
    <p>V.  K ERNEL ISOLATION</p>
    
    <p>....</p>
    
    <p>From  the  other  side,  <strong>when  the  application  tries  to  read data 
    from  the  socket</strong>,  it  executes  a  similar  process,  which 
    isdescribed below and represented in Figure 3 from right to left: </p>
    
    <p>1) Dequeue  one  or  more  packets  from  the  receive  queue, <strong>using the
    corresponding spinlock</strong> (green one). </p>
    
    <p>2)  Copy the information to user-space memory. </p>
    
    <p>3)  Release the memory used by the packet. This
    potentiallychanges the state of the socket, so two ways of locking the
    socket  can  occur:  fast  and  slow.  In  both  cases,  the packet is
    unlinked from the socket, Memory Accounting statistics are updated and
    socket is released according to the locking path taken.</p>
  </blockquote>
  
  <p>I.e. when many threads are accessing the same socket, performance degrades due to waiting on one spin-lock.</p>
  
  <hr>
  
  <p>We have 2 x Xeon 32 HT-Cores server with 64 total HT-cores, and two 10 Gbit Ethernet cards, and Linux (kernel 3.9).</p>
  
  <p>We use RFS and XPS - i.e. for the same connection TCP/IP-stack processed (kernel-space) on the same CPU-Core as an application thread (user-space).</p>
  
  <p>There are at least 3 ways to accept connections to processes it at many threads:</p>
  
  <ul>
  <li>Use <strong>one acceptor socket shared between many threads</strong>, and each thread accept connections and processes it</li>
  <li>Use <strong>one acceptor socket in 1 thread</strong>, and this thread push received socket descriptors of connections to other thread-workers by using thread-safe queue</li>
  <li>Use <strong>many acceptor sockets</strong> which listen the same <code>ip:port</code>, 1 individual acceptor socket <strong>in each thread</strong>, and the thread that receives the connection then processes it (recv/send)</li>
  </ul>
  
  <p>What is the more efficient way, If we accept a lot of new TCP-connections?</p>