title: Threading and scaling model for TCP server with epoll
body: <p>I've read the <a href="http://www.kegel.com/c10k.html" rel="noreferrer">C10K</a> doc as well as many related papers on scaling up a socket server. All roads point to the following:</p>
  
  <ol>
  <li><p>Avoid the classic mistake of "thread per connection".</p></li>
  <li><p>Prefer epoll over select.</p></li>
  <li><p>Likewise, legacy async io mechanism in unix may be hard to use.</p></li>
  </ol>
  
  <p>My simple TCP server just listens for client connections on a listen socket on a dedicated port. Upon receiving a new connection, parses the request, and sends a response back. Then gracefully closes the socket.</p>
  
  <p>I think I have a good handle on how to scale this up on a single thread using epoll. Just one loop that calls epoll_wait for the listen socket as well as for the existing client connections.  Upon return, the code will handle new creating new client connections as well as managing state of existing connections depending on which socket just got signaled. And perhaps some logic to manage connection timeouts, graceful closing of sockets, and efficient resource allocation for each connection.  Seems straightforward enough.</p>
  
  <p><strong>But what if I want to scale this to take advantage of multiple threads and multiple cpu cores?</strong>  The core idea that springs to mind is this:</p>
  
  <p>One dedicated thread for listening for incoming connections on the TCP listen socket.  Then a set of N threads (or thread pool) to handle all the active concurrent client connections.  Then invent some thread safe way in which the listen thread will "dispatch" the new connection (socket) to one of the available worker threads. (ala IOCP in Windows).  The worker thread will use an epoll loop on all the connections it is handling to do what the single threaded approach would do.</p>
  
  <p>Am I on the right track?  Or is there a standard design pattern for doing a TCP server with epoll <em>on multiple threads</em>?</p>
  
  <p>Suggestions on how the listen thread would dispatch a new connection to the thread pool?</p>