title: how can 2 threads share the same cache line
body: <p>I am using a custom network protocol library. This library is built on TCP/IP and is supposedly to be used in high frequency messaging. It is a non-blocking library and uses callbacks as the interface to integrate with the caller.</p>
  
  <p>I am no performance expert, and that is why I decided to ask this question here. The custom library comes with a particular constraint, outlined below:</p>
  
  <blockquote>
    <p>"Callee should not invoke any of the library's API under the context
    of the callback thread. If they attempt to do so, the thread will
    hang"</p>
  </blockquote>
  
  <p>The only way to overcome API restriction is that I start another thread which process message and invokes the library to send a response. The library thread and process thread would share a common queue, which would be protected by a mutex and use <code>wait_notify()</code> calls to indicate the presence of a message. </p>
  
  <p>If I am receiving 80k messages per second, then I would be putting threads to sleep and waking them up pretty often, performing thread context switches ~80k times per second.</p>
  
  <p>Plus, as there are two threads, they will not share the message buffer in the L1 cache. The cache line containing the message would first be filled by the library's thread, then evicted and pulled into the process thread's core's L1 cache. Am I missing something or is it possible that the library's design is not meant for high performance use cases?</p>
  
  <p><strong>My questions are:</strong></p>
  
  <ol>
  <li><p>I have seen the warnings like "Don't use this API in a callback's context as it can cause locks." across many libraries. What are the common design choices that cause such design constraints? They can use recursive locks if it is a simple question of same thread calling the lock multiple times. Is this a re-entrant issue, and what challenges might cause an API owner to make non re-entrant API?</p></li>
  <li><p>Is there a way in the above design model, where the library thread and process thread can share the same core, and consequently share a cache line?</p></li>
  <li><p>How expensive are volatile <code>sig_atomic_t</code>'s as a mechanism to share data between two threads?</p></li>
  <li><p>Given a high frequency scenario, what is a light-weight way to share information between two threads?</p></li>
  </ol>
  
  <p>The library and my application are built on C++ and Linux.</p>