title: C++ concurrency: Variable visibility outside of mutexes
body: <p>I'm having some trouble understanding when variables are forced to be written to memory, even outside of mutex blocks. I apologize for the convoluted code below, because I have stripped away logic that deals with whether <code>reader</code> decides if some data is stale. The important thing to note is that 99.9% of the time, readers will take the <code>fast path</code> and synchronization must be very fast, which is why I use an atomic <code>int32</code> to communicate both staleness and whether the slow path is now necessary.</p>
  
  <p>I have the following setup, which I am "fairly" certain is race-free:</p>
  
  <pre><code>#define NUM_READERS 10
  
  BigObject               mSharedObject;
  std::atomic_int32_t     mStamp = 1;
  std::mutex              mMutex;
  std::condition_variable mCondition;
  int32_t                 mWaitingReaders = 0;
  
  void reader() {
      for (;;) { // thread loop
          for (;;) { // spin until stamp is acceptible
              int32_t stamp = mStamp.load();
              if (stamp &gt; 0) { // fast path
                  if (stampIsAcceptible(stamp) &amp;&amp; 
                      mStamp.compare_exchange_weak(stamp, stamp + 1)) {
                      break;
                  }
              } else { // slow path
                  // tell the loader (writer) that we're halted
                  std::unique_lock&lt;mutex&gt; lk(mMutex);
                  mWaitingReaders++;
                  mCondition.notify_all();
                  while (mWaitingReaders != 0) {
                      mCondition.wait(lk);
                  } // ###
                  lk.unlock();
                  // *** THIS IS WHERE loader's CHANGES TO mSharedObject
                  // *** MUST BE VISIBLE TO THIS THREAD!
              }
          }
          // stamp acceptible; mSharedObject guaranteed not written to
  
          mSharedObject.accessAndDoFunStuff();
  
          mStamp.fetch_sub(1); // part of hidden staleness logic
      }
  }
  
  void loader() {
      for (;;) { // thread loop
          // spin until we somehow decide we want to change mSharedObject!
          while (meIsHappySleeping()) {}
  
          // we want to modify mSharedObject, so set mStamp to 0 and wait
          // for readers to see this and report that they are now waiting
          int32_t oldStamp = mStamp.exchange(0);
          unique_lock&lt;mutex&gt; lk(mMutex);
          while (mWaitingReaders != NUM_READERS) {
              mCondition.wait(lk);
          }
          // all readers are waiting. start writing to mSharedObject
          mSharedObject.loadFromFile("example.foo");
          mStamp.store(oldStamp);
          mWaitingReaders = 0; // report completion
          lk.unlock();
          mCondition.notify_all();
          // *** NOW loader's CHANGES TO mSharedObject
          // *** MUST BE VISIBLE TO THE READER THREADS!
      }
  }
  
  void setup() {
      for (int i = 0; i &lt; NUM_READERS; i++) {
          std::thread t(reader); t.detach();
      }
      std::thead t(loader); t.detach();
  }
  </code></pre>
  
  <p>The parts marked in stars <code>***</code> are what concerns me. This is because while my code excludes races (as far as I can see), <code>mSharedObject</code> is only protected by a mutex while being written to by <code>loader()</code>. Because <code>reader()</code> needs to be extremely fast (as noted above), I do not want its read-only accesses to <code>mSharedObject</code> to have to be protected by a mutex.</p>
  
  <p>One "guaranteed" solution is to introduce a thread-local variable <code>const BigObject *latestObject</code> at line <code>###</code>, which is set to <code>&amp;mSharedObject</code> and then use that for access. But is this bad practice? And is it really necessary? Will the atomic operations / mutex release operations guarantee that readers see the changes?</p>
  
  <p>Thanks!</p>