title: Hoist two class variables out of OpenMP parallel loop
body: <p>I'm working on a parallel version of <a href="https://www.tarsnap.com/scrypt.html" rel="nofollow noreferrer"><code>scrypt</code></a> using OpenMP. The program is at the very end of the post.</p>
  
  <p>The parallelizable part works fine (1) without OpenMP and (2) with OpenMP and the class variables <code>XY</code> and <code>V</code> in the loop:</p>
  
  <pre><code>// 2: for i = 0 to p - 1 do
  #pragma omp parallel for
  for (unsigned int i = 0; i &lt; parallel; ++i)
  {
      SecByteBlock XY(static_cast&lt;size_t&gt;(blockSize * 256U));
      SecByteBlock  V(static_cast&lt;size_t&gt;(blockSize * cost * 128U));
  
      // 3: B_i &lt;-- MF(B_i, N)
      const ptrdiff_t offset = static_cast&lt;ptrdiff_t&gt;(blockSize*i*128);
      Smix(B+offset, static_cast&lt;size_t&gt;(blockSize), cost, V, XY);
  }
  </code></pre>
  
  <p>The <code>SecByteBlock</code> wraps <code>new</code> and <code>delete</code> and acts a lot like a <code>vector</code>. The problem is, <code>SecByteBlock</code> has a zeroizer. Every time the class is destructed all the memory is set to 0. Then, a new one is created for the thread at the top of the loop. It is inefficient.</p>
  
  <p>I tried to hoist <code>XY</code> and <code>V</code> out of the loop so they would be destructed once. The attempts are shown below, but they all result in segmentation faults:</p>
  
  <pre><code>$ time OMP_NUM_THREADS=4 ./test.exe
  Threads: 4
  Segmentation fault (core dumped)
  
  real    0m9.561s
  user    0m0.011s
  sys     0m2.750s
  </code></pre>
  
  <p>How does one hoist a class variable out of the parallel for loop so it does not go through unwanted create/destruct cycles?</p>
  
  <hr>
  
  <p><strong><em>Hoist out of loop</em></strong>:</p>
  
  <pre><code>SecByteBlock XY(static_cast&lt;size_t&gt;(blockSize * 256U));
  SecByteBlock  V(static_cast&lt;size_t&gt;(blockSize * cost * 128U));
  
  // 2: for i = 0 to p - 1 do
  #pragma omp parallel for
  for (unsigned int i = 0; i &lt; parallel; ++i)
  {
      // 3: B_i &lt;-- MF(B_i, N)
      const ptrdiff_t offset = static_cast&lt;ptrdiff_t&gt;(blockSize*i*128);
      Smix(B+offset, static_cast&lt;size_t&gt;(blockSize), cost, V, XY);
  }
  </code></pre>
  
  <p><strong><em>Thread private</em></strong>:</p>
  
  <pre><code>SecByteBlock XY(static_cast&lt;size_t&gt;(blockSize * 256U));
  SecByteBlock  V(static_cast&lt;size_t&gt;(blockSize * cost * 128U));
  
  // 2: for i = 0 to p - 1 do
  #pragma omp parallel for private(XY) private(V)
  for (unsigned int i = 0; i &lt; parallel; ++i)
  {
      // 3: B_i &lt;-- MF(B_i, N)
      const ptrdiff_t offset = static_cast&lt;ptrdiff_t&gt;(blockSize*i*128);
      Smix(B+offset, static_cast&lt;size_t&gt;(blockSize), cost, V, XY);
  }
  </code></pre>
  
  <p><strong><em>Parallel section and thread private</em></strong>:</p>
  
  <pre><code>#pragma omp parallel
  {
      SecByteBlock XY(static_cast&lt;size_t&gt;(blockSize * 256U));
      SecByteBlock  V(static_cast&lt;size_t&gt;(blockSize * cost * 128U));
  
      // 2: for i = 0 to p - 1 do
      #pragma omp parallel for private(XY) private(V)
      for (unsigned int i = 0; i &lt; parallel; ++i)
      {
          // 3: B_i &lt;-- MF(B_i, N)
          const ptrdiff_t offset = static_cast&lt;ptrdiff_t&gt;(blockSize*i*128);
          Smix(B+offset, static_cast&lt;size_t&gt;(blockSize), cost, V, XY);
      }
  }
  </code></pre>
  
  <hr>
  
  <p>Here is the test program. It is Valgrind, Coverity and Sanitizer clean with the class variables in the loop. The problems arise when hoisting them out of the loop.</p>
  
  <pre><code>$ cat test.cxx
  #include "cryptlib.h"
  #include "secblock.h"
  #include "scrypt.h"
  #include "osrng.h"
  #include "files.h"
  #include "hex.h"
  
  #include &lt;iostream&gt;
  #include &lt;omp.h&gt;       
  
  int main()
  {
      int threads = 1;
      #pragma omp parallel
      {
          threads = omp_get_num_threads();
      }
      std::cout &lt;&lt; "Threads: " &lt;&lt; threads &lt;&lt; std::endl;
  
      using namespace CryptoPP;
      SecByteBlock derived(64);
  
      const byte pwd[] = "password";
      const byte salt[] = "NaCl";
  
      Scrypt scrypt;
      scrypt.DeriveKey(derived, derived.size(), pwd, 8, salt, 4, 1&lt;&lt;20, 8, 16);
  
      std::cout &lt;&lt; "Derived: ";
      StringSource(derived, 16, true, new HexEncoder(new FileSink(std::cout)));
      std::cout &lt;&lt; "..." &lt;&lt; std::endl;
  
      return 0;
  }
  </code></pre>