title: C++ code compiled using Homebrew's Clang on macOS Apple Silicon runs significantly slower with OpenMP than without
body: <p>I am trying to compile OpenMP-enabled C++ code on macOS Monterey (12.5.1) on Apple Silicon (Apple M1 Max) to get the respective speed-ups.</p>
  <p>The code is compute-heavy, well understood and uses OpenMP since years (e.g. on x64 Ubuntu) without problems; the computations are more or less embarrassing parallel, so the speed-ups from running the computation via OpenMP in multi-threaded manner are significant.</p>
  <p>When I compile the code on macOS with Apple Clang everything works, except that of course the code runs single-threaded since AppleClang does not support OpenMP.</p>
  <p>That's why I have compiled now everything with Homebrew's Clang to enable OpenMP. Unfortunately, the results are different than expected. The code compiles and uses OpenMP; I see that since I had to fix a few minor issues on <code>firstprivate</code> clauses where Clang is more strict than GCC and I see the execution using multiple threads.</p>
  <p>However, the runtime <em>with</em> OpenMP is <em>significantly slower</em> than <em>without</em> OpenMP. For instance, the computation without OpenMP takes ~15 seconds, with OpenMP it takes ~90 seconds, and even in the worse scaling case, I'd expect a speed-up of 2x-4x. Otherwise the results of the computation are correct, it just seems to affect the speed.</p>
  <p>I have tried to compile the software with both llvm 14 (stable 14.0.6 (bottled)) and llvm 13 (stable 13.0.1 (bottled), tried due to a suspected regression) but without success so far.</p>
  <p>The project is using CMake and the configuration looks as follows (this one is with LLVM 13, but I had tried with the default llvm beforehand):</p>
  <pre><code>export PATH=&quot;/opt/homebrew/opt/llvm@13/bin:$PATH&quot;
  ...
  cmake ../../ -DCMAKE_BUILD_TYPE=Release  \
    -DCMAKE_PREFIX_PATH=/opt/homebrew/opt/llvm@13   \
    -DCMAKE_CXX_COMPILER=/opt/homebrew/opt/llvm@13/bin/clang++   \
    -DCMAKE_C_COMPILER=/opt/homebrew/opt/llvm@13/bin/clang   \
    -DLDFLAGS=&quot;-L/opt/homebrew/opt/llvm@13/lib&quot;   \
    -DCPPFLAGS=&quot;-I/opt/homebrew/opt/llvm@13/include&quot;
  </code></pre>
  <p>I would be glad for any advice on how to resolve this issue. Thanks in advance!</p>
  <p>--</p>
  <p><strong>Update 1:</strong></p>
  <p>When OMP_DISPLAY_ENV is enabled, the following is shown:</p>
  <pre><code>OPENMP DISPLAY ENVIRONMENT BEGIN
     _OPENMP='201611'
    [host] OMP_AFFINITY_FORMAT='OMP: pid %P tid %i thread %n bound to OS proc set {%A}'
    [host] OMP_ALLOCATOR='omp_default_mem_alloc'
    [host] OMP_CANCELLATION='FALSE'
    [host] OMP_DEFAULT_DEVICE='0'
    [host] OMP_DISPLAY_AFFINITY='FALSE'
    [host] OMP_DISPLAY_ENV='TRUE'
    [host] OMP_DYNAMIC='FALSE'
    [host] OMP_MAX_ACTIVE_LEVELS='1'
    [host] OMP_MAX_TASK_PRIORITY='0'
    [host] OMP_NESTED: deprecated; max-active-levels-var=1
    [host] OMP_NUM_TEAMS='0'
    [host] OMP_NUM_THREADS: value is not defined
    [host] OMP_PROC_BIND='false'
    [host] OMP_SCHEDULE='static'
    [host] OMP_STACKSIZE='8176k'
    [host] OMP_TARGET_OFFLOAD=DEFAULT
    [host] OMP_TEAMS_THREAD_LIMIT='0'
    [host] OMP_THREAD_LIMIT='2147483647'
    [host] OMP_TOOL='enabled'
    [host] OMP_TOOL_LIBRARIES: value is not defined
    [host] OMP_TOOL_VERBOSE_INIT: value is not defined
    [host] OMP_WAIT_POLICY='PASSIVE'
  OPENMP DISPLAY ENVIRONMENT END
  </code></pre>
  <p>Once I set OMP_NUM_THREADS to a specific number, my test case yields the following runtimes (time via Boost's cpu_timer). One can clearly see that the code is fastest when running on a single thread (set via OMP_NUM_THREADS=1), even though it has been compiled with OpenMP support.</p>
  <ul>
  <li>OMP_NUM_THREADS=1:  <code>15.85s wall, 30.97s user + 0.56s system = 31.53s CPU (198.9%)</code></li>
  <li>OMP_NUM_THREADS=2:  <code>28.91s wall, 69.93s user + 0.71s system = 70.64s CPU (244.3%)</code></li>
  <li>OMP_NUM_THREADS=4:  <code>36.63s wall, 134.07s user + 1.47s system = 135.54s CPU (370.0%)</code></li>
  <li>OMP_NUM_THREADS=8:  <code>52.18s wall, 267.80s user + 12.35s system = 280.15s CPU (536.9%)</code></li>
  <li>OMP_NUM_THREADS=10: <code>52.29s wall, 285.78s user + 10.57s system = 296.35s CPU (566.8%)</code></li>
  </ul>
  <p><strong>Update 2:</strong></p>
  <p>System load from asitop (<a href="https://github.com/tlkh/asitop" rel="nofollow noreferrer">https://github.com/tlkh/asitop</a>):</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th>OMP_NUM_THREADS=1</th>
  <th>OMP_NUM_THREADS=8</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td><a href="https://i.stack.imgur.com/FF8oe.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/FF8oe.png" alt="System load from asitop when OMP_NUM_THREADS=1" /></a></td>
  <td><a href="https://i.stack.imgur.com/fuaGn.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/fuaGn.png" alt="System load from asitop when OMP_NUM_THREADS=8" /></a></td>
  </tr>
  </tbody>
  </table>
  </div>