participate in an MPI communication call. In the version discussed in this paper, we have not yet explored GPU-aware MPI to directly communicate data on the GPU. In each of the GPU-accelerated subroutines, the allocation of GPU memory, which implies a CPU-GPU synchronization, is performed before the main work begins by precomputing the size of the allocation. The allocated GPU memory is reused wherever possible, and is deallocated after the main work finishes. Avoiding frequent allocation and deallocation of GPU memory in a loop or in a recursive routine greatly reduces the amount of implicit CPU-GPU synchronization.

Second, the CPU code of the GPU-accelerated version of ELPA2 operates in a pure MPI mode without threading, i.e., one MPI task for each CPU core. As most (if not all) mainstream computers today have more CPU cores than GPU devices, several MPI tasks would have to share one GPU device. When the size of an eigenproblem is relatively small, the amount of work assigned to each individual MPI task may not be able to fully saturate the GPU. In such cases, the NVIDIA Multi-Process Service (MPS) transparently allows work from different MPI tasks to be executed concurrently on the GPU, increasing the 