and NVIDIA Tesla Volta V100 GPUs.

Eigensolvers operating on hybrid CPU-GPU machines have long been available in GPU-oriented linear algebra packages such as cuSOLVER [@cusolver] and MAGMA [@magma_tomov_2010; @magma_dongarra_2014]. These packages are designed and optimized for shared-memory host architectures. They can be very fast, but the problem size they can tackle is limited by the memory capacity of a single compute node. Fully exploiting the power of GPU-accelerated supercomputers would require a distributed-memory implementation. The MPI-parallel, distributed-memory ELPA library implements the conventional one-stage diagonalization method and the two-stage diagonalization proposed in Refs. [@2stage_lang_1993; @2stage_bischof_1994], known as the “ELPA1” and “ELPA2” solvers, respectively. Both of them have been ported to GPUs by substituting BLAS calls with the corresponding cuBLAS functions [@elpa_kus_2019a], making ELPA the only publicly available, distributed-memory, GPU-accelerated eigensolver to our knowledge. The CPU-only version of ELPA1 and ELPA2 and the GPU-accelerated version of ELPA1 and ELPA2 are hereafter referred to as CPU-ELPA1, CPU-ELPA2, GPU-ELPA1, and GPU-ELPA2, respectively.

In single-node tests published by Ks et al. [@elpa_kus_2019a; @elpa_kus_2019b], the performance of GPU-ELPA1 is better than that of GPU-ELPA2. When using 2 IBM POWER8 CPUs (24 cores in total) and 4 NVIDIA Pascal P100 GPUs, GPU-ELPA1 delivers up to 11.9x performance 