local BLAS calls with the corresponding cuBLAS functions, as is done for the full-to-banded transformation (Eq. \[eq:2stage\_fwd1\]), the solution of the tridiagonal system (Eq. \[eq:2stage\_solve\]), and the banded-to-full back-transformation (Eq. \[eq:2stage\_bkwd2\]). The tridiagonal-to-banded back-transformation (Eq. \[eq:2stage\_bkwd1\]) is GPU accelerated by a CUDA implementation of Eq. \[eq:householder\]. The banded-to-tridiagonal transformation (Eq. \[eq:2stage\_fwd2\]) has not been ported to GPUs, because of its low computational cost as shown in Fig. \[fig:decomp5\_summit\].

GPU Offloading via cuBLAS {#subsec:gpu_cublas}
-------------------------

The API of cuBLAS is designed to be almost identical to that of the standard CPU BLAS, making cuBLAS-based GPU offloading straightforward. Here we only comment on two technical aspects. First, the communication and synchronization between CPUs and GPUs should be avoided as much as possible. Before calling a cuBLAS function, the input arrays must reside on the GPU memory, which often requires a copy of the data from CPU to GPU. In order for the CPU to access the result of cuBLAS, another copy from GPU to CPU is needed. In our GPU porting of ELPA2, CPU-GPU memory transfers are reduced to minimum by leaving data on the GPU as long as possible. Most often, GPU data is copied to the CPU in order to 