overall GPU utilization. It is thus recommended to use ELPA with MPS switched on.

CUDA Implementation of Parallel Householder Transformations {#subsec:gpu_step4}
-----------------------------------------------------------

In this section, we present our CUDA implementation of the local computation in Fig. \[fig:step4\], which is the key step in the tridiagonal-to-banded back-transformation (Eq. \[eq:2stage\_bkwd1\]). In a CUDA program, the large number of GPU cores are arranged into a grid of blocks, each of which in turn comprises a grid of threads. All the GPU cores work in a single instruction, multiple thread (SIMT) fashion, i.e., a single instruction is simultaneously executed by multiple threads with different data [@cuda_nickolls_2008]. As explained in Sec. \[subsec:cpu\_step4\], in a given step of the tridiagonal-to-banded back-transformation, a process is responsible for applying $N_v$ Householder transformations to its local eigenvector matrix $\boldsymbol{X}_\text{local}$, which has $N_R$ rows and $N_C$ columns. A Householder transformation defined by $\tau$ and $\boldsymbol{v}$ is applied to an eigenvector $\boldsymbol{x}$ by $$\label{eq:kernel}
(\boldsymbol{I} - \tau \boldsymbol{v} \boldsymbol{v}^*) \boldsymbol{x} = \boldsymbol{x} - \tau \boldsymbol{v} (\boldsymbol{v}^* \boldsymbol{x}) .$$

The $N_v$ Householder transformations are applied sequentially. The order in which they are applied in the tridiagonal-to-banded back-transformation is the reverse of the order in which they are generated in the banded-to-tridiagonal transformation. 