is an $m \times m$ orthonormal matrix whose columns $U_i$ are called the left singular vectors. $\Sigma$ is an $m \times n$ diagonal matrix whose diagonal entries $\sigma_i$ are called the singular values and are sorted in decreasing order. $V$ is an $n \times n$ orthonormal matrix whose columns $V_i$ are called the right singular vectors. When $m > n$, we can compute a reduced form $A = \hat{U} \hat{\Sigma} V^*$ where $\hat{U}$ is an $m \times n$ matrix and $\hat{\Sigma}$ is an $n \times n$ diagonal matrix. One can easily obtain the full form from the reduced one by extending $\hat{U}$ with $(m - n)$ orthogonal vectors and $\hat{\Sigma}$ with an $(m - n)$ zero block row. Without any loss of generality, we will focus on the reduced SVD of real matrices in our discussions.

The SVD of a matrix is a crucial component in many applications in signal processing and statistics as well as matrix compression, where truncating the $(n - k)$ singular values that are smaller than some threshold gives us a rank-$k$ approximation $\tilde{A}$ of the matrix $A$. This matrix is the unique minimizer of the function $f_k(B) = || A - B ||_F$. In the 