<s>[INST] Visualization of Singular Value decomposition of a Symmetric Matrix The Singular Value Decomposition of a matrix A satisfies 
$\mathbf A = \mathbf U \mathbf \Sigma \mathbf V^\top$
The visualization of it would look like

But when $\mathbf A$ is symmetric we can do:
$\begin{align*}
\mathbf A\mathbf A^\top&=(\mathbf U\mathbf \Sigma\mathbf V^\top)(\mathbf U\mathbf \Sigma\mathbf V^\top)^\top\\
\mathbf A\mathbf A^\top&=(\mathbf U\mathbf \Sigma\mathbf V^\top)(\mathbf V\mathbf \Sigma\mathbf U^\top)
\end{align*}$
and since $\mathbf V$ is an orthogonal matrix ($\mathbf V^\top \mathbf V=\mathbf I$), so we have:
$\mathbf A\mathbf A^\top=\mathbf U\mathbf \Sigma^2 \mathbf U^\top$
I have two questions:


*

*Is the above statement correct? when Matrix $\mathbf A$ is symmetric and we compute SVD we would get $\mathbf U\mathbf \Sigma^2 \mathbf U^\top$

*How would the decomposition looks like in a symmetric matrix? As we are getting the eigenvectors and squared eigenvalues in matrices $\mathbf U $ and $\mathbf \Sigma$ 
 [/INST] Singular value decomposition
Start with a matrix with $m$ rows, $n$ columns, and rank $\rho$,
$$
  \mathbf{A}\in\mathbb{C}^{m\times n}_{\rho}
$$
which has the singular value decomposition
$$
  \mathbf{A} = 
  \mathbf{U} \, \Sigma \, \mathbf{V}^{*} =
%
\left[ \begin{array}{cc}
  \color{blue}{\mathbf{U}_{\mathcal{R}\left(\mathbf{A}\right)}} &
  \color{red} {\mathbf{U}_{\mathcal{N}\left(\mathbf{A}^{*}\right)}}
\end{array} \right]
%
 \left[ \begin{array}{cc}
   \mathbf{S} & \mathbf{0} \\ \mathbf{0} & \mathbf{0}
 \end{array} \right]
%
\left[ \begin{array}{cc}
  \color{blue}{\mathbf{V}_{\mathcal{R}\left(\mathbf{A}^{*}\right)}} &
  \color{red} {\mathbf{V}_{\mathcal{N}\left(\mathbf{A}\right)}}
\end{array} \right]^{*}
%
$$
where the color denotes $\color{blue}{range}$ spaces and $\color{red}{null}$ spaces. The dimensions of the domain matrices are
$$
%
  \color{blue}{\mathbf{U}_{\mathcal{R}\left(\mathbf{A}\right)}} 
    \in \mathbb{C}^{m\times \rho}, \quad
%
  \color{red}{\mathbf{U}_{\mathcal{N}\left(\mathbf{A}^{*}\right)}} 
    \in \mathbb{C}^{m \times m - \rho}, \quad
%
  \color{blue}{\mathbf{V}_{\mathcal{R}\left(\mathbf{A}^{*}\right)}} 
    \in \mathbb{C}^{n\times \rho}, \quad
%
  \color{red}{\mathbf{V}_{\mathcal{N}\left(\mathbf{A}\right)}} 
    \in \mathbb{C}^{n\times n - \rho}.
$$
The domain matrices are unitary:
$$
\begin{align}
 \mathbf{U}\mathbf{U}^{*} &= \mathbf{U}^{*}\mathbf{U} = \mathbf{I}_{m} \\
 \mathbf{V}\mathbf{V}^{*} &= \mathbf{V}^{*}\mathbf{V} = \mathbf{I}_{n}
\end{align}
$$
The dimensions of the singular value matrices are
$$
%
 \Sigma \in \mathbb{R}^{m\times n}, \quad
%
  \mathbf{S} 
    \in \mathbb{R}^{\rho\times \rho}.
$$
The hermitian conjugate is constructed according to
$$
  \mathbf{A}^{*} = 
  \mathbf{V} \, \Sigma^{\mathrm{T}} \, \mathbf{U}^{*} =
%
\left[ \begin{array}{cc}
  \color{blue}{\mathbf{V}_{\mathcal{R}\left(\mathbf{A}^{*}\right)}} &
  \color{red} {\mathbf{V}_{\mathcal{N}\left(\mathbf{A}\right)}}
\end{array} \right]
%
 \left[ \begin{array}{cc}
   \mathbf{S} & \mathbf{0} \\ \mathbf{0} & \mathbf{0}
 \end{array} \right]
%
\left[ \begin{array}{cc}
  \color{blue}{\mathbf{U}_{\mathcal{R}\left(\mathbf{A}\right)}} &
  \color{red} {\mathbf{U}_{\mathcal{N}\left(\mathbf{A}^{*}\right)}}
\end{array} \right]^{*}
%
$$
where $\Sigma^{\mathrm{T}}\in \mathbb{R}^{n\times m}$.
The Moore-Penrose pseudoinverse is constructed according to
$$
  \mathbf{A}^{\dagger} = 
  \mathbf{V} \, \Sigma^{\dagger} \, \mathbf{U}^{*} =
%
\left[ \begin{array}{cc}
  \color{blue}{\mathbf{V}_{\mathcal{R}\left(\mathbf{A}^{*}\right)}} &
  \color{red} {\mathbf{V}_{\mathcal{N}\left(\mathbf{A}\right)}}
\end{array} \right]
%
 \left[ \begin{array}{cc}
   \mathbf{S}^{-1} & \mathbf{0} \\ \mathbf{0} & \mathbf{0}
 \end{array} \right]
%
\left[ \begin{array}{cc}
  \color{blue}{\mathbf{U}_{\mathcal{R}\left(\mathbf{A}\right)}} &
  \color{red} {\mathbf{U}_{\mathcal{N}\left(\mathbf{A}^{*}\right)}}
\end{array} \right]^{*}
%
$$
where $\Sigma^{\dagger}\in \mathbb{R}^{n\times m}$.
The product matrix rules you stated always hold:
$$
\begin{align}
%
  \mathbf{A} \mathbf{A}^{*} &=
%
  \left( 
    \mathbf{U} \, \mathbf{\Sigma} \, \mathbf{V}^{*}
  \right)
%
  \left( 
    \mathbf{U} \, \mathbf{\Sigma} \, \mathbf{V}^{*}
  \right)^{*}
%
=
%
  \left( 
    \mathbf{U} \, \mathbf{\Sigma} \, \mathbf{V}^{*}
  \right)
%
  \left( 
    \mathbf{V} \, \mathbf{\Sigma}^{\mathrm{T}} \, \mathbf{V}^{*}
  \right) \\
%
  \mathbf{A}^{*} \mathbf{A} &=
%
  \left( 
    \mathbf{U} \, \mathbf{\Sigma} \, \mathbf{V}^{*}
  \right)^{*}
%
  \left( 
    \mathbf{U} \, \mathbf{\Sigma} \, \mathbf{V}^{*}
  \right)
%
=
%
  \left( 
    \mathbf{V} \, \mathbf{\Sigma}^{\mathrm{T}} \, \mathbf{V}^{*}
  \right)
%
  \left( 
    \mathbf{U} \, \mathbf{\Sigma} \, \mathbf{V}^{*}
  \right) \\
%
\end{align}
$$
Examples follow.
Square, full rank $m = n = \rho$
$$
  \mathbf{A} = 
  \mathbf{U} \, \Sigma \, \mathbf{V}^{*} =
%
\left[ \begin{array}{c}
  \color{blue}{\mathbf{U}_{\mathcal{R}\left(\mathbf{A}\right)}} 
\end{array} \right]
%
 \left[ \mathbf{S} \right]
%
\left[ \begin{array}{c}
  \color{blue}{\mathbf{V}_{\mathcal{R}\left(\mathbf{A}^{*}\right)}} 
\end{array} \right]^{*}
%
$$
The product matrices are
$$
\begin{align}
%
  \mathbf{A}^{*}\mathbf{A} &= \color{blue}{\mathbf{V}_{\mathcal{R}\left(\mathbf{A}^{*}\right)}}
\, \mathbf{S}^{2} \,
\color{blue}{\mathbf{V}_{\mathcal{R}\left(\mathbf{A}^{*}\right)}}^{*} \\
%
  \mathbf{A}\mathbf{A}^{*} &= \color{blue}{\mathbf{U}_{\mathcal{R}\left(\mathbf{A}^{*}\right)}}
\, \mathbf{S}^{2} \,
\color{blue}{\mathbf{U}_{\mathcal{R}\left(\mathbf{A}^{*}\right)}}^{*}
%
\end{align}
$$
Tall, full column rank $n = \rho$, $m \ge n$
$$
  \mathbf{A} = 
  \mathbf{U} \, \Sigma \, \mathbf{V}^{*} =
%
\left[ \begin{array}{cc}
  \color{blue}{\mathbf{U}_{\mathcal{R}\left(\mathbf{A}\right)}} &
  \color{red} {\mathbf{U}_{\mathcal{N}\left(\mathbf{A}^{*}\right)}}
\end{array} \right]
%
 \left[ \begin{array}{c}
   \mathbf{S} \\ \mathbf{0}
 \end{array} \right]
%
\left[ \begin{array}{cc}
  \color{blue}{\mathbf{V}_{\mathcal{R}\left(\mathbf{A}^{*}\right)}} &
  \color{red} {\mathbf{V}_{\mathcal{N}\left(\mathbf{A}\right)}}
\end{array} \right]^{*}
%
$$
The product matrices are
$$
\begin{align}
%
  \mathbf{A}^{*}\mathbf{A} &= \color{blue}{\mathbf{V}_{\mathcal{R}\left(\mathbf{A}^{*}\right)}}
\, \mathbf{S}^{2} \,
\color{blue}{\mathbf{V}_{\mathcal{R}\left(\mathbf{A}^{*}\right)}}^{*} \\
%
  \mathbf{A}\mathbf{A}^{*} &= 
%
\left[ \begin{array}{cc}
  \color{blue}{\mathbf{U}_{\mathcal{R}\left(\mathbf{A}\right)}} &
  \color{red} {\mathbf{U}_{\mathcal{N}\left(\mathbf{A}^{*}\right)}}
\end{array} \right]
%
 \left[ \begin{array}{cc}
   \mathbf{S}^{2} & \mathbf{0} \\ \mathbf{0} & \mathbf{0}
 \end{array} \right]
%
\left[ \begin{array}{cc}
  \color{blue}{\mathbf{U}_{\mathcal{R}\left(\mathbf{A}\right)}} &
  \color{red} {\mathbf{U}_{\mathcal{N}\left(\mathbf{A}^{*}\right)}}
\end{array} \right]^{*}
%
\end{align}
$$
Wide, full row rank $m = \rho$, $n \ge m$
$$
  \mathbf{A} = 
  \mathbf{U} \, \Sigma \, \mathbf{V}^{*} =
%
\left[ \begin{array}{c}
  \color{blue}{\mathbf{U}_{\mathcal{R}\left(\mathbf{A}\right)}} 
\end{array} \right]
%
 \left[ \begin{array}{cc}
   \mathbf{S} & \mathbf{0}
 \end{array} \right]
%
\left[ \begin{array}{cc}
  \color{blue}{\mathbf{V}_{\mathcal{R}\left(\mathbf{A}^{*}\right)}} &
  \color{red} {\mathbf{V}_{\mathcal{N}\left(\mathbf{A}\right)}}
\end{array} \right]^{*}
%
$$
The product matrices are
$$
\begin{align}
%
  \mathbf{A}^{*}\mathbf{A} &= 
%
\left[ \begin{array}{cc}
  \color{blue}{\mathbf{V}_{\mathcal{R}\left(\mathbf{A}^{*}\right)}} &
  \color{red} {\mathbf{V}_{\mathcal{N}\left(\mathbf{A}\right)}}
\end{array} \right]
%
 \left[ \begin{array}{cc}
   \mathbf{S}^{2} & \mathbf{0} \\ \mathbf{0} & \mathbf{0}
 \end{array} \right]
%
\left[ \begin{array}{cc}
  \color{blue}{\mathbf{V}_{\mathcal{R}\left(\mathbf{A}^{*}\right)}} &
  \color{red} {\mathbf{V}_{\mathcal{N}\left(\mathbf{A}\right)}}
\end{array} \right]^{*} \\
%
  \mathbf{A}\mathbf{A}^{*} &= 
%
\color{blue}{\mathbf{U}_{\mathcal{R}\left(\mathbf{A}\right)}} \,
\, \mathbf{S}^{2} \,
\color{blue}{\mathbf{U}_{\mathcal{R}\left(\mathbf{A}\right)}} \\
%
\end{align}
$$
For the hermitian matrix,
$$
\begin{align}
  \mathbf{A} &= \mathbf{A}^{*} \\
  \mathbf{U} \, \Sigma \, \mathbf{V}^{*} &=
  \mathbf{V} \, \Sigma \, \mathbf{U}^{*}
\end{align}
$$
because in this case $\Sigma = \Sigma^{\mathrm{T}}$.
</s>