<s>[INST] Finding $P$ such that $P^TAP$ is a diagonal matrix 
Let $$A = \left(\begin{array}{cc} 2&3 \\ 3&4  \end{array}\right) \in
M_2(\mathbb{C})$$
Find $P$ such that $P^TAP = D$ where $D$ is a diagonal matrix.

So here's the solution:
$$A = \left(\begin{array}{cc|cc} 2&3&1&0\\ 3&4&0&1    \end{array}\right) \sim \left(\begin{array}{cc|cc} 2&0&1&-3/2\\ 0&-1/2&0&1    \end{array}\right)$$
Therefore, $$P = \left(\begin{array}{cc} 1&-3/2\\ 0&1    \end{array}\right) \\ P^TAP = \left(\begin{array}{cc} 2&0\\ 0&-1/2    \end{array}\right) $$
What was done here exactly? I'd be glad elaborate about the process.
Thanks.
 [/INST] Hermite Reduction.
SEE ALSO Orthogonal basis for this indefinite symmetric bilinear form
Transforming quadratic forms, how is this theorem called?
What is the difference between using $PAP^{-1}$ and $PAP^{T}$ to diagonalize a matrix?
When you have a symmetric matrix of integers, you may use Hermite's method for diagonalizing, the order they want is $P^t A P = D.$ Alright, I will need to do an inverse at the end.
Make a column vector 
$$
V =
\left(
\begin{array}{c}
x \\
y
\end{array}
\right)
$$ 
and write out
$$  V^T A V = 2 x^2 + 6 xy + 4 y^2  $$
Next, we cancel out all $x$ terms using 
$$ \left( x + \frac{3}{2} y \right)^2 = x^2 + 3 xy  + \frac{9}{4} y^2, $$ and
$$ 2 \left( x + \frac{3}{2} y \right)^2 = 2x^2 + 6 xy  + \frac{9}{2} y^2. $$
As a result,
$$ 2 \left( x + \frac{3}{2} y \right)^2 -   \frac{1}{2} y^2 =  2 x^2 + 6 xy + 4 y^2 . $$
MORE TYPING TO COME !!!!
In matrices, the direction I did is
$$
\left(
\begin{array}{cc}
1 & 0 \\
 \frac{3}{2} & 1
\end{array}
\right)
\left(
\begin{array}{cc}
2 & 0 \\
0 & -\frac{1}{2}
\end{array}
\right)
\left(
\begin{array}{cc}
1 & \frac{3}{2} \\
0 & 1
\end{array}
\right) =
\left(
\begin{array}{cc}
2 & 3 \\
3 & 4
\end{array}
\right)
$$
With
$$ Q =
\left(
\begin{array}{cc}
1 & \frac{3}{2} \\
0 & 1
\end{array}
\right)
$$
notice that the rows correspond exactly to the linear substitutions, the first row means $x + \frac{3}{2} y$ and the second row means $y.$
EVEN MORE EXCITING TYPING ANY MINUTE !!!!!!!!!!
What I did so far is in the order $Q^T D Q = A.$ All we need to do is take $p= Q^{-1},$ which is easier than usual because $\det Q = 1.$ The result is
$$
\left(
\begin{array}{cc}
1 & 0 \\
 -\frac{3}{2} & 1
\end{array}
\right)
\left(
\begin{array}{cc}
2 & 3 \\
3 & 4
\end{array}
\right)
\left(
\begin{array}{cc}
1 & -\frac{3}{2} \\
0 & 1
\end{array}
\right) =
\left(
\begin{array}{cc}
2 & 0 \\
0 &  -\frac{1}{2}
\end{array}
\right)
$$
The second example in the question, with 3 by 3 matrix, is
$$ x^2 + 4 y^2 + 4 z^2 + 16 yz + 4 zx + 4 xy.  $$ This is an example where an extra trick must be used:
$$ (x+2y+2z)^2 =  x^2 + 4 y^2 + 4 z^2 + 8 yz + 4 zx + 4 xy.   $$
All that remains to construct is $8yz$ because we used up the $y^2$ and $z^2.$ The trick is that $(y+z)^2 - (y-z)^2 = 4yz,$ so
$$ (x+2y+2z)^2 + 2 (y+z)^2 -2 (y-z)^2=  x^2 + 4 y^2 + 4 z^2 + 16 yz + 4 zx + 4 xy.   $$
Thus the diagonal matrix gets entries $1,2,-2$ and, in this direction,
$$
Q = 
\left( 
\begin{array}{ccc}
1 & 2 & 2 \\
0 & 1 & 1 \\
0 & 1 & -1
\end{array}
\right)
$$
and then $P = Q^{-1}$
$$
P = 
\left( 
\begin{array}{ccc}
1 & -2 & 0 \\
0 & \frac{1}{2} &  \frac{1}{2} \\
0 &  \frac{1}{2} & - \frac{1}{2}
\end{array}
\right)
$$
</s>