<s>[INST] Simultaneous Diagonalization of two bilinear forms I need to diagonalize this two bilinear forms in the same basis (such that $f=I$ and $g$=diagonal matrix):
$f(x,y,z)=x^2+y^2+z^2+xy-yz $
$g(x,y,z)=y^2-4xy+8xz+4yz$
I know that it is possible because f is positive-definite, but I don't know how can I do it
 [/INST] As I commented, a more difficult version is at Congruence and diagonalizations
This one is easier, the matrix $C$ has three distinct eigenvalues that are integers.
$$
A = 
\left(
\begin{array}{rrr}
1 & \frac{1}{2} & 0 \\
\frac{1}{2} & 1 & -\frac{1}{2} \\
0 & -\frac{1}{2}  & 1
\end{array}
\right)
$$
$$
B = 
\left(
\begin{array}{rrr}
0 & -2 & 4 \\
-2 & 1 & 2 \\
4 & 2  & 0
\end{array}
\right)
$$
$$
A^{-1} = 
\left(
\begin{array}{rrr}
 \frac{3}{2} & -1 &  -\frac{1}{2} \\
-1 & 2 & 1 \\
 -\frac{1}{2}  & 1 &  \frac{3}{2}
\end{array}
\right)
$$
$$
C = A^{-1}B = 
\left(
\begin{array}{rrr}
0 & -5 & 4 \\
0 & 6 & 0 \\
4 & 5  & 0
\end{array}
\right)
$$
The theorem from Horn and Johnson (first edition hardcover was 1985, paperback 1990) is that we can continue if and only if $C$ is diagonalizable in that $R^{-1} C R =D$ is diagonal.
They were extremely careful: the eigenvalues of $C$ are $6,4,-4$ and we can make the matrix $R$ with columns as eigenvectors with
$$
R = 
\left(
\begin{array}{rrr}
1 & 1 & -1 \\
-2 & 0 & 0 \\
-1 & 1  & 1
\end{array}
\right).
$$
Confirm
$$
CR = 
\left(
\begin{array}{rrr}
6 & 4 & 4 \\
-12 & 0 & 0 \\
-6 & 4  & -4
\end{array}
\right).
$$
It follows that $R^{-1}CR$ is the diagonal matrix with entries $6,4,-4.$
$$
D = 
\left(
\begin{array}{rrr}
6 & 0 & 0 \\
0 & 4 & 0 \\
0 & 0  & -4
\end{array}
\right).
$$
Finally,
$$
R^TBR = 
\left(
\begin{array}{rrr}
12 & 0 & 0 \\
0 & 8 & 0 \\
0 & 0  & -8
\end{array}
\right),
$$
and
$$
R^TAR = 
\left(
\begin{array}{rrr}
2 & 0 & 0 \\
0 & 2 & 0 \\
0 & 0  & 2
\end{array}
\right).
$$
The reason this works is that we have arranged $R^T AR D = R^T BR.$
Let's see, this is in the first edition of Horn and Johnson, table 4.5.15T on page 229, then detail for case II on pages 231-232. The technique gives the full problem when $C$ has all eigenvalues distinct, as in this problem. With repeat eigenvalues, one needs to continue working, that is half of page 232. Indeed, when this happens, we are guaranteed to have square diagonal blocks in both the revised $A$ and $B$ matrices, such that each $B$ block is just a scalar multiple of each corresponding $A$ block. The extra work is then to diagonalize the $A$ block, the same thing will work on the $B$ block. That is exactly what is shown at Congruence and diagonalizations 
Oh, you wanted the identity matrix. Do another step with $R_2 = I/ \sqrt 2$ on right and left, the result is just to halve each diagonal matrix. The final overall matrix is my $R/\sqrt 2.$ 
Tuesday: to continue, take $Q = R/\sqrt 2.$ Then $Q^{-1} = R^{-1} \sqrt 2$
$$
Q^{-1} = R^{-1}  \sqrt 2 = \; \;
\left(
\begin{array}{rrr}
0 & \frac{-1}{\sqrt 2} & 0 \\
\frac{1}{\sqrt 2} & 0 & \frac{1}{\sqrt 2} \\
\frac{-1}{\sqrt 2} & \frac{-1}{\sqrt 2}  & \frac{1}{\sqrt 2}
\end{array}
\right).
$$
The rows of $Q^{-1}$ give us
$$ \left(\frac{-1}{\sqrt 2}y \right)^2 +  \left(\frac{1}{\sqrt 2} x + \frac{1}{\sqrt 2} z \right)^2 +  \left(\frac{-1}{\sqrt 2} x +\frac{-1}{\sqrt 2}y + \frac{1}{\sqrt 2} z \right)^2 = x^2 + y^2 + z^2 -yz +xy $$
$$ 6\left(\frac{-1}{\sqrt 2}y \right)^2 + 4 \left(\frac{1}{\sqrt 2} x + \frac{1}{\sqrt 2} z \right)^2 -4  \left(\frac{-1}{\sqrt 2} x +\frac{-1}{\sqrt 2}y + \frac{1}{\sqrt 2} z \right)^2 = y^2 + 4yz + 8zx - 4 xy $$
</s>