<s>[INST] How to transform quadratic terms to something like $u^2+v^2$? I am told that by the substitution of $x = (a\cos \theta)u - (b\sin \theta) v$ and $y = (a\sin \theta)u + (b\cos \theta)v$, we can reduce $3x^2 + 2xy + 3y^2$ to $u^2 + v^2$.
I tried to plug in the substitutions, but I did not see how the quadratic term is reduce to $u^2 + v^2$. What I got was something complicated containing $a$, $b$, $\cos \theta$ and $\sin \theta$.
I am asked to reduce $3x^2 + 2xy + 3y^2 - x - 2y$ to $u^2 + v^2$ by similar substitutions. I do not know how to construct this substitution. Could you explain how do find this kind of substitutions in general?

 [/INST] I think you are using a substitution more complicated than necessary. You also asked as to how such substitutions work in the general case, so I'll address that in this answer.
Consider the general bivariate quadratic polynomial,
$$f(x,y) = ax^2 + 2hxy + by^2 + 2gx + 2fy + c$$
There's a really neat way of writing this using matrices,
$$
f(x,y) = 
\begin{pmatrix}x & y & 1\end{pmatrix}
\begin{pmatrix} a & h & g \\ h & b & f \\ g & f & c\end{pmatrix} 
\begin{pmatrix} x \\ y \\ 1\end{pmatrix} $$
Or like this,
$$
f(x,y) = 
\begin{pmatrix}x & y\end{pmatrix}
\begin{pmatrix}a & h\\ h & b\end{pmatrix}
\begin{pmatrix}x \\ y\end{pmatrix} + 
2\begin{pmatrix}g & f\end{pmatrix}
\begin{pmatrix}x \\ y\end{pmatrix}
+c
$$
which is even better as now the terms of degree $0, 1, 2$ are all separate. What we want to do given such a quadratic polynomial, is substitute the variables $(u, v)$ which will make the cross term $xy$ or $uv$ vanish. Note that we cannot always eliminate both the cross and linear terms.
Such a substitution is essentially a rotation of the axes of the plane from $x, y$ to $u, v$. So our problem reduces to finding the appropriate angle by which the axes should be rotated.
Consider the rotation matrix,
$$
R(\theta) = 
\begin{pmatrix}
\cos \theta & \sin \theta\\
-\sin \theta & \cos \theta
\end{pmatrix}
$$
$R(\theta)$ is an orthonormal matrix with determinant 1, and hence, its inverse is simply its transpose. This can also be seen as the inverse operator must be a rotation by $-\theta$. So we get,
$$
R^{-1}(\theta) = R^{T}(\theta) = R(-\theta) = 
\begin{pmatrix}
\cos \theta & -\sin \theta\\
\sin \theta & \cos \theta
\end{pmatrix}
$$
Now do the substitution
$$
\begin{pmatrix}u \\ v\end{pmatrix} = 
R(\alpha)\begin{pmatrix}x \\ y\end{pmatrix} 
$$
where $\alpha$ is the appropriate angle.
We get,
$$
f(u,v) = 
\begin{pmatrix}u & v\end{pmatrix}
\underbrace{
R(\alpha)
\begin{pmatrix}a & h\\ h & b\end{pmatrix}
R^{-1}(\alpha)}_{M(\text{Let})}
\begin{pmatrix}u \\ v\end{pmatrix}
+ 
2\begin{pmatrix}g & f\end{pmatrix}
R^{-1}(\alpha)
\begin{pmatrix}u \\ v\end{pmatrix}
+c
$$
Doing the calculations, we find that,
$$
M = 
\begin{pmatrix}
a\cos^2 \alpha + b\sin^2 \alpha + h\sin 2\alpha & h\cos 2\alpha - \dfrac{a-b}{2}\sin 2\alpha\\
h\cos 2\alpha - \dfrac{a-b}{2}\sin 2\alpha & a\sin^2 \alpha + b\cos^2 \alpha - h\sin 2\alpha
\end{pmatrix}
$$
where I've used the double angle identities $\cos 2t = \cos^2 t - \sin^2 t$ and $\sin 2t = 2\sin t \cos t$.
Therefore, to make the cross term vanish, the angle $\alpha$ must satisfy
$$h\cos 2\alpha - \dfrac{a-b}{2}\sin 2\alpha = 0$$
There are two cases:

*

*$\alpha = \dfrac{\pi}{4}$ when $a = b$


*$\tan 2\alpha = \dfrac{2h}{a-b}$
In your particular case, $a = b = 3$, so $\alpha = \dfrac{\pi}{4}$. The substitution is just $$(x, y) = \left(\dfrac{1}{\sqrt{2}}u - \dfrac{1}{\sqrt{2}}v, \dfrac{1}{\sqrt{2}}u + \dfrac{1}{\sqrt{2}}v \right)$$
</s>