<s>[INST] How to solve similar with Transformation of a quadratic form into diagonal form? Define $\color{red}{f=f(x),f'=f'(x)}$,where the derivative with respect to $x$ of a function $f(x)$ is denoted $ f'(x)$.
Now give six postive numbers $k_{1},k_{2},k_{3},k_{4},k_{5},k_{6}$, and a quadratic form $$\color{red}{k_{1}f^2_{1}+k_{2}f^2_{2}+k_{3}f_{1}f_{2}+k_{4}f'^2_{1}+k_{5}f'^2_{2}+k_{6}f'_{1}f'_{2}}\tag{1}$$ How to Transformation of a quadratic form into diagonal form: $$A_{1}f^2_{1}+A_{2}f^2_{2}+A_{3}f'^2_{1}+A_{4}f'^2_{2}$$
For example :such
$$f^2_{1}+\dfrac{1}{2}f^2_{2}+\dfrac{1}{2}f'^2_{1}+\dfrac{3}{8}f'^2_{2}+\dfrac{1}{2}f'_{1}f'_{2}\tag{2}$$
because we have
$$f^2_{1}+\dfrac{1}{2}f^2_{2}+\dfrac{1}{2}f'^2_{1}+\dfrac{3}{8}f'^2_{2}+\dfrac{1}{2}f'_{1}f'_{2}=\dfrac{1}{3}(f_{1}+f_{2})^2+\dfrac{1}{3}(f'_{1}+f'_{2})^2+\dfrac{1}{6}(2f_{1}-f_{2})^2+\dfrac{1}{24}(2f'_{1}-f'_{2})^2\tag{3}$$
we Transformation 
$$ f_{1}+f_{2}=g_{1}, 2f_{1}-f_{2}=g_{2}$$
then $(3)$ diagonal form 
$$\dfrac{1}{3}g^2_{1}+\dfrac{1}{3}g'^2_{1}+\dfrac{1}{6}g^2_{2}+\dfrac{1}{24}g'^2_{2}$$
Unfortunately I don't know any idea  expression for this $ (1)$.  which could help. I'll be grateful for all useful suggestions.
 [/INST] You don't use a similarity transformation, the relationship for quadratic forms is called congruence. In matrices, you are solving $P^T M P = D$ diagonal, with $\det P = 1.$ Although it would be possible to force $P$ orthogonal in this instance, there is little advantage. 
Very hard to read
$$\color{red}{k_{1}f^2_{1}+k_{2}f^2_{2}+k_{3}f_{1}f_{2}+k_{4}f'^2_{1}+k_{5}f'^2_{2}+k_{6}f'_{1}f'_{2}}\tag{1}$$
$$\color{blue}{k_{1}u^2+k_{2}v^2+k_{3}uv+k_{4}x^2+k_{5}y^2+k_{6}xy }$$
All you need to do is separately complete the squares for the two binary forms.
$$ 4 k_1 (k_{1}u^2+k_{2}v^2+k_{3}uv) = 4 k_1^2 u^2 + 4 k_1 k_3 uv + 4 k_1 k_2 v^2. $$
Compare
$$ (2k_1u + k_3v)^2 = 4 k_1^2 u^2 + 4 k_1 k_3 uv +  k_3^2 v^2. $$ 
$$ 4 k_1 (k_{1}u^2+k_{2}v^2+k_{3}uv) = (2k_1u + k_3v)^2 + (4 k_1 k_2 - k_3^2)v^2 $$
$$  k_{1}u^2+k_{2}v^2+k_{3}uv = \left(\frac{1}{ 4 k_1} \right) (2k_1u + k_3v)^2 +   \left(\frac{4 k_1 k_2 - k_3^2}{ 4 k_1} \right)  v^2 $$
$$  k_{4}x^2+k_{5}y^2+k_{6}xy = \left(\frac{1}{ 4 k_4} \right) (2k_4x + k_6y)^2 +   \left(\frac{4 k_4 k_2 - k_6^2}{ 4 k_4} \right)  y^2 $$
</s>