<s>[INST] How to solve quadratic matrix equations of the form $A^T B A=C$? I want to solve the following matrix equation
$$A^T  
\begin{pmatrix}
 10 & 0 & 0 \\
 0 & 20 & 0 \\
 0 & 0 & 25 \\
\end{pmatrix} A =
\begin{pmatrix}
 \frac{35}{2} & \frac{5 \sqrt{3}}{2} & 0 \\
 \frac{5 \sqrt{3}}{2} & \frac{25}{2} & 0 \\
 0 & 0 & 25 \\
\end{pmatrix}$$
In the above formula, $A^T$ is the transpose of matrix A.
At present, I don't have a good way. I only know that the reference answer of matrix A is 
$$\left(\begin{array}{ccc}
 \frac{1}{2} & -\frac{\sqrt{3}}{2} & 0 \\
 \frac{\sqrt{3}}{2} & \frac{1}{2} & 0 \\
 0 & 0 & 1 \end{array}\right)$$
 [/INST] In our case we have two given symmetric matrices $B,C$,
$$
\begin{aligned}
B 
&= 
\begin{bmatrix}
10 &&\\&20&\\&&25
\end{bmatrix}=B_1^2 =B_1^TB_1=B_1B_1^T\ ,\text{ where }
\\
B_1
&= 
\begin{bmatrix}
\sqrt{10} &&\\&2\sqrt 5&\\&&5
\end{bmatrix}=B_1^\ ,
\\[3mm]
C &=
\begin{bmatrix}
 \frac{35}{2} & \frac{5 \sqrt{3}}{2} & 0 \\
 \frac{5 \sqrt{3}}{2} & \frac{25}{2} & 0 \\
 0 & 0 & 25 
\end{bmatrix}=C_1^2=C_1^TC_1=C_1C_1^T\ ,\text{ where }
\\
C_1
&= 
\begin{bmatrix}
a & b&\\b&c&\\&&5
\end{bmatrix}=C_1^T\ ,\text{ where its entries $a,b,c$ are}
\\
a &= \frac{\sqrt{10}}4(3\sqrt 2+1)\ ,\\
b &= \frac{\sqrt{30}}4(\sqrt 2-1)\ ,\\
c &= \frac{\sqrt{10}}4(\sqrt 2+3)\ .
\end{aligned}
$$
We  have to find one or all "unknown" matrices $A$, so that the matrix equation holds:
$$A^TBA=C\ .
$$
Equivalently, after successive transformations:
$$
\begin{aligned}
A^TBA &=C\ ,\\
A^TB_1^TB_1A &=C_1^TC_1\ ,\\
(C_1^{-1})^TA^TB_1^TB_1A(C_1^{-1}) &=I\ ,\\
(B_1AC_1^{-1})^T\;(B_1AC_1^{-1}) &=I\ ,\\
B_1AC_1^{-1}&\in SO(3)\ .
\end{aligned}
$$
Here, $SO(3)$ is the special orthogonal group of all matrices $X$ (with real entries, well, i suppose that that "complex" in the title is subjectively a synonym for "difficult"), such that $X^TX=I=XX^T=$. For each such $X$ we get a solution $A$ by isolating $A$ in $B_1AC_1^{-1}=X$. (So there are infinitely many solutions.) To have one solution, take the obvious $X=I$ as a particular solution.

Note: The diagonalization (if possible or not) is a process to put a linear transformation in a suitable form by using a base change. In our case, it is a quadratic form to be transformed by a base change. This is different.
https://en.wikipedia.org/wiki/Quadratic_form#Equivalence_of_forms
Note that the first object, the linear map given by a matrix $A$, transforms via a base change by $S$ as 
$$A\to S^{-1}AS\ ,$$
the second object, the quadratic form given by a symmetric $A$, transforms as $$A\to S^TAS\ .$$
</s>