<s>[INST] How to prove this equality of the determinant of matrix? 
Prove that
\begin{equation*}
\det\begin{pmatrix}
a^2 & b^2 & c^2 \\
ab & bc & ca \\
b^2 & c^2 & a^2
\end{pmatrix}
=(a^2-bc)(b^2-ca)(c^2-ab)\end{equation*}


My attempt:
\begin{equation*}
\det\begin{pmatrix}
a^2 & b^2 & c^2 \\
ab & bc & ca \\
b^2 & c^2 & a^2
\end{pmatrix}
=\det\begin{pmatrix}
a^2 & b^2 & c^2 \\
a(b-a) & b(c-b) & c(a-c) \\
(b+a)(b-a) & (c+b)(c-b) & (a+c)(a-c)
\end{pmatrix}
\end{equation*}
But I think my direction is incorrect. Can anyone give me some hints or the solution of this question?
 [/INST] My steps to calculate your determinant. With $R$ I indicate the row and with $C$ the column.
$$C_2=C_2-\left(\frac{b^{2}}{a^{2}}\right)C_1 $$
$$\left| \begin{array}{ccc} a^{2} & 0 & c^{2} \\\\ a b & b c - \frac{b^{3}}{a} & a c \\\\ b^{2} & c^{2} - \frac{b^{4}}{a^{2}} & a^{2} \end{array} \right| \tag{first step}$$
Subtract column $1$ multiplied by $c^2/a^2$ from column $3$; you will have
$$\left| \begin{array}{ccc} a^{2} & 0 & 0 \\\\ a b & b c - \frac{b^{3}}{a} & a c - \frac{b c^{2}}{a} \\\\ b^{2} & c^{2} - \frac{b^{4}}{a^{2}} & \frac{a^{4} - b^{2} c^{2}}{a^{2}} \end{array} \right| \tag{second step}$$
Using the first rule of Laplace:
$$\left| \begin{array}{ccc} a^{2} & 0 & 0 \\\\ a b & b c - \frac{b^{3}}{a} & a c - \frac{b c^{2}}{a} \\\\ b^{2} & c^{2} - \frac{b^{4}}{a^{2}} & \frac{a^{4} - b^{2} c^{2}}{a^{2}} \end{array} \right|=\left(a^{2}\right) \cdot(-1)^{1+1}\cdot \left| \begin{array}{cc} b c - \frac{b^{3}}{a} & a c - \frac{b c^{2}}{a} \\\\ c^{2} - \frac{b^{4}}{a^{2}} & \frac{a^{4} - b^{2} c^{2}}{a^{2}} \end{array} \right|+0 \cdot(-1)^{1+2}\cdot \left| \begin{array}{cc} a b & a c - \frac{b c^{2}}{a} \\\\ b^{2} & \frac{a^{4} - b^{2} c^{2}}{a^{2}} \end{array} \right|+0 \cdot(-1)^{1+3}\cdot \left| \begin{array}{cc} a b & b c - \frac{b^{3}}{a} \\\\ b^{2} & c^{2} - \frac{b^{4}}{a^{2}} \end{array} \right|=a^{2} \left| \begin{array}{cc} b c - \frac{b^{3}}{a} & a c - \frac{b c^{2}}{a} \\\\ c^{2} - \frac{b^{4}}{a^{2}} & \frac{a^{4} - b^{2} c^{2}}{a^{2}} \end{array} \right|$$
Considering the last determinant you have:
$$\left| \begin{array}{cc} b c - \frac{b^{3}}{a} & a c - \frac{b c^{2}}{a} \\\\ c^{2} - \frac{b^{4}}{a^{2}} & \frac{a^{4} - b^{2} c^{2}}{a^{2}} \end{array} \right|=\left(b c - \frac{b^{3}}{a}\right)\cdot\left(\frac{a^{4} - b^{2} c^{2}}{a^{2}}\right)-\left(a c - \frac{b c^{2}}{a}\right)\cdot\left(c^{2} - \frac{b^{4}}{a^{2}}\right)=\frac{\left(a^{2} - b c\right) \left(a b - c^{2}\right) \left(a c - b^{2}\right)}{a^{2}}$$
Hence:
$$\left| \begin{array}{ccc} a^{2} & b^{2} & c^{2} \\\\ a b & b c & a c \\\\ b^{2} & c^{2} & a^{2} \end{array} \right|=a^{2} \cdot \left(\frac{\left(a^{2} - b c\right) \left(a b - c^{2}\right) \left(a c - b^{2}\right)}{a^{2}}\right)=(a^2-bc)(b^2-ca)(c^2-ab)$$
</s>