<s>[INST] Find $A, B \in M_2(\mathbb{Q})$ Find $A, B \in M_2(\mathbb {Q}) $ so that $A^2+2B^2=\begin{pmatrix} 4& - 4\\
-2 & 2
\end{pmatrix}$ and $AB+BA=\begin{pmatrix}3 & - 3\\
-1 & 1
\end{pmatrix}$.
My work so far : $(A +\sqrt 2 B) ^2=A^2+2B^2+\sqrt 2 (AB+BA) $.
After taking determinants we get that $\det (A +\sqrt 2 B)=0$ and this implies that $Tr A\cdot Tr B=Tr(AB) $and $\det A=-2\det B$. Here I am stuck.
Edit: We can similarly get that $\det(A-\sqrt 2 B) =0$ and this implies that $\det A=\det B=0$ and I think now we can easily find the traces and then just substitute back into the equations. 
 [/INST] If we denote $v=(1,1)$, then the given conditions imply that
$$
\ker (A+\sqrt 2B)^2 =\ker(A-\sqrt 2B)^2=\text{span}\{v\}.
$$ Thus $A\pm \sqrt{2}B$ are non-invertible matrices, which are not $O$, and it follows that
$$
 \ker(A\pm \sqrt{2}B)=\text{span}\{v\}
$$ since
$$
(0)< \ker(A\pm \sqrt{2}B) \le \ker (A\pm \sqrt 2B)^2=\text{span}\{v\}.
$$This implies that $Av=Bv=0$, so both $A,B$ are non-invertible.
Denote $\alpha =\text{tr}(A)$ and $\beta=\text{tr}(B)$. Then Caley-Hamilton theorem implies that
$$
A^2=\alpha A, \ \ B^2=\beta B
$$ thus giving $A^2+2B^2=\alpha A+2\beta B\ (*)$. Take trace to $(*)$, then we get $\alpha^2+2\beta^2=6$. By squaring $(*)$, we obtain
$$
\alpha^3A+4\beta^3B+2\alpha\beta(AB+BA)=\left(\begin{array}{cc}24& -24\\-12&12\end{array}\right)
$$ and by taking trace again, it follows
$$
\alpha^4+4\beta^4+8\alpha\beta=36=(\alpha^2+2\beta^2)^2=\alpha^4+4\beta^4+4\alpha^2\beta^2.
$$ Note that $\alpha\beta=0$ or $\alpha\beta=2$. But $\alpha\beta=0$ would imply $\alpha^2=6$ or $\beta^2=3$, which is absurd since $\alpha,\beta\in \Bbb Q$. So, $\alpha\beta$ is $2$, and $(\alpha+\sqrt 2\beta)^2=6+4\sqrt 2=(2+\sqrt 2)^2$ shows that $(\alpha,\beta)=\pm(2,1)$. Note that if $(A,B)$ is a solution for $(\alpha,\beta)=(2,1)$, then $(-A,-B)$ is a solution for $(\alpha,\beta)=(-2,-1)$.
Now, assume $\alpha=2,\beta =1$ and observe that
$$
A^2+2B^2=2(A+B)=\left(\begin{array}{cc}4& -4\\-2&2\end{array}\right).
$$ Squaring $A+B$ gives
$$
A^2+B^2+AB+BA=2A+B+\left(\begin{array}{cc}3& -3\\-1&1\end{array}\right)=\left(\begin{array}{cc}6& -6\\-3&3\end{array}\right).
$$ By solving the equations simultaneously we finally obtain the solution for $(\alpha,\beta)=(2,1)$:
$$
A=\left(\begin{array}{cc}1& -1\\-1&1\end{array}\right),\quad B=\left(\begin{array}{cc}1& -1\\0&0\end{array}\right).
$$ For the case where $(\alpha,\beta)=(-2,-1)$, put $-$ signs to the above $A,B$. These are the only solutions to the given equations.
</s>