<s>[INST] Factor $3x^2-11xy+6y^2-xz-4yz-2z^2$ This problem is from my Math Challenge II Algebra class, and it's really confusing. How can you factor something like this? Here's the question again: Factor $3x^2-11xy+6y^2-xz-4yz-2z^2$.
 [/INST] First, split up the expression into pieces.
\begin{align*}
F(x,y,z) &= 3x^2-11xy+6y^2-xz-4yz-2z^2 \\
&= 3x^2\underbrace{-11xy-xz}_{G(x,y,z)}+\underbrace{6y^2-4yz-2z^2}_{H(y,z)}
\end{align*}
Start with the easy one : $H(y,z)$.
\begin{align*}
H(y,z) &= 6y^2-4yz-2z^2 \\
&= 6y^2-6yz+2yz-2z^2 \\
&= 6y(y-z)+2z(y-z) \\
&= 2(y-z)(3y+z)
\end{align*}
Now, the tuff one : $G(x,y,z)$. Remember, we want to find some factors like those in $H(x,y)$. So, we have to use our imagination...
\begin{align*}
G(x,y,z) &= -11xy-xz \\
&= -3x(3y+z)-2xy+2xz \\
&= -3x(3y+z)-2x(y-z) \\
\end{align*}
Finish, with all the pieces together.
\begin{align*}
F(x,y,z) &= 3x^2 + G(x,y,z) + H(y,z) \\
&= 3x^2-3x(3y+z)-2x(y-z)+2(y-z)(3y+z) \\
&= (3y+z)[2(y-z)-3x]-x[2(y-z)-3x] \\
&= [2(y-z)-3x][(3y+z)-x] \\
&= (3x-2y+2z)(x-3y-z)
\end{align*}
</s>