<s>[INST] Factor $x^4 - 11x^2y^2 + y^4$ This is an exercise from Schaum's Outline of Precalculus. It doesn't give a worked solution, just the answer. 
The question is:
Factor $x^4 - 11x^2y^2 + y^4$
The answer is:
$(x^2 - 3xy -y^2)(x^2 + 3xy - y^2)$
My question is:
How did the textbook get this?
I tried the following methods (examples of my working below):


*

*U-Substitution.

*Guess and Check.

*Reversing the question (multiplying the answer out).


Here is my working for each case so far.
(1) U-Substitution.
I tried a simpler case via u-substitution. 
Let $u = x^2$ and $v = y^2$. Then $x^4 - 11x^2y^2 + y^4 = u^2 -11uv + v^2$. Given the middle term is not even, then it doesn't factor into something resembling $(u + v)^2$ or $(u - v)^2$. Also, there doesn't appear to be any factors such that $ab = 1$ (the coefficient of the third term) and $a + b = -11$ (the coefficient of the second term) where $u^2 -11uv + v^2$ resembles $(u \pm a)(v \pm b)$ and the final answer.
(2) Guess and Check.
To obtain the first term in $x^4 - 11x^2y^2 + y^4$ it must be $x^2x^2 = x^4$. To obtain the third term would be $y^2y^2 = y^4$. So I'm left with something resembling:
$(x^2 \pm y^2)(x^2 \pm y^2)$. 
But I'm at a loss as to how to get the final solution's middle term from guessing and checking.
(3) Reversing the question
Here I took the answer, multiplied it out, to see if the reverse of the factoring process would illuminate how the answer was generated.
The original answer:
$(x^2 - 3xy -y^2)(x^2 + 3xy - y^2) = [(x^2 - 3xy) - y^2][(x^2 + 3xy) - y^2]$
$= (x^2 - 3xy)(x^2 + 3xy) + (x^2 - 3xy)(-y^2) + (x^2 + 3xy)(-y^2) + (-y^2)(-y^2)$
$= [(x^2)^2 - (3xy)^2] + [(-y^2)x^2 + 3y^3x] + [(-y^2)x^2 - 3y^3x] + [y^4]$
$= x^4 - 9x^2y^2 - y^2x^2 + 3y^3x - y^2x^2 - 3y^3x + y^4$
The $3y^3x$ terms cancel out, and we are left with:
$x^4 - 9x^2y^2 - 2x^2y^2 + y^4 = x^4 - 11x^2y^2 + y^4$, which is the original question.
The thing I don't understand about this reverse process is where the $3y^3x$ terms came from. Obviously $3y^3x - 3y^3x = 0$ by additive inverse, and $a + 0 = a$, but I'm wondering how you would know to add $3y^3x - 3y^3x$ to the original expression, and then make the further leap to factoring out like terms (by splitting the $11x^2y^2$ to $-9x^2y^2$, $-y^2x^2$, and $-y^2x^2$).
 [/INST] A start: Divide the original by $y^4$, and set $z=x/y$. We are trying to factor $z^4-11z^2+1$ as a product of two quadratics. Without loss of generality we may assume they both begin with $z^2$. So we are looking for a factorization of type $(z^2+az+b)(z^2+cz+d)$. By looking at the coefficient of $z^3$ in the product, we can see that $c=-a$, and you are on your way.
The next step is to see what $c=-a$ and the coefficient of $x$ in the product being $0$ tells us about the relationship between $b$ and $d$.
The idea works smoothly for, say $(11.3)x^2y^2$ replacing $11x^2y^2$, except of course the factorization involves square roots.
Another way: Divide the original polynomial by $x^2y^2$. We get
$$\frac{x^4-11x^2y^2+y^4}{x^2y^2}=\frac{x^2}{y^2}-11+\frac{y^2}{x^2}.\tag{1}$$
Make the substitution $w=\frac{x}{y}+\frac{y}{x}$. Then $\frac{x^2}{y^2}+\frac{y^2}{x^2}=w^2-2$. The right-hand side of Expression (1) becomes $w^2-9$, which factors as $(w-3)(w+3)$.  Thus (1) can be expressed as the product
$$\left(\frac{x}{y}+\frac{y}{x}-3\right)\left(\frac{x}{y}+\frac{y}{x}+3\right).\tag{2}$$
Finally, multiply Expression (2) by $x^2y^2$, by multiplying each term by $xy$, and we get our factorization. 
The idea generalizes immediately to $x^4-ax^2y^2+y^4$ where $a\ge 2$, and can be  pushed well beyond.
</s>