<s>[INST] How to factor $x^6-4x^4+2x^3+1$ by hand? I generated this polynomial after playing around with the golden ratio. I first observed that (using various properties of $\phi$), $\phi^3+\phi^{-3}=4\phi-2$. This equation has no significance at all, I just mention it because the whole problem stems from me wondering: which other numbers does this equation hold for?
The six possible answers are the roots of $x^6-4x^4+2x^3+1=0$. Note that I am not interested in solving for $x$ itself as much as I am interested in a method which would allow me to completely factor out this polynomial into lowest degree factors which still have real coefficients. Note that I am treating this equation as if I had no clue that the golden ratio is one of the solutions. In other words, I am trying to factor this equation as if I never saw it before, so I can't just immediately factor out $(x^2-x-1)$ without a justifiable process, even though it is indeed one of the factors.
I first observed that the equation holds for $x=1$, so I was able to divide out $(x-1)$ to get the factorization of:
$$(x-1)(x^5+x^4-3x^3-x^2-x-1)$$
I tried making an assumption that the quintic reduces to a product of $(x^3+Ax^2+Bx+C)(x^2+Dx+E)$, multiplying out, and equalling coefficients, but I ended up with a system of two extremely convoluted equations which I had no idea how to solve. I also tried to turn the first five terms of the quintic into a palindromic polynomial and then perform the standard method of factoring palindromic polynomials, to no avail.
I am either missing something, or I don't know of a nice method that would let this expression be factored. I'm looking forward to being enlightened, thanks for any help.
 [/INST] Your original method is tedious but it can be done.
You can show that $(x^3+Ax^2+Bx+C)(x^2+Dx+E)$ is equal to:
$$x^5+(D+A)x^4+(1+AD+B)x^3 + (AE+BD+C)x^2 + (BE+CD) + CE$$
so $A+D = 1, B+AD+1 = -3, AE+BD+C=-1, BE+CD=-1, CE=-1$.
Assuming $A,B,C,D,E$ are all integers, we either have $C=-1, E=1$ or $C=1, E=-1$.
If $C=-1, E=1$, then we have:
$$A+D=1 \tag{1}$$
$$B+AD=-4 \tag{2}$$
$$A+BD=0 \tag{3}$$
$$B-D=-1 \tag{4}$$
$(1)+(4)$ gives $A+B=0$ so $A=-B$, which gives:
$$-B+D=1 \tag{5}$$
$$B-BD=-4 \tag{6}$$
$$-B+BD=0 \tag{7}$$
$$B-D=-1 \tag{8}$$
and this is clearly impossible since $(6) + (7)$ gives $0=-4$.
Therefore we must have $C=1, E=-1$:
$$A+D=1 \tag{9}$$
$$B+AD=-4 \tag{10}$$
$$-A+BD=-2 \tag{11}$$
$$-B+D=-1 \tag{12}$$
This time $(9)-(12)$ gives $A+B=2$, so $A=2-B$:
$$-B+D=-1 \tag{13}$$
$$B+2D-BD=-4 \tag{14}$$
$$B+BD=0 \tag{15}$$
$$-B+D=-1 \tag{16}$$
$(14)+(15)$ gives $2B+2D = -4$, so $B+D=-2$. When we add this to $(16)$, $2D=-2$ so $D=-1$.
And the rest follows:
$$B - D = 1 \Rightarrow B+1=1, B=0$$
$$A=2-B \Rightarrow A=2$$
so the factorisation is $(x-1)(x^3+2x^2+1)(x^2-x-1)$.
I wouldn't wish this method on anybody.
</s>