<s>[INST] find the coefficient of $x^5$ in $(1+2x-3x^2)^6$ I know that I should first factor the expression within the brackets to $-(3x+1)^6(x-1)^6$. However, after that I do not know what to do.
 [/INST] We can also apply the binomial theorem twice in order to find the coeffcient. It is convenient to use the coefficient of operator $[x^j]$ to denote the coefficient of $x^j$ of an expression.

We obtain
\begin{align*}
\color{blue}{[x^5]}&\color{blue}{(1+2x-3x^2)^{6}}\\
&=[x^5]\sum_{j=0}^{6}\binom{6}{j}(-3x^2)^j(1+2x)^{6-j}\tag{1}\\
&=\sum_{j=0}^2\binom{6}{j}(-3)^j[x^{5-2j}]\sum_{k=0}^{6-j}\binom{6-j}{k}(2x)^k\tag{2}\\
&=\sum_{j=0}^2\binom{6}{j}(-3)^j\binom{6-j}{5-2j}2^{5-2j}\tag{3}\\
&=\binom{6}{0}(-3)^0\binom{6}{5}2^5+\binom{6}{1}(-3)^1\binom{5}{3}2^3+\binom{6}{2}(-3)^2\binom{4}{1}2^1\\
&=192-1\,440+1\,080\\
&\,\,\color{blue}{=-168}
\end{align*}

Comment:

*

*In (1) we apply the binomial theorem once to $((1+2x)-3x^2))^6$.


*In (2) we apply the rule $[x^p]x^qA(x)=[x^{p-q}]A(x)$ and we set the upper index of the outer sum to $2$, since other terms do not contribute. We also apply the binomial theorem once again.


*In (3) we select the coefficient of $x^{5-2j}$.
</s>