<s>[INST] Coefficient of a generating function I have the following generating function, 
$$f(x)=(1+x+x^2+x^3)^4$$
I want to find the $[x^5]$ coefficient, to do so I wrote, 
$$[x^5]f(x)=[x^5](1+x+x^2+x^3)^4=[x^5]\left(\frac{1-x^4}{1-x}\right)^4=
[x^5](1-x^4)^4(1-x)^{-4}
$$
I am wondering how to proceed from here? 
 [/INST] 
We obtain
  \begin{align*}
\color{blue}{[x^5]}&\color{blue}{(1-x^4)^4(1-x)^{-4}}\\
&=[x^5]\sum_{j=0}^\infty\binom{-4}{j}(-x)^{j}(1-x^4)^4\tag{1}\\
&=\sum_{j=0}^5\binom{3+j}{3}[x^{5-j}](1-x^4)^4\tag{2}\\
&=\sum_{j=0}^5\binom{8-j}{3}[x^{j}]\sum_{k=0}^4\binom{4}{k}(-1)^kx^{4k}\tag{3}\\
&=\binom{8}{3}\binom{4}{0}-\binom{4}{3}\binom{4}{1}\tag{4}\\
&\,\,\color{blue}{=40}
\end{align*}

Comment:


*

*In (1) we apply the binomial series expansion to $(1-x)^{-4}$.

*In (2) we use the binomial identity $\binom{-p}{q}=\binom{p+q-1}{q}(-1)^q$ and apply the rule $[x^{p-q}]A(x)=[x^p]x^qA(x)$. We set the upper limit of the series to $5$ since other terms do not contribute to $[x^5]$.

*In (3) we change the order of summation $j\to 5-j$ and expand the binomial.

*In (4) we select the coefficients of $x^0$ and $x^4$. Other terms do not contribute.
</s>