<s>[INST] Find the coefficient of $x^n$ in the generating functions: $g(x) = \frac{x^3}{(1+x)^5 (1−x)^6}$ One of the problems in my Discrete Math course states that we need to find the coefficient of $x^n$ in generating function $g(x) = \frac{x^3} {(1+x)^5 (1−x)^6}$
I separated $g(x) = \frac{x^3} {(1+x)^5 (1−x)^6}$ into $x^3$ * $\frac{1}{(1+x)^5}$ * $\frac{1}{(1-x)^6}$
Then got $f(x) = x^3 * \sum_{k=0}^\infty \binom{n+4}{4}(-1)^kx^k * \sum_{k=0}^\infty \binom{n+5}{5}x^k$
I dont know what to do from here, can you tell me what is the next step?
 [/INST] $$
\begin{align}
\frac{x^3}{(1+x)^5 (1−x)^6}
&=\frac{x^3+x^4}{\left(1-x^2\right)^6}\tag1\\
&=\left(x^3+x^4\right)\sum_{k=0}^\infty(-1)^k\binom{-6}{k}x^{2k}\tag2\\
&=\left(x^3+x^4\right)\sum_{k=0}^\infty\binom{k+5}{5}x^{2k}\tag3\\
&=\sum_{k=0}^\infty\binom{k+5}{5}\left(x^{2k+3}+x^{2k+4}\right)\tag4\\
&=\sum_{k=2}^\infty\binom{k+3}{5}\left(x^{2k-1}+x^{2k}\right)\tag5
\end{align}
$$
Explanation:
$(1)$: multiply by $\frac{1+x}{1+x}$
$(2)$: use the series for $(1+x)^{-6}$
$(3)$: $\binom{-6}{k}=(-1)^k\binom{k+5}{5}$ when $k\ge0$
$\phantom{\text{(3):}}$ (negative binomial coefficients)
$(4)$: distribute the $x^3+x^4$
$(5)$: substitute $k\mapsto k-2$
Thus, if $n=2k-1$ or $n=2k$, then the coefficient of $x^n$ is $\binom{k+3}{5}$.
This can be stated as the coefficient of $x^n$ is $\binom{\left\lfloor\frac{n+7}2\right\rfloor}{5}$
</s>