<s>[INST] Compute the coefficient of $x^n$ in the given function Compute the coefficient of $x^n$ in 
$$(x+x^2+...+x^5)(x^2+x^3....+x^n+...)^3,$$
when $x<1$. So, I have tried this, but I am not sure how to continue where I left off.. So far I have done the following steps:
$$x(1 + x + x^2 + x^3 + x^4)\cdot x^6(1 + x + x^2+\cdots)^3$$
which equals
$$x^7\frac{1-x^5}{1-x}\frac{1}{(1-x)^3} = x^7\frac{1-x^5}{(1-x)^4}$$
However, I am stuck now as to how to get the coefficient of $x^n$. Can someone explain to me what I should be doing?
 [/INST] By Newton's binomial theorem, we know that $$\sum_{k=0}^\infty \binom{s+k-1}{k}x^k = \frac{1}{(1-x)^s}.$$
Using this we may rewrite $x^7\frac{1-x^5}{(1-x)^4}$ as
$$\frac{x^7}{(1-x)^4} - \frac{x^{12}}{(1-x)^4}=
\sum_{k=0}^\infty \binom{3+k}{k}x^{k+7} - \sum_{k=0}^\infty \binom{3+k}{k}x^{k+12}.$$
Since we want a power series in $x$, we must shift our indices accordingly so that our two sums can share the same power of $x$.
$$
\frac{x^7}{(1-x)^4} - \frac{x^{12}}{(1-x)^4} 
= \sum_{k=7}^\infty \binom{k-4}{k-7}x^{k} - \sum_{k=12}^\infty \binom{k-9}{k-12}x^{k}$$
Now we take out a couple terms from the first sum so that we end up with only one infinite sum.
$$\frac{x^7}{(1-x)^4} - \frac{x^{12}}{(1-x)^4} = \sum_{k=7}^{11} \binom{k-4}{k-7}x^k 
+ \sum_{k=12}^\infty \left[ \binom{k-4}{k-7} - \binom{k-9}{k-12} \right]x^k. $$
So the coefficient of $x^n$ in $x^7\dfrac{1-x^5}{(1-x)^4}$ is
$$
\begin{align}
\binom{n-4}{n-7} &\text{ for } 7 \leq n \leq 11
\\\binom{n-4}{n-7} - \binom{n-9}{n-12} &\text{ for } n \geq 12.
\end{align} $$
</s>