<s>[INST] coefficient of $x^{17}$ in $(x^2+x^3+x^4+x^5+x^6+x^7)^3$ coefficient of $x^{17}$ in $(x^2+x^3+x^4+x^5+x^6+x^7)^3$ - I'm reviewing for an exam and don't understand the answer, C(11+3−1,11)−C(3,1)×C(5+3−1,5).
 [/INST] 
Using the coefficient of operator $[x^n]$ to denote the coefficient of $x^n$ we obtain
\begin{align*}
\color{blue}{[x^{17}](x^2+x^3+\cdots+x^7)^3}&=[x^{17}]x^6(1+x+\cdots+x^5)^3\tag{1}\\
&=[x^{11}]\left(\frac{1-x^6}{1-x}\right)^3\tag{2}\\
&=[x^{11}]\left(1-\binom{3}{1}x^6\right)\sum_{n=0}^\infty \binom{n+3-1}{n}x^n\tag{3}\\
&=\left([x^{11}]-\binom{3}{1}[x^{5}]\right)\sum_{n=0}^\infty \binom{n+3-1}{n}x^n\\
&\color{blue}{=\binom{11+3-1}{11}-\binom{3}{1}\binom{5+3-1}{5}}\tag{4}\\
&=\binom{13}{2}-3\binom{7}{2}\\
&=15
\end{align*}

Comment:


*

*In (1) we factor out $x^6$.

*In (2) we apply the rule $[x^p]x^qA(x)=[x^{p-q}]A(x)$ and use the finite geometric series formula.

*In (3) we expand $(1-x^6)^3$ up to $x^6$ since other terms do no contribute to $x^{11}$.

*In (4) we select the coefficient of $x^{11}$ and $x^5$.
</s>