<s>[INST] Finding closed form expression for $n^{th}$ term of sequence with generating function $F(x)$? I have been asked the above question regarding the generating function
$$F(x) = \frac{x^2(1-x)}{(1-x)^3}$$
I have no idea what procedure this type of question follows. The solution gives that $F(x)$ can be written as
$$\frac{x^2}{(1-x)^3}-\frac{x^3}{(1-x)^3}$$
This I understand. But then it then says for $F(x) = \dfrac{1}{(1-x)^3}$ we get
$$\frac{F^n(0)}{n!} = \frac{(n+1)(n+2)}{2} \tag1$$
From which we can get the coefficients.
Can someone help me understand what is computed here? Why do we use $F(x) = \dfrac{1}{(1-x)^3}$, and how is $(1)$ computed? All help appreciated!
 [/INST] 
Denoting with $[x^n]$ the coefficient of $x^n$ of a series we obtain
\begin{align*}
\color{blue}{[x^n]F(x)}&=[x^n]\frac{x^2(1-x)}{(1-x)^3}\\
&=[x^n]\left(\frac{x^2}{(1-x)^3}-\frac{x^3}{(1-x)^3}\right)\tag{1}\\
&\,\,\color{blue}{=[x^{n-2}]\frac{1}{(1-x)^3}-[x^{n-3}]\frac{1}{(1-x)^3}}\tag{2}
\end{align*}
In the last line (2) we applied the rule $[x^p]x^qF(x)=[x^{p-q}]F(x)$.

We see in (2) what we essentially need is
\begin{align*}
[x^n]G(x)=[x^n]\frac{1}{(1-x)^3}
\end{align*}
Recalling the Taylor series expansion of $G(x)$ at $x=0$ we have
\begin{align*}
\color{blue}{[x^n]\frac{1}{(1-x)^3}}=[x^n]\sum_{k=0}^\infty \frac{G^{(k)}(0)}{k!}x^k
\color{blue}{=\frac{G^{(n)}(0)}{n!}}\tag{3}
\end{align*}
In the right-hand part of the equality chain we select the coefficient of $[x^n]$ which is the middle step of the hint to this problem. Here we use $G$ instead of $F$ to avoid using the same name for different functions.
We can easily calculate the $n$-th derivative of $G(x)$ as
\begin{align*}
G^{(n)}(x)&=\left(\frac{1}{(1-x)^3}\right)^{(n)}=\left((1-x)^{-3}\right)^{(n)}\\
&=(3)(4)\cdots(3+(n-1))(1-x)^{-3-n}\\
&=3\cdot4\cdots(n+2)\frac{1}{(1-x)^{n+3}}\\
&=\frac{(n+2)!}{2}\,\frac{1}{(1-x)^{n+3}} \tag{4}
\end{align*}

Combining (3) and (4) we see
\begin{align*}
\color{blue}{\frac{G^{(n)}(0)}{n!}}=\frac{(n+2)!}{2n!}\,\,\color{blue}{=\frac{(n+1)(n+2)}{2}}\tag{5}
\end{align*}
which was the final hint to this problem.

We see the Taylor series of $G(x)$ has the triangle numbers as coefficients
\begin{align*}
G(x)=1+3x+6x^2+10x^3+15x^4+\cdots\tag{6}
\end{align*}
We are now well prepared to derive the wanted coefficient $[x^n]F(x)$. We obtain from (2) and (5) for $n\geq 3$
\begin{align*}
\color{blue}{[x^n]F(x)}&=[x^{n-2}]\frac{1}{(1-x)^3}-[x^{n-3}]\frac{1}{(1-x)^3}\\
&=\frac{(n-1)n}{2}-\frac{(n-2)(n-1)}{2}\\
&=\frac{n-1}{2}(n-n+2)\\
&\,\,\color{blue}{=n-1}
\end{align*}
We finally need to determine the coeffcients of $[x^n]F(x)$ for $0\leq n\leq 2$. This is easy, since we already know $G(x)$ and we conclude from (2) and (6)
\begin{align*}
F(x)&=\left(x^2+3x^3+6x^4+\cdots\right)-\left(x^3+3x^4+6x^5+\cdots\right)\\
&=x^2+2x^3+3x^4+\cdots
\end{align*}

from which
\begin{align*}
\color{blue}{[x^n]F(x)=\begin{cases}
n-1&\qquad n\geq 1\\
0&\qquad n=0
\end{cases}}
\end{align*}
follows in accordance with the answer from @RobPratt.

</s>