<s>[INST] Finding a closed form for $\sum_{k=0}^n (k^3-k-3)$ using generating functions I have been given a few hints. first of all I've been told to use the formula (for general series $a_n,b_n$):
$$\sum_{n=0}^\infty\left(\sum_{k=0}^n a_k b_{n-k}\right)x^n=\left(\sum_{i=0}^\infty a_i x^i\right)\left(\sum_{i=0}^\infty b_i x^i \right)$$
The direction I'm going with is simplifying the generating function of the series $\sum_{n=0}^\infty\left(\sum_{k=0}^n k^3-k-3 \right)x^n$ such that we set $a_k=k^3-k-3,b_k=1$ and thus get:
$$\left(\sum_{n=0}^\infty \left(k^3-k-3\right)x^n\right)\left(\sum_{n=0}^\infty x^n\right)$$
After simplifying A lot I finally got to the form $\frac{3x^{3}-3x^{2}+9x-3}{\left(-x+1\right)^{5}}$. I know this is not the intended way to do it because I have no idea how to get the coefficients from this. any help would be apperciated.
 [/INST] \begin{align}
&\frac{3x^3-3x^2+9x-3}{(1-x)^5}\\
&=(3x^3-3x^2+9x-3)\sum_{n=0}^\infty \binom{n+4}{4}x^n\\
&=\sum_{n=0}^\infty (3x^3-3x^2+9x-3) \binom{n+4}{4}x^n\\
&= 3 \sum_{n=0}^\infty \binom{n+4}{4}x^{n+3}-3\sum_{n=0}^\infty \binom{n+4}{4}x^{n+2}+9 \sum_{n=0}^\infty \binom{n+4}{4}x^{n+1}-3\sum_{n=0}^\infty \binom{n+4}{4} x^n\\
&= 3 \sum_{n=3}^\infty \binom{n+1}{4}x^n-3\sum_{n=2}^\infty \binom{n+2}{4}x^n+9 \sum_{n=1}^\infty \binom{n+3}{4}x^n-3\sum_{n=0}^\infty \binom{n+4}{4} x^n\\
&= 3 \sum_{n=0}^\infty \binom{n+1}{4}x^n -3\sum_{n=0}^\infty \binom{n+2}{4}x^n+9 \sum_{n=0}^\infty \binom{n+3}{4}x^n-3\sum_{n=0}^\infty \binom{n+4}{4} x^n\\
&= \sum_{n=0}^\infty\left(3\binom{n+1}{4} -3\binom{n+2}{4}+9 \binom{n+3}{4}-3 \binom{n+4}{4}\right) x^n\\
&=\sum_{n=0}^\infty \left(\frac{n^4}{4} + \frac{n^3}{2} - \frac{n^2}{4} - \frac{7 n}{2} - 3\right) x^n,
\end{align}
which yields
$$\sum_{k=0}^n (k^3-k-3) = \frac{n^4}{4} + \frac{n^3}{2} - \frac{n^2}{4} - \frac{7 n}{2} - 3$$

Here's a more direct snake-oil approach from start to finish, using partial fraction decomposition.
\begin{align}
&\sum_{n=0}^\infty \left(\sum_{k=0}^n (k^3-k-3)\right)x^n\\
&=\sum_{k=0}^\infty (k^3-k-3) \sum_{n=k}^\infty x^n\\
&=\sum_{k=0}^\infty (k^3-k-3) \frac{x^k}{1-x}\\
&=\frac{1}{1-x}\sum_{k=0}^\infty (k^3-k-3)x^k \\
&=\frac{1}{1-x}\left(\sum_{k=0}^\infty k^3 x^k-\sum_{k=0}^\infty k x^k-3\sum_{k=0}^\infty x^k\right) \\
&=\frac{1}{1-x}\left(\frac{x(1+4x+x^2)}{(1-x)^4}-\frac{x}{(1-x)^2}-\frac{3}{1-x}\right) \\
&=\frac{x(1+4x+x^2)}{(1-x)^5}-\frac{x}{(1-x)^3}-\frac{3}{(1-x)^2} \\
&= \frac{-3}{(1-x)^2} + \frac{6}{(1-x)^3} - \frac{12}{(1-x)^4} + \frac{6}{(1-x)^5}\\
&= \sum_{n=0}^\infty \left(-3\binom{n+1}{1} + 6\binom{n+2}{2} - 12\binom{n+3}{3} + 6\binom{n+4}{4}\right)x^n\\
&=\sum_{n=0}^\infty \left(\frac{n^4}{4} + \frac{n^3}{2} - \frac{n^2}{4} - \frac{7 n}{2} - 3\right) x^n
\end{align}
</s>