<s>[INST] Please help me compute this$ \sum_m\binom{n}{m}\sum_k\frac{\binom{a+bk}{m}\binom{k-n-1}{k}}{a+bk+1}$ Compute following:
$$
\sum_m\binom{n}{m}\sum_k\frac{\binom{a+bk}{m}\binom{k-n-1}{k}}{a+bk+1}
$$
Only consider real numbers a, b such that the denominators are never 0.
Now I simplify it into
$$
-\frac{1}{n}\sum_k\binom{n}{k}\binom{-n}{a+bk+1}(-1)^{a+bk+k}
$$
I have problem with this question in which I can't eliminate coefficient b.
But I can't find any formula to get answer.Please help me!
 [/INST] Here is a slightly different variation of the theme. It is convenient to use the coefficient of operator $[z^n]$ to denote the coefficient of $z^n$ of a series. This way we can write e.g.
\begin{align*}
[z^k](1+z)^n=\binom{n}{k}
\end{align*}

We obtain
  \begin{align*}
\sum_{m=0}^{n}&\binom{n}{m}\sum_{k=0}^{n}\frac{1}{a+bk+1}\binom{a+bk}{m}\binom{k-n-1}{k}\tag{1}\\
&=\sum_{k=0}^{n}\frac{(-1)^k}{a+bk+1}\binom{n}{k}\sum_{m=0}^{n}\binom{n}{m}\binom{a+bk}{m}\tag{2}\\
&=\sum_{k=0}^{n}\frac{(-1)^k}{a+bk+1}\binom{n}{k}\binom{a+bk+n}{n}\tag{3}\\
&=\frac{1}{n}\sum_{k=0}^{\infty}(-1)^k\binom{n}{k}\binom{a+bk+n}{n-1}\tag{4}\\
&=\frac{1}{n}\sum_{k=0}^{\infty}(-1)^k[z^{k}](1+z)^{n}[u^{n-1}](1+u)^{a+bk+n}\tag{5}\\
&=\frac{1}{n}[u^{n-1}](1+u)^{a+n}\sum_{k=0}^\infty(-1)^k(1+u)^{bk}[z^k](1+z)^n\tag{6}\\
&=\frac{1}{n}[u^{n-1}](1+u)^{a+n}(1-(1+u)^b)^n\tag{7}\\
&=\frac{(-1)^n}{n}[u^{n-1}](1+u)^{a+n}\left(\sum_{j=1}^\infty\binom{b}{j}u^j\right)^n\tag{8}\\
&=0
\end{align*}

Comment:


*

*In (1) we write lower and upper limits of the sum.

*In (2) we exchange the sums, do small rearrangements and use the identity
\begin{align*}
  \binom{k-n-1}{k}=(-1)^k\binom{n}{k}
  \end{align*}

*In (3) we apply Vandermonde's identity. With $q:= a+bk$ we get
\begin{align*}
  \sum_{m=0}^{n}\binom{n}{m}\binom{q}{m}=\sum_{m=0}^{n}\binom{q}{m}\binom{n}{n-m}=\binom{q+n}{n}
  \end{align*}

*In (4) we use the identity
\begin{align*}
\frac{q}{p-q+1}\binom{p}{q}=\binom{p}{q-1}\\
  \end{align*}
and we also change the uppper limit of the series to $\infty$ without changing anything since we add only zeros.

*In (5) we apply the coefficient of operator twice

*In (6) we use the linearity of the coefficient of operator and do some rearrangements

*In (7) we apply the substitution rule of the coefficient of operator
\begin{align*}
  A(u)=\sum_{k=0}^\infty a_ku^k=\sum_{k=0}^\infty u^k[z^k]A(z)
  \end{align*}

*In (8) we use the binomial series expansion and observe the smallest power of $u$ is $n$ so that the coefficient $[u^{n-1}]$ is zero.
</s>