I'm trying to solve a exercise about Shannon entropy. I've tried some things but I can't solve it, and couldn't find information on the internet.

Let's see. We have the Shannon entropy, and a system defined by a probability distribution $p(n)$:

$$S = \sum _{n\in N} p(n) \log (p(n)) $$

Where $N$ can be a finite set of probabilites or a infinite one, and $n$ is integer.

Now let's define the generating function of information (not sure if this is its name in English):

$$G(u) = \sum _{n \in N} p^u(n)$$

The question is, how can I write $S$ in terms of $G(u)$? Also it would be great to have a physical interpretation of this function (but the exercise doesn't ask for it)

My attempts to get the solution: I tried to write $G(1)$ and insert it into the expression of $S$, but I cannot get rid of the logarithm. Also I've tried to make a Taylor expansion of the logarithm, but I cannot write the series around $x=0$ because the logarithm is not analytic in this point. Around $x=1$,

$$\log(x) = - \sum _{k=1} ^{+\infty} \dfrac{(-1)^k (x-1)^k}{k}$$

But it looks like I can't substitute the expression of $G(u)$ inside this using $x\equiv p(n)$.

Any advice or hint is well received. As I've said, I haven't found any information about the function $G(u)$ on the Internet.

  • 1
    $\begingroup$ And what do you get if you expand G(u) around u = 1? $\endgroup$ – Count Iblis Mar 21 '16 at 19:33
  • 1
    $\begingroup$ Try using partial derivarives with respect (say, for example, $\frac{\partial G}{\partial u}$, then evaluate the function at $u=1$, your initial idea). That is the way most generating functions work. $\endgroup$ – AGEscovar Mar 21 '16 at 19:45

It wasn't. After a recalculus I've realized that if you take the derivative of $G(u)$:

$$\dfrac{dG(u)}{du} = \sum _{n\in N} p^u(n)\log(p(n))$$

using that the derivative of $a^x$ is $a^x \log(a)$. Now, taking $u=1$ in the above expression, you find the Shannon entropy, so

$$S=\left [\dfrac{dG(u)}{du} \right ]_{u=1}$$


Your Answer

