The principle of insufficient reason says that all outcomes are equiprobable when we have no knowledge to guess otherwise. I understand this and that this corresponds to uniform distribution. However, different sources say that this is only true for discrete case. For continuous distributions, normal distribution corresponds to maximum entropy. Here is WP:

the maximum entropy prior on a discrete space, given only that the probability is normalized to 1, is the prior that assigns equal probability to each state. And in the continuous case, the maximum entropy prior given that the density is normalized with mean zero and variance unity is the standard normal distribution.

I cannot understand why line starts bending when we divide it into continuum of outcomes. What is the expectation (mean, peak) and variance of such normal distribution? How can constant converge to a curve, divided into more intervals? The normal distribution is different from uniform in that latter has a peak and, thus, some outcomes are more probable. How that be based on the equiprobability principle? Where the variance comes from in the continuous case? I read the article http://www.math.uconn.edu/~kconrad/blurbs/analysis/entropypost.pdf on how that it is derived, but could not grasp it. Can you explain qualitatively?

Furthermore, I see that they prove Lemma 4.2

$$\sum {p \log p \leq p \log q}$$

However, I do not understand how they prove Teorems 3.1-3.3 using it in Chapter 4. They just choose q distribution to be uniform, $q_i = 1/n$ in discrete case and normal in continuous case and using the lemma prove that entropy of q, $h(q) = \sum {p_i \log q_i}$ is greater or equal to any distribution p with entropy $h(p) = \sum {p_i \log p_i}$. This indeed follows from the Lemma 4.2. However, I do not understand two things:

  1. Why they define entropy of distribution p as $h(p) = p \log p$ but treat two entries of p as independent variables in computation of h(q). How can they replace only one entry with q and say that this is entropy of q? IMo, entropy of q is $\sum q \log q$ and it is $\neq \sum p \log q$
  2. What it has to do with the uniform (normal) distribution? I can take q to be any other distribution? Lemma 4.2 will prove that it is greater than the entropy of p for sure!
  • 1
    $\begingroup$ Two comments: 1) the reference you are citing for Lemma 4.2 and so on was written by one person, so it reads a bit strangely to keep saying "they prove". You aren't multiple people either. 2) If you don't understand the proofs of Theorems 3.1--3.3, at what part of those proofs are you stuck? I think that would be a more productive line of questioning than the two things you say you don't understand, because for instance your second question makes it sound like you simply haven't even tried to identify why you don't follow the proof of Theorem 3.2 (involving the normal distribution). $\endgroup$ – KCd Jan 12 '13 at 9:33
  • $\begingroup$ Nowhere in the link is an expression like $\sum p\log q$ or $\int p\log q$ called an entropy; it's just an auxiliary expression, so you're misreading the role of such expressions by claiming (incorrectly) that they are ever called entropy. $\endgroup$ – KCd Jan 12 '13 at 9:35
  • $\begingroup$ There is no convergence going on from the discrete to the continuous. The two situations are treated on their own terms, one using Lemma 4.1 and the other using Lemma 4.2 (readers who know measure theory don't have to remind me that those two lemmas can be unified, as that is beyond the scope of this discussion). It appears that you would like to think there is a secret limiting process that makes the max entropy uniform distribution on a finite set "converge" to a max entropy normal distribution on the real line, but the way those are handled in the link uses no such idea at all. $\endgroup$ – KCd Jan 12 '13 at 9:40
  • $\begingroup$ Thanks KCd. It makes my main question more important. How uniform transforms into normal in the limiting case? How is this possible? $\endgroup$ – Val Jan 12 '13 at 10:18
  • $\begingroup$ Can you tell us what you mean by the phrase "limiting case" using math (not just words)? In the central limit theorem, where a normal distribution is a limit of suitably defined averages of i.i.d. random variables, even if the random variables are uniform (like flipping a fair coin over and over) those averages that are converging to the normal distribution are not uniform after the very first step. $\endgroup$ – KCd Jan 12 '13 at 21:12

This is only an answer to your first question.


In the paper $h(q)$ is not computed this way. The inequality of Lemma 4.2 is used to prove that $h(p) \le log(n)$ and $h(p) \lt log(n)$ if $p$ is not the uniform distribution with $p_1=p_2=\ldots p_n=\frac{1}{n}$

Lemma 4.2: $$-\sum_{i=1}^{n}p_i \log{p_i} \le -\sum_{i=1}^{n}p_i \log{q_i} \tag{1} $$ Equality holds iff $$p_i=q_i, i=1,\ldots , n \tag{2}$$


We know that the entropy is defined by $$h(p)=-\sum_{i=1}^{n}p_i \log{p_i} \tag{3} $$ This can be used to reformulate the inequation of the Lemma as

$$ h(p)\le -\sum_{i=1}^{n}p_i \log{q_i} \tag{4} $$

This is valid for all discrete distributions so also for the uniform distribution with $$q_i=\frac{1}{n} ,i=1,\ldots,n \tag{4a} $$ Substituting $\frac{1}{n}$ for $q_i$ gives

$$ h(p)\le \sum_{i=1}^{n}p_i \log{n} = (\log{n}) \cdot \sum_{i=1}^{n}p_i = \log{n} \tag{5} $$

But $log{(n)}$ is also $h(q)$, if $q$ is the uniform distribution. This can checked simply by using the definition of the entropy:

$$h(q)=-\sum_{i=1}^{n}q_i \log{q_i}=-\sum_{i=1}^{n}\frac{1}{n} \log{\frac{1}{n}} = \log{n} \sum_{i=1}^{n}\frac{1}{n} = \log{n} \tag{6} $$ So it follows that for the uniform distribution $q$ $$h(p) \le \log{n} = h(q) \tag{7} $$

Because of $(6)$ and $(2)$ equality holds exactly if $p$ is the uniform distribution too.


Theorem 5.1 states, that the continous probability density on [a,b] with $\mu = \frac{a+b}{2}$ that maximizes entropy is uniform distribution $q(x)=\frac{1}{b-a}, x \in [a,b]$. This complies with the principle of indifference for coninous variable found here.

On the whole real line there is no uniform probability density. On the whole real line there is also no continous probability density with highest entropy, because there are continous probability densities with arbitrary high entropies, e.g. the gaussian distribution has entropy $\frac{1}{2}(1+\log(2 \pi \sigma^2))$: if we increase $\sigma$ the entropy increases.

Because there is no maximal entropy for continuous densities over $R$ we must have additional constraints, e.g. the constraint that $\sigma$ is fixed and that $\mu$ is fixed. The fact that there is a given finite $\sigma^2$ and $\mu$ for me makes intuitively clear that there values nearer to $\mu$ must have higher probability. If you don't fix $\mu$ then you will get no unique solution.The Gaussian distribution for each real $\mu$ is a solution: this is some kind of "uniformness", all $\mu$ can be used for a solution.

Notice that it is crucial to fix $\sigma$, $\mu$ and to demand $p(x)>0 , \forall x \in R$. If you fix other values or change the form $R$ to another domain for the density funtion , e.g. $R^+$, you will get other solution: the exponential distribution, the truncated exponential distribution, the laplace distribution, the lognorma distribution (Theorems 3.3, 5.1, 5.2, 5.3)

  • $\begingroup$ Ok, the key point is that entropy of uniform, $\sum {1/n \log {1/n}} = \log n$, happens to be an upper limit as $\sum {p \log {1/n} = \log n}$. Thanks. I'll accept your question temporarily, in case nobody else responds. I think that you also suggest why the uniform and nothing else is the max distribution: the upper limit must be unique (do you call it well-definiteness?). This is, perhaps, because there is only one p log p that turns inequality into equality and, since we have got one, it is it. This is definitely something. I still do not understand qualitatively why uniform = normal? $\endgroup$ – Val Jan 11 '13 at 18:18
  • $\begingroup$ I am not sure if I ubderstand you right. The uniform is the only upper limit because of the fact that equality in (5) ( which is (1) from Lemma 4.2 where the uniform distribution is substituted for q) holds exactly if (2) holds (where $\frac{1}{n}$ is substituted for $q_i$) $\endgroup$ – miracle173 Jan 12 '13 at 7:12
  • $\begingroup$ @Val: Section 7 of the link addresses the issue of uniqueness. That there could be an issue about that at all is illustrated by the normals with the same $\mu$ all being max. entropy distributions, so something has to be imposed to expect there to be at most one max. entropy distribution. It is definitely not guaranteed in advance in all situations. Theorem 7.7 (which relies on notation presented before Example 7.4) is the main general result there on uniqueness, but the whole of section 7 is probably written at a level above your mathematical background right now. $\endgroup$ – KCd Jan 12 '13 at 9:50

Your Answer

