<p>(Taken from <a href="https://arxiv.org/pdf/1711.07758.pdf" rel="nofollow noreferrer">UNDERSTANDING DEEP LEARNING
GENERALIZATION
BY
MAXIMUM
ENTROPY (Zheng et al., 2017)</a>:</p>
<p>(Original Maximum Entropy Model) Supposing the dataset has input X and label
Y, the task is to find a good prediction of Y using X. The prediction Yˆ needs to maximize the
conditional entropy H(Yˆ |X) while preserving the same distribution with data (X, Y ). This is
formulated as:</p>
<p>min −H(Yˆ |X) (1)</p>
<p>s.t. P(X, Y ) = P(X, Yˆ ),
\sum(Yˆ) P(Yˆ |X) = 1</p>
<p>Berger et al., 1996 solves this with lagrange multipliers ωi as an exponential form:</p>
<p>Pω(Yˆ = y|X = x) = 1/Zω(x) exp (\sum(i) ωifi(x, y))</p>
What is Maximum Entropy?
<p>Can someone give me a clear and simple definition of Maximum entropy classification? It would be very helpful if someone can provide a clear analogy, as I am struggling to understand.</p>