The idea of relative entropy as discrimination information led Kullback to propose the Principle of  (MDI): given new facts, a new distribution  should be chosen which is as hard to discriminate from the original distribution  as possible; so that the new data produces as small an information gain  as possible.
For example, if one had a prior distribution  over  and , and subsequently learnt the true distribution of  was , then the relative entropy between the new joint distribution for  and , , and the earlier prior distribution would be:
i.e. the sum of the relative entropy of  the prior distribution for  from the updated distribution , plus the expected value (using the probability distribution ) of the relative entropy of the prior conditional distribution  from the new conditional distribution . (Note that often the later expected value is called the conditional relative entropy (or conditional Kullback–Leibler divergence) and denoted by ) This is minimized if  over the whole support of ; and we note that this result incorporates Bayes' theorem, if the new distribution  is in fact a δ function representing certainty that  has one particular value.
MDI can be seen as an extension of Laplaces Principle of Insufficient Reason, and the Principle of Maximum Entropy of E.T. Jaynes. In particular, it is the natural extension of the principle of maximum entropy from discrete to continuous distributions, for which Shannon entropy ceases to be so useful (see differential entropy'), but the relative entropy continues to be just as relevant.
In the engineering literature, MDI is sometimes called the Principle of Minimum Cross-Entropy (MCE) or Minxent for short. Minimising relative entropy from  to  with respect to  is equivalent to minimizing the cross-entropy of  and , since
which is appropriate if one is trying to choose an adequate approximation to . However, this is just as often not the task one is trying to achieve. Instead, just as often it is  that is some fixed prior reference measure, and  that one is attempting to optimise by minimising  subject to some constraint. This has led to some ambiguity in the literature, with some authors attempting to resolve the inconsistency by redefining cross-entropy to be , rather than  .