In information theory, the Kraft–McMillan theorem establishes that any directly decodable coding scheme for coding a message to identify one value  out of a set of possibilities  can be seen as representing an implicit probability distribution  over , where  is the length of the code for  in bits. Therefore, relative entropy can be interpreted as the expected extra message-length per datum that must be communicated if a code that is optimal for a given (wrong) distribution  is used, compared to using a code based on the true distribution : it is the excess entropy.
where  is the cross entropy of  and , and  is the entropy of  (which is the same as the cross-entropy of P with itself).
The relative entropy  can be thought of geometrically as a statistical distance, a measure of how far the distribution  is from the distribution . Geometrically it is a divergence: an asymmetric, generalized form of squared distance. The cross-entropy  is itself such a measurement (formally a loss function), but it cannot be thought of as a distance, since  is not zero. This can be fixed by subtracting  to make  agree more closely with our notion of distance, as the excess loss. The resulting function is asymmetric, and while this can be symmetrized (see ), the asymmetric form is more useful. See  for more on the geometric interpretation.
Relative entropy relates to "rate function" in the theory of large deviations.
Arthur Hobson proved that relative entropy is the only measure of difference between probability distributions that satisfies some desired properties, which are the canonical extension to those appearing in a commonly used characterization of entropy. Consequently, mutual information is the only measure of mutual dependence that obeys certain related conditions, since it can be defined in terms of Kullback–Leibler divergence.