In Bayesian statistics, relative entropy can be used as a measure of the information gain in moving from a prior distribution to a posterior distribution: . If some new fact  is discovered, it can be used to update the posterior distribution for  from  to a new posterior distribution  using Bayes' theorem:
This distribution has a new entropy:
which may be less than or greater than the original entropy . However, from the standpoint of the new probability distribution one can estimate that to have used the original code based on  instead of a new code based on  would have added an expected number of bits:
to the message length. This therefore represents the amount of useful information, or information gain, about , that has been learned by discovering .
If a further piece of data, , subsequently comes in, the probability distribution for  can be updated further, to give a new best guess . If one reinvestigates the information gain for using  rather than , it turns out that it may be either greater or less than previously estimated:
: may be ≤ or > than 
and so the combined information gain does not obey the triangle inequality:
: may be 
All one can say is that on average, averaging using , the two sides will average out.