Take the 2-minute tour ×

A measure of dependence is a way to assign a number (usually normalized between 0 and 1) to a couple of random variable, such that $\delta(X,Y)=0$ if and only of $X$ and $Y$ are independent, and $\delta(X,Y)=1$ as soon as there is a complete dependence between $X$ and $Y$. Rényi introduced in the paper On measures of dependence (Acta Math. Acad. Sci. Hungar. 10 1959 441--451) a set of axioms that a measure of dependence should fulfill. Among them are nonparametricity ($\delta(f(X),g(Y))=\delta(X,Y)$ as soon as $f,g$ are inversible bimeasurable functions, so that $\delta$ dos not rely on a metric, or affine structure unlike the correlation) and symmetry ($\delta(X,Y)=\delta(Y,X)$), and the "complete dependence" is defined by the existence of a relation $X=f(Y)$ of $Y=f(X)$.

I feel that the symmetry assumption is unnatural. In view of, for example, Bell inequalities, one would want to have some sort of triangle inequality, so that if $\delta(X,Y)$ and $\delta(Y,Z)$ are both very high, $\delta(X,Z)$ should be quite high too. This rules out symmetry since one can easily construct random variables such that $X=f(Y)$, $Z=g(Y)$ but $X$ and $Z$ are independent (in this example, one would want to say that $X$ depends heavily on $Y$, but that $Y$ does not depend that much on $X$).

Question: do you know a measure of dependence, or a similar tool, that is nonparametric and nonsymmetric? Does it satisfy a triangle inequality? Any reference would be useful.

share|improve this question
According to the work of Schweitzer and Wolff, it was not the symmetry condition which is unnatural, rather the "existence of a relation". His original conditions were satisfied only by the maximal correlation coefficient. projecteuclid.org/… –  Gjergji Zaimi Apr 23 '10 at 21:03
@Gjergji Zaimi: this reference is interesting, but I would not call a measure of dependency nonparameteric when it relies so heavily on the order structure (although it is certainly very weakly parametric, since the order is quite flexible compared to an affine structure). Imagine for example that you have a sociological study and you want to examine the correlation between the gender and some other characteristic, you do not have a structure, not even an order, on the set of gender. –  Benoît Kloeckner Apr 24 '10 at 6:32

1 Answer 1

Conditional entropy might be close to what you are looking for. You will have to transform it in an obvious way to fit Rényi's framework. See http://en.wikipedia.org/wiki/Conditional_entropy and especially the section titled "Intuition".

share|improve this answer
This is effectively close to what I am looking for; my question could mostly be recast into: « has (possibly renormalized) conditional entropy be used has a measure of dependency, and can it be nonparametrically extended to non-discrete random variable? » –  Benoît Kloeckner Apr 26 '10 at 8:21
Conditional entropy is often used as a measure of dependency: one specific example is in the realm of databases, where it's used to capture weak functional dependencies (Dalkilic/Robertson, Principles Of Database Systems, 2000). There, it's discrete. In general, you can define it via an integration, but like with differential entropy, you have to be careful –  Suresh Venkat Apr 27 '10 at 0:59

Your Answer


