<p>Mutual information is defined for distribution and not individual points. So, I will write the next part assuming v1 and v2 are samples from a distribution, p. I will also take that you have n samples from p, n&gt;1.</p>
<p>You want a method to estimate mutual information from samples. There are many ways to do this. One of the simplest ways to do this would be to use a non-parametric estimator like NPEET (<a href="https://github.com/gregversteeg/NPEET" rel="nofollow noreferrer">https://github.com/gregversteeg/NPEET</a>). It works with numpy (you can convert from torch to numpy for this). There are more involved parametric models for which you may be able to find implementation in pytorch (See <a href="https://arxiv.org/abs/1905.06922" rel="nofollow noreferrer">https://arxiv.org/abs/1905.06922</a>).</p>
<p>If you only have two vectors and want to compute a similarity measure, a dot product similarity would be more suitable than mutual information as there is no distribution.</p>
How to calculate mutual information in PyTorch (differentiable estimator)
<p>I am training a model with pytorch, where I need to calculate the degree of dependence between two tensors (let's say they are the two tensors each containing values very close to zero or one, e.g. v1 = [0.999, 0.998, 0.001, 0.98] and v2 = [0.97, 0.01, 0.997, 0.999]) as a part of my loss function. I am trying to calculate <a href="https://en.wikipedia.org/wiki/Mutual_information" rel="nofollow noreferrer">mutual information</a>, but I can't find any mutual information estimation implementation in PyTorch. Has such a thing been provided anywhere?</p>