<p>This is the framing of the "Information Buttleneck" problem, which can be solved in some cases in EM-like iterative algorithm or heuristically through greedy, agglomerative clustering process. </p>

<p>Useful references:</p>

<ul>
<li><a href="https://en.wikipedia.org/wiki/Information_bottleneck_method" rel="nofollow">Wikipedia</a></li>
<li><a href="https://arxiv.org/pdf/physics/0004057.pdf" rel="nofollow">Paper</a></li>
<li>Matlab code: <a href="http://ai.stanford.edu/~gal/code.html" rel="nofollow">(1) by G. Chechik</a> or <a href="http://www.vlfeat.org/matlab/matlab.html" rel="nofollow">(2) in the VLfeat library</a></li>
</ul>

<p>Indeed it is deeply related to CCA; under certain assumptions on the problem (i.e. Gaussianity) this relation can be made exact (see Wikipedia link and <a href="http://www.jmlr.org/papers/volume6/chechik05a/chechik05a.pdf" rel="nofollow">this paper</a>).</p>
Is there any feature extraction methods based on mutual information?
<p>I got a dataset D={X,y} which have 800 input features and single continuous output. I am looking for any feature extraction methods that satisfy two conditions</p>

<p>(1) Matlab codes are available to download</p>

<p>(2). The method should somehow map input x to transformed input z where z is d vector, (d&lt;&lt;800), such that the mutual information between z_i, and y as high as possible. </p>

<p>I think the methods should relate to CCA, however when perform CCA(X,y) I will obtain vector z that have only one dimension. I hope the methods should have the options to select top d good features like PCA does.</p>

<p>Thanks,</p>