<h4 dir="auto">Description</h4>
<p dir="auto">ValueError thrown when applying AgglomerativeClustering on textual data because distance matrix contains infinite values</p>
<h4 dir="auto">Steps/Code to Reproduce</h4>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import numpy as np
from sklearn.datasets import fetch_20newsgroups
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.cluster import AgglomerativeClustering
def main():
        dataset = fetch_20newsgroups(subset='all', shuffle=True, random_state=42, 
                                                            remove=('headers', 'footers', 'quotes') )
        data_samples = dataset.data
	targets = dataset.target
	categories = dataset.target_names
	k = np.unique(targets).shape[0]
	tf_vectorizer = TfidfVectorizer(max_features=50000, max_df=1.0, min_df=1)
	tfs = tf_vectorizer.fit_transform(data_samples)
	agg = AgglomerativeClustering(linkage=&quot;complete&quot;, n_clusters=k, affinity=&quot;cosine&quot;)
	agg.fit(tfs.toarray())
	return dataset

if __name__ == '__main__':
	main()"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">numpy</span> <span class="pl-k">as</span> <span class="pl-s1">np</span>
<span class="pl-k">from</span> <span class="pl-s1">sklearn</span>.<span class="pl-s1">datasets</span> <span class="pl-k">import</span> <span class="pl-s1">fetch_20newsgroups</span>
<span class="pl-k">from</span> <span class="pl-s1">sklearn</span>.<span class="pl-s1">feature_extraction</span>.<span class="pl-s1">text</span> <span class="pl-k">import</span> <span class="pl-v">TfidfVectorizer</span>
<span class="pl-k">from</span> <span class="pl-s1">sklearn</span>.<span class="pl-s1">cluster</span> <span class="pl-k">import</span> <span class="pl-v">AgglomerativeClustering</span>
<span class="pl-k">def</span> <span class="pl-en">main</span>():
        <span class="pl-s1">dataset</span> <span class="pl-c1">=</span> <span class="pl-en">fetch_20newsgroups</span>(<span class="pl-s1">subset</span><span class="pl-c1">=</span><span class="pl-s">'all'</span>, <span class="pl-s1">shuffle</span><span class="pl-c1">=</span><span class="pl-c1">True</span>, <span class="pl-s1">random_state</span><span class="pl-c1">=</span><span class="pl-c1">42</span>, 
                                                            <span class="pl-s1">remove</span><span class="pl-c1">=</span>(<span class="pl-s">'headers'</span>, <span class="pl-s">'footers'</span>, <span class="pl-s">'quotes'</span>) )
        <span class="pl-s1">data_samples</span> <span class="pl-c1">=</span> <span class="pl-s1">dataset</span>.<span class="pl-s1">data</span>
	<span class="pl-s1">targets</span> <span class="pl-c1">=</span> <span class="pl-s1">dataset</span>.<span class="pl-s1">target</span>
	<span class="pl-s1">categories</span> <span class="pl-c1">=</span> <span class="pl-s1">dataset</span>.<span class="pl-s1">target_names</span>
	<span class="pl-s1">k</span> <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-en">unique</span>(<span class="pl-s1">targets</span>).<span class="pl-s1">shape</span>[<span class="pl-c1">0</span>]
	<span class="pl-s1">tf_vectorizer</span> <span class="pl-c1">=</span> <span class="pl-v">TfidfVectorizer</span>(<span class="pl-s1">max_features</span><span class="pl-c1">=</span><span class="pl-c1">50000</span>, <span class="pl-s1">max_df</span><span class="pl-c1">=</span><span class="pl-c1">1.0</span>, <span class="pl-s1">min_df</span><span class="pl-c1">=</span><span class="pl-c1">1</span>)
	<span class="pl-s1">tfs</span> <span class="pl-c1">=</span> <span class="pl-s1">tf_vectorizer</span>.<span class="pl-en">fit_transform</span>(<span class="pl-s1">data_samples</span>)
	<span class="pl-s1">agg</span> <span class="pl-c1">=</span> <span class="pl-v">AgglomerativeClustering</span>(<span class="pl-s1">linkage</span><span class="pl-c1">=</span><span class="pl-s">"complete"</span>, <span class="pl-s1">n_clusters</span><span class="pl-c1">=</span><span class="pl-s1">k</span>, <span class="pl-s1">affinity</span><span class="pl-c1">=</span><span class="pl-s">"cosine"</span>)
	<span class="pl-s1">agg</span>.<span class="pl-en">fit</span>(<span class="pl-s1">tfs</span>.<span class="pl-en">toarray</span>())
	<span class="pl-k">return</span> <span class="pl-s1">dataset</span>

<span class="pl-k">if</span> <span class="pl-s1">__name__</span> <span class="pl-c1">==</span> <span class="pl-s">'__main__'</span>:
	<span class="pl-en">main</span>()</pre></div>
<h4 dir="auto">Expected Results</h4>
<p dir="auto">No error is thrown and the distance matrix should not contain infinite values</p>
<h4 dir="auto">Actual Results</h4>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="File &quot;/venv/lib/python3.5/site-packages/sklearn/cluster/hierarchical.py&quot;, line 750, in fit
    **kwargs)
  File &quot;/venv/lib/python3.5/site-packages/sklearn/externals/joblib/memory.py&quot;, line 362, in __call__
    return self.func(*args, **kwargs)
  File &quot;/venv/lib/python3.5/site-packages/sklearn/cluster/hierarchical.py&quot;, line 527, in _complete_linkage
    return linkage_tree(*args, **kwargs)
  File &quot;/venv/lib/python3.5/site-packages/sklearn/cluster/hierarchical.py&quot;, line 417, in linkage_tree
    out = hierarchy.linkage(X, method=linkage, metric=affinity)
  File &quot;/venv/lib/python3.5/site-packages/scipy/cluster/hierarchy.py&quot;, line 713, in linkage
    raise ValueError(&quot;The condensed distance matrix must contain only &quot;
ValueError: The condensed distance matrix must contain only finite values."><pre class="notranslate">File <span class="pl-s"><span class="pl-pds">"</span>/venv/lib/python3.5/site-packages/sklearn/cluster/hierarchical.py<span class="pl-pds">"</span></span>, line 750, <span class="pl-k">in</span> fit
    <span class="pl-k">**</span>kwargs)
  File <span class="pl-s"><span class="pl-pds">"</span>/venv/lib/python3.5/site-packages/sklearn/externals/joblib/memory.py<span class="pl-pds">"</span></span>, line 362, <span class="pl-k">in</span> __call__
    <span class="pl-k">return</span> self.func(<span class="pl-k">*</span>args, <span class="pl-k">**</span>kwargs)
  File <span class="pl-s"><span class="pl-pds">"</span>/venv/lib/python3.5/site-packages/sklearn/cluster/hierarchical.py<span class="pl-pds">"</span></span>, line 527, <span class="pl-k">in</span> _complete_linkage
    <span class="pl-k">return</span> linkage_tree(<span class="pl-k">*</span>args, <span class="pl-k">**</span>kwargs)
  File <span class="pl-s"><span class="pl-pds">"</span>/venv/lib/python3.5/site-packages/sklearn/cluster/hierarchical.py<span class="pl-pds">"</span></span>, line 417, <span class="pl-k">in</span> linkage_tree
    out = hierarchy.linkage(X, method=linkage, metric=affinity)
  File <span class="pl-s"><span class="pl-pds">"</span>/venv/lib/python3.5/site-packages/scipy/cluster/hierarchy.py<span class="pl-pds">"</span></span>, line 713, <span class="pl-k">in</span> linkage
    raise ValueError(<span class="pl-s"><span class="pl-pds">"</span>The condensed distance matrix must contain only <span class="pl-pds">"</span></span>
ValueError: The condensed distance matrix must contain only finite values.</pre></div>
<h4 dir="auto">Versions</h4>

<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="&gt;&gt;&gt; import platform; print(platform.platform())
Linux-4.4.0-81-generic-x86_64-with-Ubuntu-16.04-xenial
&gt;&gt;&gt; import sys; print(&quot;Python&quot;, sys.version)
Python 3.5.2 (default, Nov 17 2016, 17:05:23) 
[GCC 5.4.0 20160609]
&gt;&gt;&gt; import numpy; print(&quot;NumPy&quot;, numpy.__version__)
NumPy 1.13.3
&gt;&gt;&gt; import scipy; print(&quot;SciPy&quot;, scipy.__version__)
SciPy 1.0.0
&gt;&gt;&gt; import sklearn; print(&quot;Scikit-Learn&quot;, sklearn.__version__)
Scikit-Learn 0.19.0
&gt;&gt;&gt; "><pre class="notranslate">&gt;&gt;&gt; import platform<span class="pl-k">;</span> <span class="pl-en">print(platform.platform</span>())
Linux-4.4.0-81-generic-x86_64-with-Ubuntu-16.04-xenial
&gt;&gt;&gt; import sys<span class="pl-k">;</span> print(<span class="pl-s"><span class="pl-pds">"</span>Python<span class="pl-pds">"</span></span>, sys.version)
Python 3.5.2 (default, Nov 17 2016, 17:05:23) 
[GCC 5.4.0 20160609]
&gt;&gt;&gt; import numpy<span class="pl-k">;</span> print(<span class="pl-s"><span class="pl-pds">"</span>NumPy<span class="pl-pds">"</span></span>, numpy.__version__)
NumPy 1.13.3
&gt;&gt;&gt; import scipy<span class="pl-k">;</span> print(<span class="pl-s"><span class="pl-pds">"</span>SciPy<span class="pl-pds">"</span></span>, scipy.__version__)
SciPy 1.0.0
&gt;&gt;&gt; import sklearn<span class="pl-k">;</span> print(<span class="pl-s"><span class="pl-pds">"</span>Scikit-Learn<span class="pl-pds">"</span></span>, sklearn.__version__)
Scikit-Learn 0.19.0
&gt;&gt;&gt; </pre></div>
<p dir="auto"><strong>Comment</strong><br>
I have used the same code on a subset of Reuters-21578 text data set and no error was thrown. I was not able to track down what might have caused the infinite values in the distance matrix</p>
<p dir="auto">Original title: <em><strong>"Cosine" affinity type in FeatureAgglomeration somehow casue memory overflow in a particular dataset</strong></em></p>


<h4 dir="auto">Description</h4>

<p dir="auto">Please carefully test the codes below! Using "cosine" affinity type in FeatureAgglomeration, the codes will cause memory overflow with a particular dataset (<a href="http://gleason.case.edu/webdata/tpot/" rel="nofollow">Download here</a>). It is all right with other affinity types. And this issue cannot be reproduced using simulation data (like using make_classification in sci-kit learn). Not sure why it happened.</p>
<h4 dir="auto">Steps/Code to Reproduce</h4>

<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="from sklearn.cluster import FeatureAgglomeration
import numpy as np
import time
train_data = np.genfromtxt('fold_2_trainFeatVec.csv', delimiter=',')
train_labels= np.genfromtxt('fold_2_trainLabels.csv', delimiter=',')


fa = FeatureAgglomeration(affinity=&quot;cosine&quot;, linkage=&quot;average&quot;) # no matter linage =&quot;average&quot; or &quot;complete&quot;
time_start= time.time()
fa.fit(train_data, train_labels) # memory keeps increasing here
time_end = time.time()
print('Time usage:',time_end-time_start)
"><pre class="notranslate"><code class="notranslate">from sklearn.cluster import FeatureAgglomeration
import numpy as np
import time
train_data = np.genfromtxt('fold_2_trainFeatVec.csv', delimiter=',')
train_labels= np.genfromtxt('fold_2_trainLabels.csv', delimiter=',')


fa = FeatureAgglomeration(affinity="cosine", linkage="average") # no matter linage ="average" or "complete"
time_start= time.time()
fa.fit(train_data, train_labels) # memory keeps increasing here
time_end = time.time()
print('Time usage:',time_end-time_start)

</code></pre></div>