title: AgglomerativeClustering on a correlation matrix
body: <p>I have a correlation matrix of typical structure that is of size 288x288 that is defined by:</p>
  
  <pre><code>from sklearn.cluster import AgglomerativeClustering
  df = read_returns()
  correl_matrix = df.corr()
  </code></pre>
  
  <p>where read_returns gives me a dataframe with a date index, and columns of the returns of assets.</p>
  
  <p>Now - I want to cluster these correlations to reduce the population size.</p>
  
  <p>By doing some reading and experimenting I discovered AgglomerativeClustering - and it appears at first pass to be an appropriate solution to my problem.</p>
  
  <p>I define a distance metric as <code>((.5*(1-correl_matrix))**.5)</code> and have:</p>
  
  <pre><code>cluster = AgglomerativeClustering(n_clusters=40, linkage='average')
  cluster.fit(((.5*(1-correl_matrix))**.5).values)
  label_groups = cluster.labels_
  </code></pre>
  
  <p>To observe some of the data and cross check my work I pick out cluster <code>1</code> and observe the pairwise correlations and find the min correlation between two items with that group in my dataset to find :</p>
  
  <pre><code>single_cluster = []
  for i in range(0,correl_matrix.shape[0]):
      if label_groups[i]==1:
          single_cluster.append(correl_matrix.index[i])
  
  min_correl = 1.0
  for x in single_cluster:
      for y in single_cluster:
          if x&lt;&gt;y:
              if correl_matrix[x][y]&lt;min_correl:
                  min_correl = correl_matrix[x][y]
  
  print min_correl
  </code></pre>
  
  <p>and get a min pairwise correlation of .20</p>
  
  <p>To me this seems quite low - but "low based off what?" is a fair question to which I have no answer.</p>
  
  <p>I would like to anticipate/enforce each pairwise correlation of a cluster to be >=.7 or something like this.</p>
  
  <p>Is this possible in AgglomerativeClustering?</p>
  
  <p>Am I accidentally going down the wrong path?</p>