title: How to find cosine similarity for of a very Large Array
body: <p>I have a very large domain name dataset. Approx size of the dataset is 1 million.</p>
  
  <p>I want to find similar domains which are duplicate in dataset due to wrong spelling.</p>
  
  <p>So I have been using cosine similarity for finding similar documents.</p>
  
  <pre><code>dataset = ["example.com","examplecom","googl.com","google.com"........]
  tfidf_vectorizer = TfidfVectorizer(analyzer="char")
  tfidf_matrix = tfidf_vectorizer.fit_transform(documents)
  cs = cosine_similarity(tfidf_matrix, tfidf_matrix)
  </code></pre>
  
  <p>Above example is working fine for small dataset but for a large dataset, it is throwing out of memory error.</p>
  
  <p>System Configuration:</p>
  
  <p>1)8GB Ram</p>
  
  <p>2)64 bit system and 64 bit python installed</p>
  
  <p>3)i3-3210 processor</p>
  
  <p>How to find cosine similarity for a large dataset?</p>