title: Python multithreading with local dictionary to speed up process
body: <p>I have a process that is very slow, and I want to speed it up using multithreading. The goal of my process is to read through a very large dataset and do an expensive calculation on each line, then store the result in a dictionary. I want to use multithreading but I am not sure how. Here is my attempt.</p>
  
  <pre><code>from sklearn.feature_extraction.text import TfidfVectorizer
  from sklearn.feature_extraction.text import CountVectorizer
  from sklearn.metrics.pairwise import cosine_similarity
  from multiprocessing.pool import ThreadPool
  
  def findTweets(side):
       cosine_dict = {}
       for t in tweets:
              topic = [side, t]
              tfidf_vectorizer = TfidfVectorizer()
              topic_matrix = tfidf_vectorizer.fit_transform(topic)
              cosine = cosine_similarity(topic_matrix[0:1], topic_matrix[1:2])
              cosine = float(cosine)
              key = side + "&amp;&amp;" + t
              cosine_dict[key] = cosine
       return cosine_dict
  
  left = [] #just some strings
  
  for l in left:
         pool = ThreadPool(processes = 10)
         result = pool.apply_async(findTweets, (l,))
         cosine_dict_left = result.get()
  </code></pre>
  
  <p>This does not seem to be speeding up performance. How can I apply multithreading here to speed up this process?</p>