title: simple k-means clustering for bag of words model using python
body: <p>The input dataset looks like this: </p>
  
  <pre><code>{"666": ["abc",
           "xyz"],
   "888": ["xxxo",
           "xxxo"], 
   "007": ["abc"]}  
  </code></pre>
  
  <p>We start by creating a bag-of-words model using the following function: </p>
  
  <pre><code>def associate_terms_with_user(unique_term_set, all_users_terms_dict):
  
      associated_value_return_dict = {}
  
      # consider the first user
      for user_id in all_users_terms_dict:
  
          # what terms *could* this user have possibly used
          this_user_zero_vector = []
  
          # this could be refactored somehow
          for term in  unique_term_set:
              this_user_zero_vector.extend('0')
  
          # what terms *did* this user use
          terms_belong_to_this_user = all_users_terms_dict.get(user_id)
  
          # let's start counting all the possible terms that this term in the personal
          # user list of words could correspond to... 
          global_term_element_index = 0
  
          # while this one term is in the range of all possible terms
          while global_term_element_index &lt; len(unique_term_set):
  
              # start counting the number of terms he used
              local_term_set_item_index = 0
  
              # if this one term he used is still in the range of terms he used, counting them one by one
              while local_term_set_item_index &lt; len(terms_belong_to_this_user):
  
                  # if this one user term is the same as this one global term
                  if list(unique_term_set)[global_term_element_index] == terms_belong_to_this_user[local_term_set_item_index]:
  
                      # increment the number of times this user used this term
                      this_user_zero_vector[global_term_element_index] = '1'
  
                  # go to the next term for this user
                  local_term_set_item_index += 1
  
              # go to the next term in the global list of all possible terms
              global_term_element_index += 1
  
          associated_value_return_dict.update({user_id: this_user_zero_vector})
  
      pprint.pprint(associated_value_return_dict)
  </code></pre>
  
  <p>The output of the program looks like this: </p>
  
  <pre><code>{'007': ['0', '0', '1'], 
   '666': ['0', '1', '1'], 
   '888': ['1', '0', '0']}
  </code></pre>
  
  <p>How could we implement a simple function to cluster those vectors based on their similarity to one another? I envisage using <a href="https://en.wikipedia.org/wiki/K-means_clustering" rel="nofollow noreferrer">k-means</a> and possibly scikit-learn. </p>
  
  <p>I've never done that before and I don't know how, I'm new to machine learning in generally and I don't really even know where to start. </p>
  
  <p>Finally <code>666</code> and <code>007</code> would probably be clustered together, and <code>888</code> would be alone in a cluster by itself, isn't it? </p>
  
  <p>The full code lives <a href="https://github.com/s-matthew-english/hackensack-ke-2017/blob/master/hashtag_clustering.py" rel="nofollow noreferrer">here</a>.</p>