title: Clustering brands using words embeddings
body: <p>Ok, so the title might sound a bit confusing, but here's an analogy of what I'm trying to achieve. Let's imagine that we have the following dataset:</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th style="text-align: left;">Brand name</th>
  <th style="text-align: center;">Product type</th>
  <th style="text-align: right;">Product_Description</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td style="text-align: left;">Nike</td>
  <td style="text-align: center;">Shoes</td>
  <td style="text-align: right;">These black shoes are wonderful. They are elegant, and really comfortable</td>
  </tr>
  <tr>
  <td style="text-align: left;">BMW</td>
  <td style="text-align: center;">Car</td>
  <td style="text-align: right;">This car goes fast. If you like speed, you'll like it.</td>
  </tr>
  <tr>
  <td style="text-align: left;">Suzuki</td>
  <td style="text-align: center;">Car</td>
  <td style="text-align: right;">A family car, elegant and made for long journeys.</td>
  </tr>
  <tr>
  <td style="text-align: left;">Call of Duty</td>
  <td style="text-align: center;">VideoGame</td>
  <td style="text-align: right;">A nervous shooter, putting you in the shoes of a desperate soldier, who has nothing left to lose.</td>
  </tr>
  <tr>
  <td style="text-align: left;">Adidas</td>
  <td style="text-align: center;">Shoes</td>
  <td style="text-align: right;">Sneakers made for men, and women, who always want to go out with style.</td>
  </tr>
  </tbody>
  </table>
  </div>
  <p>This is just a made-up sample, but let's imagine this list goes on for a lot of other products.</p>
  <p>What I'm trying to achieve here, is to cluster the elements (whether it is shoes, cars, or videogames), based on the words used in their respective description. Thus, I would obtain brands that are clustered together according to their description, but perhaps not belonging to the same type (e.g.: Suzuki + Adidas), and to get the name of the brands that are clustered together.</p>
  <p>To do so, I relied on a word embedding method. After cleaning the description (stop words, non-alphanumeric characters) and tokenized it, I used a FastText model (the Wikipedia one) to evaluate the embeddings in the product descriptions.</p>
  <pre><code>def clean_text(text, tokenizer, stopwords):
      text = str(text).lower()  # Lowercase words
      text = re.sub(r&quot;\[(.*?)\]&quot;, &quot;&quot;, text)  # Remove [+XYZ chars] in content
      text = re.sub(r&quot;\s+&quot;, &quot; &quot;, text)  # Remove multiple spaces in content
      text = re.sub(r&quot;\w+…|…&quot;, &quot;&quot;, text)  # Remove ellipsis (and last word)
      text = re.sub(r&quot;&lt;a[^&gt;]*&gt;(.*?)&lt;/a&gt;&quot;, r&quot;\1&quot;, text) #Remove html tags
      text = re.sub(f&quot;[{re.escape(punctuation)}]&quot;, &quot;&quot;, text)
      text = re.sub(r&quot;(?&lt;=\w)-(?=\w)&quot;, &quot; &quot;, text)  # Replace dash between words
      text = re.sub(
          f&quot;[{re.escape(string.punctuation)}]&quot;, &quot;&quot;, text
      )  # Remove punctuation
      doc = nlp_model(text)
      tokens = [token.lemma_ for token in doc]
      #tokens = tokenizer(text)  # Get tokens from text
      tokens = [t for t in tokens if not t in stopwords]  # Remove stopwords
      tokens = [&quot;&quot; if t.isdigit() else t for t in tokens]  # Remove digits
      tokens = [t for t in tokens if len(t) &gt; 1]  # Remove short tokens
      return tokens #Clean the Text
  
  def sent_vectorizer(sent, model): 
      sent_vec =[]
      numw = 0
      for w in sent:
          try:
              if numw == 0:
                  sent_vec = model[w]
              else:
                  sent_vec = np.add(sent_vec, model[w])
              numw+=1
          except:
              pass
  
      return np.asarray(sent_vec) / numw
  
  df = pd.read_csv(&quot;./mockup.csv&quot;)
  custom_stopwords = set(stopwords.words(&quot;english&quot;))
  df[&quot;Product_Description&quot;] = df[&quot;Product_Description&quot;].fillna(&quot;&quot;)
  df[&quot;tokens&quot;] = df[&quot;Product_Description&quot;].map(lambda x: clean_text(x, word_tokenize, custom_stopwords))
  model = KeyedVectors.load_word2vec_format('./wiki-news-300d-1M.vec')
  </code></pre>
  <p>The problem is that I'm a bit of a beginner in word embeddings and clustering. As I said, my goal would be to cluster brands according to the words used in their description (the hypothesis is perhaps some brands are linked together through the words used in their description?), thus forgoing the old classification (shoes, cars, videogames...). I would also like to get the key brands of each cluster (so cluster 1 = Suzuki + Adidas, Cluster 2 = Call of Duty + Nike, Cluster 3 = BMW + ..., etc...).</p>
  <p>Does anyone have any ideas on how to tackle this problem? I read several tutorials online on word embeddings and clustering, and to be completely honest, I am a bit lost.</p>
  <p>Thank you for your help.</p>