title: Clustering script fails with German, but works like expected with English
body: <p>I have a script to cluster keywords, utilizing pandas and <a href="https://maartengr.github.io/PolyFuzz/" rel="nofollow noreferrer">polyfuzz</a>. With English, it works like expected. Trying to use the script with keywords in German, the script recognizes multiple keywords wrongly.</p>
  <p>What means &quot;wrongly recognized&quot;: clustering recognizes the first and second word in the keyword. And as you can see on the screenshot, columns G and H (First Word and Second Word) contain other words, then corresponding keywords in column B (Keyword):</p>
  <p><a href="https://i.stack.imgur.com/XTHwr.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/XTHwr.png" alt="enter image description here" /></a></p>
  <p>The script fails not always with German - multiple keywords are clustered correctly. But the part of wrongly recognized keywords is very high, up to 20%.</p>
  <p>Could somebody explain to me why the script failed with German keywords and, in the best case, improve the script enabling it to work with German?</p>
  <p>Here is the part of the script, which does clustering:</p>
  <pre><code># find keywords from one column in another in any order and count the frequency
  df_matched['Cluster Name'] = df_matched['Cluster Name'].str.strip()
  df_matched['Keyword'] = df_matched['Keyword'].str.strip()
  
  df_matched['First Word'] = df_matched['Cluster Name'].str.split(&quot; &quot;).str[0] 
  df_matched['Second Word'] = df_matched['Cluster Name'].str.split(&quot; &quot;).str[1] 
  df_matched['Total Keywords'] = df_matched['First Word'].str.count(' ') + 1
  
  def ismatch(s):
      A = set(s[&quot;First Word&quot;].split())
      B = set(s['Keyword'].split())
      return A.intersection(B) == A
  
  df_matched['Found'] = df_matched.apply(ismatch, axis=1)
  
  df_matched = df_matched. fillna('')
  
  def ismatch(s):
      A = set(s[&quot;Second Word&quot;].split())
      B = set(s['Keyword'].split())
      return A.intersection(B) == A
  df_matched['Found 2'] = df_matched.apply(ismatch, axis=1)
  
  
  # todo - document this algo. Essentially if it matches on the second word only, it renames the cluster to the second word
  # clean up code nd variable names
  df_matched.loc[(df_matched[&quot;Found&quot;] == False) &amp; (df_matched[&quot;Found 2&quot;] == True), &quot;Cluster Name&quot;] = df_matched[&quot;Second Word&quot;]
  df_matched.loc[(df_matched[&quot;Found&quot;] == False) &amp; (df_matched[&quot;Found 2&quot;] == False), &quot;Cluster Name&quot;] = &quot;zzz_no_cluster_available&quot;
  
  # count cluster_size
  df_matched['Cluster Size'] = df_matched['Cluster Name'].map(df_matched.groupby('Cluster Name')['Cluster Name'].count())
  df_matched.loc[df_matched[&quot;Cluster Size&quot;] == 1, &quot;Cluster Name&quot;] = &quot;zzz_no_cluster_available&quot;
  
  
  df_matched = df_matched.sort_values(by=&quot;Cluster Name&quot;, ascending=True)
  </code></pre>
  <p>Here are two datasets:</p>
  <ul>
  <li><p>Working dataset in English: <a href="http://dl.dropboxusercontent.com/s/zrobh2x4bs3ztlf/working-dataset-english.txt" rel="nofollow noreferrer">http://dl.dropboxusercontent.com/s/zrobh2x4bs3ztlf/working-dataset-english.txt</a></p>
  </li>
  <li><p>Badly working dataset in German: <a href="http://dl.dropboxusercontent.com/s/i1p3j3zi1t0cev3/badly-working-dataset-german.txt" rel="nofollow noreferrer">http://dl.dropboxusercontent.com/s/i1p3j3zi1t0cev3/badly-working-dataset-german.txt</a></p>
  </li>
  </ul>
  <p>And here, the <a href="https://colab.research.google.com/drive/1nI7l14teSDanZMSltAumITBaQL98_w35?usp=sharing" rel="nofollow noreferrer">working Colab</a> with the whole script.</p>