title: How to identify multi-word expression in a sentence without a predefined multi-word list
body: <p>I am trying to identify every multi-word expression in a sentence and tokenize that sentence. For instance, the example input sentence is &quot;In short, this merchandise is in short supply.&quot; and I wish the output could be shown as below:</p>
  <pre><code>['In short', ',', 'this', 'merchandise', 'is', 'in short supply', '.']
  </code></pre>
  <p>I have already achieved the aforesaid result by using a predefined list and the following python code.</p>
  <pre><code>from nltk import sent_tokenize, word_tokenize
  from nltk.tokenize import MWETokenizer
  
  multiwordExpressionList = [(&quot;In&quot;, &quot;short&quot;), (&quot;in&quot;, &quot;short&quot;, &quot;supply&quot; )]  ## this is a predefined list 
  sentence = &quot;In short, this merchandise is in short supply.&quot;
  
  mwe = MWETokenizer(multiwordExpressionList, separator = ' ')
  resultList = mwe.tokenize(word_tokenize(sentence))
  print(resultList)
  </code></pre>
  <p>However, the drawback is quite clear. This program need a predefined multi-word expression list to identify whether any multi-word expression exist in a sentence. Is there any suggested python package, module or method can identify any multi-word expression exist in a sentence ?</p>