title: Tokenize Sentences or Tweets with Emoji Skin Tone Modifiers
body: <p>I want to tokenize a tweet containing multiple emojis and they are not space-separated. I tried both <code>NLTK TweetTokenizer</code> and <code>Spacy</code> but they fail to tokenize Emoji Skin Tone Modifiers. This needs to be applied to a huge dataset so performance might be an issue. Any suggestions?</p>
  <p><strong>You may need to use Firefox or Safari to see the exact color tone emoji because Chrome sometimes fails to render it!</strong></p>
  <pre><code># NLTK
  from nltk.tokenize.casual import TweetTokenizer
  sentence = &quot;I'm the most famous emoji  but what about  and &quot;
  t = TweetTokenizer()
  print(t.tokenize(sentence))
  
  # Output
  [&quot;I'm&quot;, 'the', 'most', 'famous', 'emoji', '', '', '', 'but', 'what', 'about', '', 'and', '', '', '', '', '', '']
  </code></pre>
  <p>And</p>
  <pre><code># Spacy
  import spacy
  nlp = spacy.load(&quot;en_core_web_sm&quot;)
  sentence = nlp(&quot;I'm the most famous emoji  but what about  and &quot;)
  print([token.text for token in sentence])
  
  Output
  ['I', &quot;'m&quot;, 'the', 'most', 'famous', 'emoji', '', '', '', 'but', 'what', 'about', '', 'and', '', '', '', '', '', '']
  </code></pre>
  <p>Expected Output</p>
  <pre><code>[&quot;I'm&quot;, 'the', 'most', 'famous', 'emoji', '', '', '', 'but', 'what', 'about', '', 'and', '', '', '', '']
  </code></pre>