title: How is WordPiece tokenization helpful to effectively deal with rare words problem in NLP?
body: <p>I have seen that NLP models such as <a href="https://github.com/google-research/bert" rel="noreferrer">BERT</a> utilize WordPiece for tokenization. In WordPiece, we split the tokens like <strong><code>playing</code></strong> to <strong><code>play</code></strong> and <strong><code>##ing</code></strong>. It is mentioned that it covers a wider spectrum of Out-Of-Vocabulary (OOV) words. Can someone please help me explain how WordPiece tokenization is actually done, and how it handles effectively helps to rare/OOV words? </p>