<p>There is indeed a few recent developments that try to counteract this problem. The most notable ones are probably <em>subword units</em> (also known as Byte Pair Encodings, or BPEs), which you can imagine as a notion similar to syllables in a word (but not the same!); A word like <code>basketball</code> could then be transformed into variations like <code>bas @@ket @@ball</code> or <code>basket @@ball</code>. Note that this is a constructed example and might not reflect the actually chosen subwords.</p>

<p>The idea itself is <a href="https://dl.acm.org/doi/10.5555/177910.177914" rel="nofollow noreferrer">relatively old</a> (an article from 1994), but has been recently popularized by <a href="https://arxiv.org/pdf/1508.07909.pdf" rel="nofollow noreferrer">Sennrich et al.</a>, and is basically used in every state-of-the-art NLP library that has to deal with large vocabularies.</p>

<p>The two biggest implementations of this idea are probably <a href="https://github.com/glample/fastBPE" rel="nofollow noreferrer">fastBPE</a> and Google's <a href="https://github.com/google/sentencepiece" rel="nofollow noreferrer">SentencePiece</a>.</p>

<p>With subword units, you now basically have the freedom to determine a fix vocabulary size, and the algorithm will then try to optimize towards a mix of word diversity, and basically splitting "more complex words" into several pieces, such that your desired vocabulary size can cover any word in the corpus. For the exact algorithm, though, I highly recommend you to look into the linked paper or implementations.</p>
Are there good ways to reduce the size of a vocabulary in natural language processing?
<p>While working on tasks like text classification, QA, the original vocabulary generated from the corpus is usually too large, containing a lot of 'unimportant' words. The most popular ways I've seen to reduce the vocabulary size are discarding stop words and words with low frequencies.</p>

<p>For example, in <code>gensim</code></p>

<pre><code>gensim.utils.prune_vocab(vocab, min_reduce, trim_rule=None):
    Remove all entries from the vocab dictionary with count smaller than min_reduce.
    Modifies vocab in place, returns the sum of all counts that were pruned.
</code></pre>

<p>But in practice, setting the minimum count is empirical and does not seems quite exact. I notice that the term frequency of each word in the vocabulary often follows long-tail distribution, is it a good way if I only keep the top-K words that occupies X% (95%, 90%, 85%, ...) of the total term frequency? Or are there any sensible ways to reduce the vocabulary, without seriously influencing the NLP task? </p>