def subsample_frequent(self, words_compact, threshold=1e-5):
        self._check_finalized()
        freq = self.keys_frequency + 1e-10
        pw = 1.0 - (np.sqrt(threshold / freq) + threshold / freq)
        prob = fast_replace(words_compact, self.keys_compact, pw)
        draw = np.random.uniform(size=prob.shape)
        ret = words_compact.copy()
        # If probability greater than draw, skip the word
        ret[prob > draw] = self.specials_to_compact['skip']
        return ret
Subsample the most frequent words. This aggressively
dataset/ETHPy150Open cemoody/lda2vec/lda2vec/corpus.py/Corpus.subsample_frequent