    def _get_token_idx(self, x):
        """
        This is to extend noising functions to be able to apply to non-bpe
        tokens, e.g. word or characters.
        """
        x = torch.t(x)
        word_idx = np.array([range(len(x_i)) for x_i in x])
        return np.transpose(word_idx)
This is to extend noising functions to be able to apply to non-bpe
tokens, e.g. word or characters.
fairseq/data/noising.py