    def num_tokens(self, index):
        """Return the number of tokens in a sample. This value is used to
        enforce ``--max-tokens`` during batching."""
        raise NotImplementedError
Return the number of tokens in a sample. This value is used to
enforce ``--max-tokens`` during batching.
fairseq/data/fairseq_dataset.py