    def num_tokens(self, index):
        """Return the number of tokens in a sample. This value is used to
        enforce ``--max-tokens`` during batching."""
        return max(
            self.src_sizes[index],
            self.tgt_sizes[index] if self.tgt_sizes is not None else 0,
        )
Return the number of tokens in a sample. This value is used to
enforce ``--max-tokens`` during batching.
fairseq/data/language_pair_dataset.py