original_docstring: Tokenizes, mapping a python string to a sequence of indices.
docstring: Tokenizes, mapping a python string to a sequence of indices.
docstring_tokens: Tokenizes mapping a python string to a sequence of indices .
code: def string_to_indices(self,
                          string: str,
                          prepend_bos: bool = False,
                          append_eos: bool = False,
                          max_num_tokens: Optional[int] = 32) -> Sequence[int]:
      """Tokenizes, mapping a python string to a sequence of indices."""
      tokenized = self._sp_model.EncodeAsIds(string)
      tokenized = [self._bos_token] * prepend_bos + tokenized
      tokenized += [self._eos_token] * append_eos
      if max_num_tokens:
        tokenized = tokenized[:max_num_tokens]
        num_tokens = len(tokenized)
        tokenized = tokenized + [self._pad_token] * (max_num_tokens-num_tokens)
      return tokenized
code_tokens: def string_to_indices ( self , string : str , prepend_bos : bool = False , append_eos : bool = False , max_num_tokens : Optional [ int ] = 32 ) -> Sequence [ int ] : tokenized = self . _sp_model . EncodeAsIds ( string ) tokenized = [ self . _bos_token ] * prepend_bos + tokenized tokenized += [ self . _eos_token ] * append_eos if max_num_tokens : tokenized = tokenized [ : max_num_tokens ] num_tokens = len ( tokenized ) tokenized = tokenized + [ self . _pad_token ] * ( max_num_tokens - num_tokens ) return tokenized
short_docstring: Tokenizes, mapping a python string to a sequence of indices.
short_docstring_tokens: Tokenizes mapping a python string to a sequence of indices .
comment: """Tokenizes, mapping a python string to a sequence of indices."""