original_docstring: Tokenizes input text, mapping a tensor of strings to a tensor of ints.
  
      Args:
        string_tensor: Input string tensor of shape [num_texts].
        prepend_bos: Whether to prepend the BOS (beginning of sentence) token to
          the output tokens.
        append_eos: Whether to append the EOS (end of sentence) token to the
          output tokens.
        max_num_tokens: Maximum number of tokens to return per caption. If
          provided, the tokens will be paded / cut at a the given size. If not, a
          tensor of unknown size will be returned.
  
      Returns:
        A `tf.int32` tensor of shape [num_texts, `max_num_tokens`] if
          `max_num_tokens` is provided or [num_texts, max_num_tokens_in_batch]
          otherwise.
      
docstring: Tokenizes input text, mapping a tensor of strings to a tensor of ints.
docstring_tokens: Tokenizes input text mapping a tensor of strings to a tensor of ints .
code: def string_tensor_to_indices(self,
                                 string_tensor: tf.Tensor,
                                 prepend_bos: bool = False,
                                 append_eos: bool = False,
                                 max_num_tokens: Optional[int] = 32) -> tf.Tensor:
      """Tokenizes input text, mapping a tensor of strings to a tensor of ints.
  
      Args:
        string_tensor: Input string tensor of shape [num_texts].
        prepend_bos: Whether to prepend the BOS (beginning of sentence) token to
          the output tokens.
        append_eos: Whether to append the EOS (end of sentence) token to the
          output tokens.
        max_num_tokens: Maximum number of tokens to return per caption. If
          provided, the tokens will be paded / cut at a the given size. If not, a
          tensor of unknown size will be returned.
  
      Returns:
        A `tf.int32` tensor of shape [num_texts, `max_num_tokens`] if
          `max_num_tokens` is provided or [num_texts, max_num_tokens_in_batch]
          otherwise.
      """
code_tokens: def string_tensor_to_indices ( self , string_tensor : tf . Tensor , prepend_bos : bool = False , append_eos : bool = False , max_num_tokens : Optional [ int ] = 32 ) -> tf . Tensor :
short_docstring: Tokenizes input text, mapping a tensor of strings to a tensor of ints.
short_docstring_tokens: Tokenizes input text mapping a tensor of strings to a tensor of ints .
comment: """Tokenizes input text, mapping a tensor of strings to a tensor of ints.
  
      Args:
        string_tensor: Input string tensor of shape [num_texts].
        prepend_bos: Whether to prepend the BOS (beginning of sentence) token to
          the output tokens.
        append_eos: Whether to append the EOS (end of sentence) token to the
          output tokens.
        max_num_tokens: Maximum number of tokens to return per caption. If
          provided, the tokens will be paded / cut at a the given size. If not, a
          tensor of unknown size will be returned.
  
      Returns:
        A `tf.int32` tensor of shape [num_texts, `max_num_tokens`] if
          `max_num_tokens` is provided or [num_texts, max_num_tokens_in_batch]
          otherwise.
      """