Converts a space-separated string of tokens to lists of ids.

    Also store temporary vocabulary IDs for source OOV tokens. OOVs are
    represented by their temporary OOV number. E.g., if the vocabulary size
    is 50k and the source has 3 OOVs, then these temporary OOV numbers will
    be 50000, 50001, 50002.

    Args:
      s: human-readable string to be converted.

    Returns:
      ids: list of integers
      ids_extend: list of integers including extended temporary vocab IDs for
      source OOVs.
      oovs: A dict storing source OOV words, used for the decoder to copy. The
      key is OOV word, and the value is the order they appear in the source,
      starting from 0.
      source_oov_id_to_token: a list of source OOV tokens, in the same order as
      they appear in the source.

def encode(self, s):
    """Converts a space-separated string of tokens to lists of ids.

    Also store temporary vocabulary IDs for source OOV tokens. OOVs are
    represented by their temporary OOV number. E.g., if the vocabulary size
    is 50k and the source has 3 OOVs, then these temporary OOV numbers will
    be 50000, 50001, 50002.

    Args:
      s: human-readable string to be converted.

    Returns:
      ids: list of integers
      ids_extend: list of integers including extended temporary vocab IDs for
      source OOVs.
      oovs: A dict storing source OOV words, used for the decoder to copy. The
      key is OOV word, and the value is the order they appear in the source,
      starting from 0.
      source_oov_id_to_token: a list of source OOV tokens, in the same order as
      they appear in the source.
    """
    sentence = s
    tokens = sentence.strip().split()
    ids = []
    ids_extend = []
    oovs = {}
    for t in tokens:
      if t in self._token_to_id:
        ids.append(self._token_to_id[t])
        ids_extend.append(self._token_to_id[t])
      else:
        next_oov_id = len(oovs)
        oov_num = oovs.get(t, next_oov_id)
        if oov_num == next_oov_id:
          oovs[t] = oov_num
        ids_extend.append(self.vocab_size + oov_num)
        ids.append(self._token_to_id[self._replace_oov])
    source_oov_id_to_token = [""] * len(oovs)
    for oov in oovs:
      source_oov_id_to_token[oovs[oov]] = oov
    if self._reverse:
      return ids[::-1], ids_extend[::-1], oovs, source_oov_id_to_token
    else:
      return ids, ids_extend, oovs, source_oov_id_to_token