original_docstring: 
          Tokenizes single sentence and adds special BOS and EOS tokens.
  
          :param line: sentence
  
          returns: list representing tokenized sentence
          
docstring: Tokenizes single sentence and adds special BOS and EOS tokens.
docstring_tokens: Tokenizes single sentence and adds special BOS and EOS tokens .
code: def segment(self, line):
          """
          Tokenizes single sentence and adds special BOS and EOS tokens.
  
          :param line: sentence
  
          returns: list representing tokenized sentence
          """
          line = line.strip().split()
          entry = [self.tok2idx[i] for i in line]
          entry = [wmt16_config.BOS] + entry + [wmt16_config.EOS]
          return entry
code_tokens: def segment ( self , line ) : line = line . strip ( ) . split ( ) entry = [ self . tok2idx [ i ] for i in line ] entry = [ wmt16_config . BOS ] + entry + [ wmt16_config . EOS ] return entry
short_docstring: Tokenizes single sentence and adds special BOS and EOS tokens.
short_docstring_tokens: Tokenizes single sentence and adds special BOS and EOS tokens .
comment: """
          Tokenizes single sentence and adds special BOS and EOS tokens.
  
          :param line: sentence
  
          returns: list representing tokenized sentence
          """