<SYSTEM_TASK:>
Solve the following problem using Python, implementing the functions described below, one line at a time
<END_TASK>
<USER_TASK:>
Description:
 def tokenize(self, string):
 """ Tokenizer utility. When processing byte, outputs the string unaltered. The character unit type is used for unicode data, the string is decoded according to the encoding provided. In the case of word unit, EOL characters are detached from the preceding word, and outputs the list of words, i.e. the list of non-space strings separated by space strings. True 10 True 10 ['miss', 'issi', 'ppi'] ['miss', 'issi', '\\n', 'ppi'] """

        if self.unit == UNIT_WORD:
            # the EOL character is treated as a word, hence a substitution
            # before split

            return [token for token in string.replace("\n", " \n ").split(self.tokSep) if token != ""]
        elif self.unit == UNIT_CHARACTER:
            return string.decode(self.encoding)
        else:
            return string