<SYSTEM_TASK:>
Solve the following problem using Python, implementing the functions described below, one line at a time
<END_TASK>
<USER_TASK:>
Description:
 def _tokenize(self, stream):
 """ Tokenizes data from the provided string. ``stream`` ``File``-like object. """


        self._tokens = []
        self._reset_token()
        self._state = self.ST_TOKEN

        for chunk in iter(lambda: stream.read(8192), ''):
            for char in chunk:
                if char in self.NEWLINES:
                    self._process_newline(char)

                else:
                    state = self._state

                    if state == self.ST_STRING:
                        self._process_string(char)

                    elif state == self.ST_TOKEN:
                        self._process_tokens(char)