def make_tokens(self, blob):
        # Kill the punctuation.
        blob = self.PUNCTUATION.sub(' ', blob)
        tokens = []

        # Split on spaces.
        for token in blob.split():
            # Make sure everything is in lowercase & whitespace removed.
            token = token.lower().strip()

            if not token in self.STOP_WORDS:
                tokens.append(token)

        return tokens
Given a string (``blob``) of text, this will return a list of tokens.
dataset/ETHPy150Open toastdriven/microsearch/microsearch.py/Microsearch.make_tokens