lib/two_chainz/generator.rb
# Internal: Remove unwanted tokens from the specified text.
#
# text - Text to remove unwanted tokens from.
#
# NOTE: This should prolly be refactored into a Sanitizer object with options
# and its own tests and stuff. If this method starts getting big, do that.
#
# ANOTHER NOTE: This hurts accuracy a little cuz it glues together two words
# that did not originally follow each other directly. Might be worth breaking
# up the string into multiple #hears on removed tokens or some other way of
# decreasing the importance of a pair made in this way. Honestly though, it
# probably doesn't matter that much, so don't spend a ton of time on it.
#
# Returns a string.
Remove unwanted tokens from the specified text.
text - Text to remove unwanted tokens from.

This should prolly be refactored into a Sanitizer object with options
and its own tests and stuff. If this method starts getting big, do that.

ANOTHER NOTE: This hurts accuracy a little cuz it glues together two words
that did not originally follow each other directly. Might be worth breaking
up the string into multiple #hears on removed tokens or some other way of
decreasing the importance of a pair made in this way. Honestly though, it
probably doesn't matter that much, so don't spend a ton of time on it.

Returns a string.
Remove unwanted tokens from the specified text . text - Text to remove unwanted tokens from . This should prolly be refactored into a Sanitizer object with options and its own tests and stuff . If this method starts getting big do that . ANOTHER NOTE : This hurts accuracy a little cuz it glues together two words that did not originally follow each other directly . Might be worth breaking up the string into multiple #hears on removed tokens or some other way of decreasing the importance of a pair made in this way . Honestly though it probably doesn ' t matter that much so don ' t spend a ton of time on it . Returns a string .
def sanitize(text)
    sanitized_text = text.dup
    sanitized_text.gsub!(URL_REGEX, '')
    sanitized_text.gsub!(MENTION_REGEX, '')
    sanitized_text
  end
def sanitize ( text ) sanitized_text = text . dup sanitized_text . gsub! ( URL_REGEX , '' ) sanitized_text . gsub! ( MENTION_REGEX , '' ) sanitized_text end
Internal: Remove unwanted tokens from the specified text.
Internal : Remove unwanted tokens from the specified text .
# Strip URLs # Strip @mention style tokens
text