original_docstring: Removes special characters from text.
docstring: Removes special characters from text.
docstring_tokens: Removes special characters from text .
code: def remove_special_characters(text):
      """Removes special characters from text."""
  
      # If text is empty, return None.
      if not text: return None
      # If texts is tokenized, merge tokens.
      if is_tokenized(text):
          was_tokenized = True
          normalized_text = merge_tokens(text)
      else:
          was_tokenized = False
          normalized_text = text
      # Retrieve special characters pattern.
      special_characters_pattern = get_special_characters_pattern()
      # Remove all tailing white spaces.
      normalized_text = normalized_text.strip()
      # Replace all special characters with spaces.
      normalized_text = sub(special_characters_pattern, r' ', normalized_text)
      # Then remove multiple adjacent spaces.
      normalized_text = sub(' +', ' ', normalized_text)
      # Strip text.
      normalized_text = normalized_text.strip()
      # If text was tokenized, re-tokenize text.
      if was_tokenized:
          normalized_text = word_tokenize(normalized_text)
      # Return normalized text.
      return normalized_text
code_tokens: def remove_special_characters ( text ) : if not text : return None if is_tokenized ( text ) : was_tokenized = True normalized_text = merge_tokens ( text ) else : was_tokenized = False normalized_text = text special_characters_pattern = get_special_characters_pattern ( ) normalized_text = normalized_text . strip ( ) normalized_text = sub ( special_characters_pattern , r' ' , normalized_text ) normalized_text = sub ( ' +' , ' ' , normalized_text ) normalized_text = normalized_text . strip ( ) if was_tokenized : normalized_text = word_tokenize ( normalized_text ) return normalized_text
short_docstring: Removes special characters from text.
short_docstring_tokens: Removes special characters from text .
comment: """Removes special characters from text."""$SEP$# If text is empty, return None.$SEP$# If texts is tokenized, merge tokens.$SEP$# Retrieve special characters pattern.$SEP$# Remove all tailing white spaces.$SEP$# Replace all special characters with spaces.$SEP$# Then remove multiple adjacent spaces.$SEP$# Strip text.$SEP$# If text was tokenized, re-tokenize text.$SEP$# Return normalized text.