public static Word[] prepareStringToken(String str){
  if (str == null) {
    return new Word[0];
  }
  final ArrayList<Word> strWithDelim=new ArrayList<Word>();
  splitWordBySpace(str,strWithDelim);
  final Word[] toRet=new Word[Math.min(strWithDelim.size(),MAX_NUM_WORDS)];
  for (int i=0; i < Math.min(strWithDelim.size(),MAX_NUM_WORDS); i++) {
    toRet[i]=strWithDelim.get(i);
  }
  return toRet;
}

public static Word [ ] prepareStringToken ( String str ) { if ( str == null ) { return new Word [ 0 ] ; } final ArrayList < Word > strWithDelim = new ArrayList < Word > ( ) ; splitWordBySpace ( str , strWithDelim ) ; final Word [ ] toRet = new Word [ Math . min ( strWithDelim . size ( ) , MAX_NUM_WORDS ) ] ; for ( int i = 0 ; i < Math . min ( strWithDelim . size ( ) , MAX_NUM_WORDS ) ; i ++ ) { toRet [ i ] = strWithDelim . get ( i ) ; } return toRet ; }
split the string into an array of words separated by whitespace or specific delimiters . the input string should be all one token .
