title: Training word2vec model streaming data from file and tokenize to sentence
body: <p>I need to process a large number of <code>txt</code> files for building a <code>word2vec</code> model.
  Now, my txt-files are a bit messy and I need to remove all ´<code>\n</code>´ newlines, read all sentences from my loaded string (txt-file) and then tokenize each sentence for using the word2vec model.</p>
  
  <p>The thing is: I cant read the files line-by-line, cause some sentences do not end after one line. Therefore, I use ´<code>nltk.tokenizer.tokenize()</code>´, which splits the file into sentences.</p>
  
  <blockquote>
    <p>I cant figure out, how to convert a list of strings into a list of list, where each sub-list contains the sentences, while passing it thourgh a generator.</p>
  </blockquote>
  
  <p>Or do I actually need to save each sentences into a new file (one sentence per line) to pass it through a generator?</p>
  
  <p>Well, my code looks like this:
  ´<code>tokenizer = nltk.data.load('tokenizers/punkt/english.pickle')</code></p>
  
  <pre><code># initialize tokenizer for processing sentences
  
  class Raw_Sentences(object):
      def __init__(self, dirname):
          self.dirname = dirname
  
      def __iter__(self):
          for file in file_loads: ## Note: file_loads includes directory name of files (e.g. 'C:/Users/text-file1.txt')
              with open(file,'r', encoding='utf-8') as t:     
                 # print(tokenizer.tokenize(t.read().replace('\n', ' ')))           
                  storage = tokenizer.tokenize(t.read().replace('\n', ' '))
  # I tried to temporary store the list of sentences to a list for an iteration
                  for sentence in storage:
                      print(nltk.word_tokenize(sentence))
                      yield nltk.word_tokenize(sentence)´
  </code></pre>
  
  <p>So the goal is: 
  load file 1: ´<code>'some messy text here. And another sentence'</code>´
   tokenize into sentences ´<code>['some messy text here','And another sentence']</code>´
  and then split each sentence into words ´<code>[['some','messy','text','here'],['And','another','sentence']]</code>´</p>
  
  <p>load file 2: <code>'some other messy text. sentence1. sentence2.'</code>
  etc.</p>
  
  <p>and input sentences into word2vec model:
  ´<code>sentences = Raw_Sentences(directory)</code>´</p>
  
  <p>´<code>model = gensim.models.Word2Vec(sentences)</code>´</p>