title: tf-idf on a somewhat large (65k) amount of text files
body: <p>I want to try tfidf with scikit-learn (or nltk or am open to other suggestions).  The data I have is a relatively large amount of discussion forum posts (~65k) we have scraped and stored in a mongoDB.  Each post has a Post title, Date and Time of post, Text of the post message (or a re: if a reply to an existing post), User name, message ID and whether it is a child or parent post (in a thread, where you have the original post, and then replies to this op, or nested replies, the tree).</p>
  
  <p>I figure each post, would be a separate document, and similar to the 20newsgroups, each document would have the fields I mentioned at the top, and the text of the message post at the bottom which I would extract out of mongo and write into the required format for each text file.</p>
  
  <p>For loading the data into scikit, I know of:<br>
  <a href="http://scikit-learn.org/dev/modules/generated/sklearn.datasets.load_files.html" rel="nofollow">http://scikit-learn.org/dev/modules/generated/sklearn.datasets.load_files.html</a>  (but my data is not categorized)
  <a href="http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html" rel="nofollow">http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html</a> - For the input, I know I would be using filenames, but because I would have a large amount of files (each post), is there a way to either have filenames read from a text file? Or is there some example implementation someone could point me towards?</p>
  
  <p>Also, any advice on structuring the filenames for each these discussion forum posts, for later identifying when I get the tfidf vectors and cosine similarity array</p>
  
  <p>Thanks</p>