title: How to calculate readabilty in R with the tm package
body: <p>Is there a pre-built function for this in the <code>tm</code> library, or one that plays nicely with it?  </p>
  
  <p>My current corpus is loaded into tm, something like as follows:</p>
  
  <pre><code>s1 &lt;- "This is a long, informative document with real words and sentence structure:  introduction to teaching third-graders to read.  Vocabulary is key, as is a good book.  Excellent authors can be hard to find." 
  s2 &lt;- "This is a short jibberish lorem ipsum document.  Selling anything to strangers and get money!  Woody equal ask saw sir weeks aware decay. Entrance prospect removing we packages strictly is no smallest he. For hopes may chief get hours day rooms. Oh no turned behind polite piqued enough at. "
  stuff &lt;- rbind(s1,s2) 
  d &lt;- Corpus(VectorSource(stuff[,1]))
  </code></pre>
  
  <p>I tried using <a href="https://stackoverflow.com/questions/14835894/how-do-i-extract-contents-from-a-korpus-object-in-r">koRpus</a>, but it seems silly to retokenize in a different package than the one I'm already using. I also had problems vectorizing its return object in a way that would allow me to reincorporate the results into <code>tm</code>.  (Namely, due to errors, it would often return more or fewer readability scores than the number of documents in my collection.)</p>
  
  <p>I understand I could do a naive calculation parsing vowels as syllables, but want a more thorough package that takes care of the edge cases already (address silent e's, etc.).  </p>
  
  <p>My readability scores of choice are Flesch-Kincaid or Fry.</p>
  
  <p>What I had tried originally where d is my corpus of 100 documents:</p>
  
  <pre><code>f &lt;- function(x) tokenize(x, format="obj", lang='en')
  g &lt;- function(x) flesch.kincaid(x)
  x &lt;- foreach(i=1:length(d), .combine='c',.errorhandling='remove') %do% g(f(d[[i]]))
  </code></pre>
  
  <p>Unfortunately, x returns less than 100 documents, so I can't associate successes with the correct document.  (This is partly my misunderstanding of 'foreach' versus 'lapply' in R, but I found the structure of a text object sufficiently difficult that I could not appropriately tokenize, apply flesch.kincaid, and successfully check errors in a reasonable sequence of  apply statements.)</p>
  
  <p><strong>UPDATE</strong></p>
  
  <p>Two other things I've tried, trying to apply the koRpus functions to the tm object...</p>
  
  <ol>
  <li><p>Pass arguments into the tm_map object, using the default tokenizer:
  <code>tm_map(d,flesch.kincaid,force.lang="en",tagger=tokenize)</code></p></li>
  <li><p>Define a tokenizer, pass that in.</p>
  
  <pre><code> f &lt;- function(x) tokenize(x, format="obj", lang='en')
   tm_map(d,flesch.kincaid,force.lang="en",tagger=f)
  </code></pre></li>
  </ol>
  
  <p>Both of these returned:</p>
  
  <pre><code>   Error: Specified file cannot be found:
  </code></pre>
  
  <p>Then lists the full text of d[<a href="https://stackoverflow.com/questions/14835894/how-do-i-extract-contents-from-a-korpus-object-in-r">1</a>].  Seems to have found it?  What should I do to pass the function correctly?</p>
  
  <p><strong>UPDATE 2</strong></p>
  
  <p>Here's the error I get when I try to map koRpus functions directly with lapply:</p>
  
  <pre><code>&gt; lapply(d,tokenize,lang="en")
  Error: Unable to locate
   Introduction to teaching third-graders to read.  Vocabulary is key, as is a good book.  Excellent authors can be hard to find. 
  </code></pre>
  
  <p>This looks like a strange error---I almost don't think it means it can't locate the text, but that it can't locate some blank error code (such as, 'tokenizer'), before dumping the located text.</p>
  
  <p><strong>UPDATE 3</strong></p>
  
  <p>Another problem with retagging using <code>koRpus</code> was that retagging (versus the tm tagger) was extremely slow and output its tokenization progress to stdout.  Anyway, I've tried the following:</p>
  
  <pre><code>f &lt;- function(x) capture.output(tokenize(x, format="obj", lang='en'),file=NULL)
  g &lt;- function(x) flesch.kincaid(x)
  x &lt;- foreach(i=1:length(d), .combine='c',.errorhandling='pass') %do% g(f(d[[i]]))
  y &lt;- unlist(sapply(x,slot,"Flesch.Kincaid")["age",])
  </code></pre>
  
  <p>My intention here would be to rebind the <code>y</code> object above back to my <code>tm(d)</code> corpus as metadata, <code>meta(d, "F-KScore") &lt;- y</code>.</p>
  
  <p>Unfortunately, applied to my actual data set, I get the error message:</p>
  
  <pre><code>Error in FUN(X[[1L]], ...) : 
    cannot get a slot ("Flesch.Kincaid") from an object of type "character"
  </code></pre>
  
  <p>I think one element of my actual corpus must be an NA, or too long, something else prohibitive---and due to the nested functionalizing, I am having trouble tracking down exactly which it is.  </p>
  
  <p>So, currently, it looks like there is no pre-built function for reading scores that play nicely with the <code>tm</code> library.  Unless someone sees an easy error-catching solution I could sandwich into my function calls to deal with inability to tokenize some apparently erroneous, malformed documents?  </p>