title: Paragraph Segmentation using Machine Learning
body: <p>I have a large repository of documents in PDF format. The documents come from different sources, and have no one single style. I use <a href="https://tika.apache.org/" rel="noreferrer">Tika</a> to extract the text from the documents, and now I'd like to segment the text into paragraphs.</p>
  
  <p>I can't use regexes, because the documents have no single style:</p>
  
  <ul>
  <li>The number of <code>\nl</code> between paragraphs vary between 2 and 4.</li>
  <li>In some documents the lines within a single paragraph are separated by 2 <code>\nl</code>, some with single <code>\nl</code>.</li>
  </ul>
  
  <p>So I turn to machine learning. In the (great) Python NLTK book there's an excellent use of classification for <a href="http://www.nltk.org/book/ch06.html#sentence-segmentation" rel="noreferrer">segmentation of sentences</a> using attributes like characters before and after a '.' with a Bayesian network, but no paragraph segmentation.</p>
  
  <p>So my questions are:</p>
  
  <ul>
  <li>Is there another way for paragraph segmentation?</li>
  <li>If I go with machine learning, is there tagged data of segmented paragraphs I can use for training?</li>
  </ul>