title: Word, PDF document parsing - Hadoop/in-general Java
body: <p>My objective is to load MS-Word, PDF etc. documents onto HDFS and extract certain 'content' out of each document and use it further for some analysis.</p>
  
  <p>Instead of beginning to fiddle with InputFormat etc., I thought that libraries like Tika can be used and incorporated in MR.</p>
  
  <p>The partial content of one of the Word doc. is as follows:</p>
  
  <pre><code>6.  Statement of Strategy 
  We have 4 strategic interventions that will deliver a competitive advantage.
   Innovate upstream and downstream
  1.  Biopulp.
  We will execute Biopulp initially in corrugate for Haircare in China. This will validate the operational process of enzymatically converting straw into pulp and paper. Then we will establish a Joint Development with Family care to extend the sources of value. And finally re-apply globally including across for other sectors and customers to maximize the value generation.
  2.  Mandrel Case Forming
  We will extend the use of MCF technology within WE for the businesses that already use MCF cases. (i.e. F&amp;HC). In parallel we will establish this as the global standard for HDL’s and HDW. We will seek additional suppliers to execute this technology in other regions (e.g. NA and Asia) to increase capacity and reduce cost of execution.
  
  Supplier Strategy for Competition
  3.  Competition in practice 
  We have used negotiation as the primary process for establishing prices and supply agreements. We will more effectively create and utilize competition by using enquiries for each of our plants. This may require that we trigger new investments and qualify additional facilities, but with the consolidation going on in the industry it should not cause a net increase in suppliers.
  4.  Cost input pass-through
  Our current agreements in general use paper as the primary driver of our feedstock clauses. If paper prices go up then our suppliers are happy and we are not. If paper prices go down then we are happy and our suppliers’ are not. This means that almost 100% of the time one party is not happy. If we change our pass-through clauses to be driven by our suppliers’ input costs, then we align ourselves with their interests which will generate less transaction cost and increase collaboration
  
  Optimum Sourcing Principles for Corrugates
  
  &lt;A TABLE HERE&gt;
  
  7.  Tactical Planning and Execution
  
  &lt;A TABLE HERE&gt;
  </code></pre>
  
  <p>Suppose, I wish to do the following :</p>
  
  <ol>
  <li>Extract the table under 'Optimum Sourcing Principles for Corrugates'</li>
  <li>The bullet points under 'Innovate upstream and downstream'</li>
  </ol>
  
  <p>While this seems crazy and absurd, I was wondering if Tika(tried this but stuck with just metadata and the file as string), Lucene/Solr, POI etc. can help to parse and 'understand' the Word, PDF documents and allow to extract a block of data based on some search string(or regex).</p>
  
  <p>For example, I used Tika Parser and got the following output which is too naive('A TABLE HERE' i.e a table in the Word doc. interpreted as paragraphs !) to help in content extraction :</p>
  
  <pre><code>6.  Statement of Strategy 
  We have 4 strategic interventions that will deliver a competitive advantage to P&amp;G.
   Innovate upstream and downstream
  Biopulp.
  We will execute Biopulp initially in corrugate for Haircare in China. This will validate the operational process of enzymatically converting straw into pulp and paper. Then we will establish a Joint Development with Family care to extend the sources of value. And finally re-apply globally including across for other sectors and customers to maximize the value generation.
  Mandrel Case Forming
  We will extend the use of MCF technology within WE for the businesses that already use MCF cases. (i.e. F&amp;HC). In parallel we will establish this as the global standard for HDL’s and HDW. We will seek additional suppliers to execute this technology in other regions (e.g. NA and Asia) to increase capacity and reduce cost of execution.
  
  Supplier Strategy for Competition
  Competition in practice 
  We have used negotiation as the primary process for establishing prices and supply agreements. We will more effectively create and utilize competition by using enquiries for each of our plants. This may require that we trigger new investments and qualify additional facilities, but with the consolidation going on in the industry it should not cause a net increase in suppliers.
  Cost input pass-through
  Our current agreements in general use paper as the primary driver of our feedstock clauses. If paper prices go up then our suppliers are happy and we are not. If paper prices go down then we are happy and our suppliers’ are not. This means that almost 100% of the time one party is not happy. If we change our pass-through clauses to be driven by our suppliers’ input costs, then we align ourselves with their interests which will generate less transaction cost and increase collaboration.
  
  
  
  
  
  Optimum Sourcing Principles for Corrugates
      principle
      optimum
      rationale
  
      Number of  suppliers
      2-3 per plant
  &gt;80% with 5 per region/country cluster
      Competition is local
  Scale the spend with central accounts
  
      Global/local suppliers
      Regional is sufficient
      No advantage to global as scale is regional only and there is limited IP to transfer.
  Larger regional suppliers can consolidate local single-plant suppliers to make it efficient for us. They also bring capital for machinery upgrading and scale for paper source.
  
      Approach to suppliers
      collaborative
      Competition to drive price is clear; preferential and value-add deals require collaboration
  
      Make v buy
      buy
      Multiple suppliers; commoditised technologies
  
      Distance of suppliers to plant
      Max 300km for boxes (300miles in NA); up to 1000km for paper reels.
  Can be longer for specialist print grades or to countries with no high quality local supply
      Economic max as high volume product (air in the fluting)
  Need recent built paper machines to produce paper strong enough to run on high-speed corrugators
  
      Type of suppliers
      Integrated with containerboard making
  
  Corrugators on-site
      To assure supply and avoid being leveraged by paper making scale
  Cost structure not competitive if have to buy in board (shipping air)
  
      Purchase of feedstocks
      Not if integrated suppliers
      Integrated suppliers have 20x our scale
  
      Length and nature of contracts
      Multiple year (2-3), but with fixed glidepath pricing/value every year
      Significant effort for Purchases to re-enquire annually. High number of specs and low resources mean long time to qualify relative to additional value if only 12 month allocation.
  
      Specifications
      Standard board weights
  
  
  Tailored box sizes
      Paper scale much higher so uneconomic to make tailored weight
  Maximising pallet fit delivers better savings and stronger pallet (less transport damages) than scale savings of standard box size.
  
      Terms
      Standard, including payment terms
      High degree of competition, no specialist investment. Paper making has good cash-flow, so no need for shorter payment terms.
  </code></pre>
  
  <p>Below is the sample <strong>TIKA code</strong> I wrote(I couldn't figure out how to do the above when different types(pdf, MS-Word etc.) of documents arrive</p>
  
  <pre><code>private void parseFileForContent(String absolutePath) throws IOException,
              SAXException, TikaException {
          // TODO Auto-generated method stub
  
          System.out.println("absolutePath : " + absolutePath);
  
          Tika tika = new Tika();
  
          File path = new File(absolutePath);
  
          if (path.isDirectory()) {
  
              File[] files = path.listFiles();
  
              for (File file : files) {
  
                  System.out.println("File type is " + tika.detect(file));
              }
          } else {
              System.out.println("File type is " + tika.detect(path));
  
              Parser parser = new AutoDetectParser();
  
              ContentHandler handler = new BodyContentHandler();
              Metadata metadata = new Metadata();
  
              parser.parse(TikaInputStream.get(path), handler, metadata,
                      new ParseContext());
  
              //displayMetadata(metadata);
  
              System.out.println("Handler "+handler.toString());
          }
  
      }
  </code></pre>
  
  <p><strong>I wish to use Tika as Apache POI is confined to MS documents but I could do something sensible with POI like extracting paragraphs, tables etc.</strong> </p>
  
  <pre><code>package com.lnt.sap.sp2.scratchpad;
  
  import java.io.FileInputStream;
  import java.io.FileNotFoundException;
  import java.io.IOException;
  import java.util.Iterator;
  import java.util.List;
  
  import org.apache.poi.xwpf.usermodel.IBodyElement;
  import org.apache.poi.xwpf.usermodel.XWPFDocument;
  import org.apache.poi.xwpf.usermodel.XWPFParagraph;
  import org.apache.poi.xwpf.usermodel.XWPFTable;
  import org.apache.poi.xwpf.usermodel.XWPFTableCell;
  import org.apache.poi.xwpf.usermodel.XWPFTableRow;
  
  public class POIScratchpad {
  
      public static void main(String[] args) {
          // TODO Auto-generated method stub
  
          String absolutePath = args[0];
  
          POIScratchpad poiScratchpad = new POIScratchpad();
  
          poiScratchpad.parseMSDocuments(absolutePath);
      }
  
      private void parseMSDocuments(String absolutePath) {
          // TODO Auto-generated method stub
  
          try {
  
              XWPFDocument doc = new XWPFDocument(new FileInputStream(
                      absolutePath));
  
              displayElements(doc);
              // displayParagraphs(doc);
              // displayTables(doc);
  
          } catch (FileNotFoundException e) {
              // TODO Auto-generated catch block
              e.printStackTrace();
          } catch (IOException e) {
              // TODO Auto-generated catch block
              e.printStackTrace();
          }
      }
  
      private void displayElements(XWPFDocument doc) {
          // TODO Auto-generated method stub
  
          java.util.Iterator&lt;IBodyElement&gt; bodyElementIterator = doc
                  .getBodyElementsIterator();
  
          int cnt = 0;
  
          while (bodyElementIterator.hasNext()) {
              IBodyElement element = bodyElementIterator.next();
  
              System.out.println("**********" + cnt + "**********");
  
              System.out.println("Element type is " + element.getElementType());
              System.out.println("Part is : " + element.getPart());
              System.out.println("Part Type is : " + element.getPartType());
              System.out.println("Body is : " + element.getBody());
              System.out.println("element is " + element);
  
              System.out.println("**********");
  
              cnt++;
          }
      }
  
      private void displayParagraphs(XWPFDocument doc) {
          // TODO Auto-generated method stub
          List&lt;XWPFParagraph&gt; paragraphs = doc.getParagraphs();
  
          int cnt = 0;
  
          for (XWPFParagraph paragraph : paragraphs) {
  
              System.out.println("**********" + cnt + "**********");
              System.out.println(paragraph.getParagraphText());
              System.out.println("********************");
  
              cnt++;
          }
      }
  
      private void displayTables(XWPFDocument doc) {
          // TODO Auto-generated method stub
  
          Iterator&lt;XWPFTable&gt; tableIterator = doc.getTablesIterator();
  
          int cnt = 0;
  
          while (tableIterator.hasNext()) {
  
              XWPFTable table = tableIterator.next();
  
              System.out.println("**********" + cnt + "**********");
  
              List&lt;XWPFTableRow&gt; rows = table.getRows();
  
              for (XWPFTableRow row : rows) {
  
                  List&lt;XWPFTableCell&gt; cells = row.getTableCells();
  
                  for (XWPFTableCell cell : cells) {
                      System.out.println(cell.getText());
                  }
              }
  
              System.out.println("********************");
  
              cnt++;
          }
      }
  }
  </code></pre>
  
  <p>How do I proceed? Where are my assumptions unrealistic or more information from the document is required?</p>