title: Lucene, indexing already/externally tokenized tokens and defining own analyzing process
body: <p>in the process of using Lucene, I am a bit disapointed. I do not see or understand how i should proceed to feed any Lucene analyzers with something that is already and directly indexable. Or how i should proceed to create my own analyzer...</p>
  
  <p>for example, if i have a <code>List&lt;MyCustomToken&gt;</code>, which already contains many tokens (and actually many more informations about capitalization, etc. that i would also like to index as features on each of MyCustomToken)</p>
  
  <p>if i understand well what i have read, i need to subclass an Analyzer, which will call my own tokenizer subclassing a TokenStream, where i will only have to provide a <code>public final boolean incrementToken()</code> that will do the job of inserting <code>TermAttribute</code> @ position.</p>
  
  <p>BTW here is where i am confused => this TokenStream is a subclass of a java.io.Reader, and thus only capable of analyzing a stream object like a file, a string...</p>
  
  <p>how can i proceed to have my own document analyzer that will consume my List rather thant this stream-ed one? </p>
  
  <p>Looks like the whole Lucene API is builded on the idea that it first starts analyzing @ a very low level that are "characters" point of view, while i need to start using it later / plug from an already tokenized words or even expressions (groups of words). </p>
  
  <p>Typical samples of Lucene usage are like this (taken from <a href="http://www.lucenetutorial.com/lucene-in-5-minutes.html" rel="nofollow">here</a>) :</p>
  
  <pre><code>StandardAnalyzer analyzer = new StandardAnalyzer(Version.LUCENE_35);
  
  // 1. create the index
  Directory index = new RAMDirectory();
  
  IndexWriterConfig config = new IndexWriterConfig(Version.LUCENE_35, analyzer);
  
  IndexWriter w = new IndexWriter(index, config);
  addDoc(w, "Lucene in Action");   // BUT here i would like to have a addDoc(w, MyOwnObject)
  addDoc(w, "Lucene for Dummies");
  addDoc(w, "Managing Gigabytes");
  addDoc(w, "The Art of Computer Science");
  w.close();
  
  [...]   
  
  private static void addDoc(IndexWriter w, String value) throws IOException {
    Document doc = new Document();
    doc.add(new Field("title", value, Field.Store.YES, Field.Index.ANALYZED));
    // SO that i can add here my own analysis base on many fields, with them built from a walk through List or complex structures...
    w.addDocument(doc);
  }
  </code></pre>
  
  <p><br></p>
  
  <p>ps : (my java/lucene knowledge is still very poor, so i may have miss something obvious about the Reader &lt;=> List pattern?)</p>
  
  <p>this question is almost the same as mine <a href="http://lucene.472066.n3.nabble.com/Apply-custom-tokenization-tt3803765.html#none" rel="nofollow">on lucene list</a></p>
  
  <p><strong>EDIT:</strong> @ <em>Jilles van Gurp</em> => yes you are quite right, and it was another issue a i think of, but first hope to find a more elegant solution. So, if continuing, I can still do some kind of serialization, feed this serialized string as a document to my own analyzer, and own tokenizer that will then deserialize and re-do some basic tokenization (actually, just walking through the one already done...) BTW it will add some slower and stupid extra steps that i would have like to avoid...</p>
  
  <p>about this part => does someone have any sample of a recent (Lucene >3.6) custom tokenizer providing all the underlying data necessary to a Lucene Index? i have read about emitting Token like that :</p>
  
  <pre><code>        posIncrement.setPositionIncrement(increment); 
          char[] asCharArray = myAlreadyTokenizedString.toCharArray(); // here is my workaround 
          termAttribute.copyBuffer(asCharArray, 0, asCharArray.length); 
          //termAttribute.setTermBuffer(kept); 
          position++; 
  </code></pre>
  
  <p>for the why i am here part, it is because i use some external libraries, that tokenize my texts, do some part-of-speech annotation, and others analysis (one may think of a expression recognition or named entity recognition, can also include some special features about capitalization, etc.) that i would like to keep track in a Lucene Index (the real part that interest me is <strong>Indexing</strong> and <strong>Querying</strong>, not the first step of Analysis which is almost from the Lucene libary only Tokenising for what i have read).</p>
  
  <p>(also, i do not think i can do something smarter from these previous/early steps as i use many different tools, not all of them are Java or could be easily wrapped to Java)</p>
  
  <p>so i think this is a bit sad, that Lucene which is aim @ working with text is so bounded to word/tokens (sequence of chars) while text is much more than only juxtaposition of single/isolated words/tokens...</p>