title: How to train a Chinese segmenter model by Stanford NLP Tools
body: <p>I am new to the Stanford CoreNLP Tools. Now I do not get an excellent segment result in Chinese, so I want to change the granulity of the Segmenter. I thought I could do this by training my own dict. </p>
  
  <p>I downloaded the <a href="http://nlp.stanford.edu/software/trainSegmenter-20080521.tar.gz" rel="nofollow" title="trainSegmenter-20080521.tar.gz"><code>trainSegmenter-20080521</code></a> file, and follow the <code>trainSegmenter-20080521/README.txt</code>.</p>
  
  <p>This is the README.txt:</p>
  
  <pre><code>Sat Jun 21 00:57:22 2008
  Author: Pi-Chuan Chang
  
  Here's a documentation of how to train and test the segmenter on specific split 
  range of the CTB data.
  
  The following steps assumes you have 3 files defining the ranges of train/dev/test.
  They should be named as "ctb6.train", "ctb6.dev", "ctb6.test" respectively.
  The format should be like:
        chtb_0003.fid
        chtb_0015.fid
        ...
  
  [STEP 1] change the CTB6 path in the Makefile:
        CTB6=/afs/ir/data/linguistic-data/Chinese-Treebank/6/
  
  [STEP 2] download and uncompress the lastest segmenter from:
        http://nlp.stanford.edu/software/stanford-chinese-segmenter-2008-05-21.tar.gz
  and change this path in the Makefile to your local path:
        SEGMENTER=/tmp/stanford-chinese-segmenter-2008-05-21/
  
  [STEP 3] simply type:
        make all
  You can also split down into these sub-steps:
        make internaldict # make internal dictionaries for affixation feaetures
        make data         # make datasets
        make traintest    # train &amp; test the CRF segmenter
  </code></pre>
  
  <p>But I still have some problems:</p>
  
  <ol>
  <li><p>What is the format of the training file and what is <code>train/dev/test</code> each for?</p></li>
  <li><p>What is the <code>chtb_0003.fid</code>, <code>chtb_0015.fid</code> and so on?</p></li>
  <li><p>What is the <code>CTB6 path</code> in the Makefile, it seems that I shoud change the variable <code>CTB6</code> into <code>/afs/ir/data/linguistic-data/Chinese-Treebank/6/</code>. But it is aready there and it seems not a valid subpath.</p></li>
  </ol>
  
  <p>By the way, there are many properties should be set for special demands, e.g., <code>sighanPostProcessing</code> and <code>serDictionary</code>. </p>
  
  <p>Is there somewhere I can get all of those properties and their explanation?</p>
  
  <p>All I can do now is to read the source code, e.g., <code>edu.stanford.nlp.sequences.SeqClassifierFlags.java</code>, but I still get confused with these property flags.</p>
  
  <p>So appreciated for anyone's help.</p>