TLex+: A Hybrid Method Using Conditional Random Fields and Dictionaries for Thai Word Segmentation
This paper proposes a new hybrid method of using Conditional Random Fields (CRFs) integrated with three dictionaries for Thai word segmentation. Based on TLex (Thai Lexeme Analyser), a pre-processing phase with an unambiguous list dictionary is added to deal with long expressions and long named entities (NE). Following this, the rest of the text is sent to the original TLex system, based on CRFs, to be segmented into words. Next, another dictionary is applied in a post-processing phase to check the number of unknown words in each of the top scored alternative segmentations from TLex in order to choose the best one. Finally, another NE dictionary is employed to merge each segmented named entity into one word. The results show that this hybrid method can improve the precision, recall and F-measure of TLex from 93.63%, 94.91%, and 94.27% to 97.64%, 97.37%, and 97.50%, respectively to become the most accurate Thai word segmentation system presently available.