title: Feature Selection and Reduction for Text Classification
body: <p>I am currently working on a project, a <strong>simple sentiment analyzer</strong> such that there will be <strong>2 and 3 classes</strong> in <strong>separate cases</strong>. I am using a <strong>corpus</strong> that is pretty <strong>rich</strong> in the means of <strong>unique words</strong> (around 200.000). I used <strong>bag-of-words</strong> method for <strong>feature selection</strong> and to reduce the number of <strong>unique features</strong>, an elimination is done due to a <strong>threshold value</strong> of <strong>frequency of occurrence</strong>. The <strong>final set of features</strong> includes around 20.000 features, which is actually a <strong>90% decrease</strong>, but <strong>not enough</strong> for intended <strong>accuracy</strong> of test-prediction. I am using <strong>LibSVM</strong> and <strong>SVM-light</strong> in turn for training and prediction (both <strong>linear</strong> and <strong>RBF kernel</strong>) and also <strong>Python</strong> and <strong>Bash</strong> in general.</p>
  
  <p>The <strong>highest accuracy</strong> observed so far <strong>is around 75%</strong> and I <strong>need at least 90%</strong>. This is the case for <strong>binary classification</strong>. For <strong>multi-class training</strong>, the accuracy falls to <strong>~60%</strong>. I <strong>need at least 90%</strong> at both cases and can not figure how to increase it: via <strong>optimizing training parameters</strong> or <strong>via optimizing feature selection</strong>?</p>
  
  <p>I have read articles about <strong>feature selection</strong> in text classification and what I found is that three different methods are used, which have actually a clear correlation among each other. These methods are as follows:</p>
  
  <ul>
  <li>Frequency approach of <strong>bag-of-words</strong> (BOW)</li>
  <li><strong>Information Gain</strong> (IG)</li>
  <li><strong>X^2 Statistic</strong> (CHI)</li>
  </ul>
  
  <p>The first method is already the one I use, but I use it very simply and need guidance for a better use of it in order to obtain high enough accuracy. I am also lacking knowledge about practical implementations of <strong>IG</strong> and <strong>CHI</strong> and looking for any help to guide me in that way.</p>
  
  <p>Thanks a lot, and if you need any additional info for help, just let me know.</p>
  
  <hr>
  
  <ul>
  <li><p>@larsmans: <strong>Frequency Threshold</strong>: I am looking for the occurrences of unique words in examples, such that if a word is occurring in different examples frequently enough, it is included in the feature set as a unique feature.   </p></li>
  <li><p>@TheManWithNoName: First of all thanks for your effort in explaining the general concerns of document classification. I examined and experimented all the methods you bring forward and others. I found <strong>Proportional Difference</strong> (PD) method the best for feature selection, where features are uni-grams and <strong>Term Presence</strong> (TP) for the weighting (I didn't understand why you tagged <strong>Term-Frequency-Inverse-Document-Frequency</strong> (TF-IDF) as an indexing method, I rather consider it as a <strong>feature weighting</strong> approach).  <strong>Pre-processing</strong> is also an important aspect for this task as you mentioned. I used certain types of string elimination for refining the data as well as <strong>morphological parsing</strong> and <strong>stemming</strong>. Also note that I am working on <strong>Turkish</strong>, which has <strong>different characteristics</strong> compared to English. Finally, I managed to reach <strong>~88% accuracy</strong> (f-measure) for <strong>binary</strong> classification and <strong>~84%</strong> for <strong>multi-class</strong>. These values are solid proofs of the success of the model I used. This is what I have done so far. Now working on clustering and reduction models, have tried <strong>LDA</strong> and <strong>LSI</strong> and moving on to <strong>moVMF</strong> and maybe <strong>spherical models</strong> (LDA + moVMF), which seems to work better on corpus those have objective nature, like news corpus. If you have any information and guidance on these issues, I will appreciate. I need info especially to setup an interface (python oriented, open-source) between <strong>feature space dimension reduction</strong> methods (LDA, LSI, moVMF etc.) and <strong>clustering methods</strong> (k-means, hierarchical etc.).</p></li>
  </ul>