title: Accuracy with TF-IDF and non-TF-IDF features
body: <p>I run a Random Forest algorithm with TF-IDF and non-TF-IDF features.</p>
  
  <p>In total the features are around 130k in number (after a feature selection conducted on the TF-IDF features) and the observations of the training set are around 120k in number.</p>
  
  <p>Around 500 of them are the non-TF-IDF features.</p>
  
  <p><strong>The issue is that the accuracy of the Random Forest on the same test set etc with</strong></p>
  
  <p><strong>- only the non-TF-IDF features is 87%</strong></p>
  
  <p><strong>- the TF-IDF and non-TF-IDF features is 76%</strong></p>
  
  <p>This significant aggravation of the accuracy raises some questions in my mind.</p>
  
  <p>The relevant piece of code of mine with the training of the models is the following:</p>
  
  <pre><code>drop_columns = ['labels', 'complete_text_1', 'complete_text_2']
  
  # Split to predictors and targets
  X_train = df.drop(columns=drop_columns).values
  y_train = df['labels'].values
  
  
  # Instantiate, train and transform with tf-idf models
  vectorizer_1 = TfidfVectorizer(analyzer="word", ngram_range=(1,2), vocabulary=tf_idf_feature_names_selected)
  X_train_tf_idf_1 = vectorizer_1.fit_transform(df['complete_text_1'])
  
  vectorizer_2 = TfidfVectorizer(analyzer="word", ngram_range=(1,2), vocabulary=tf_idf_feature_names_selected)
  X_train_tf_idf_2 = vectorizer_2.fit_transform(df['complete_text_2'])
  
  
  # Covert the general features to sparse array
  X_train = np.array(X_train, dtype=float)
  X_train = csr_matrix(X_train)
  
  
  # Concatenate the general features and tf-idf features array
  X_train_all = hstack([X_train, X_train_tf_idf_1, X_train_tf_idf_2])
  
  
  # Instantiate and train the model
  rf_classifier = RandomForestClassifier(n_estimators=150, random_state=0, class_weight='balanced', n_jobs=os.cpu_count()-1)
  rf_classifier.fit(X_train_all, y_train)
  </code></pre>
  
  <p>Personally, I have not seen any bug in my code (this piece above and in general).</p>
  
  <p>The hypothesis which I have formulated to explain this decrease in accuracy is the following.</p>
  
  <ol>
  <li>The number of non-TF-IDF features is only 500  (out of the 130k features in total)</li>
  <li>This gives some chances that the non-TF-IDF features are not picked that much at each split by the trees of the random forest (eg because of <code>max_features</code> etc) </li>
  <li>So if the non-TF-IDF features do actually matter then this will create problems because they are not taken enough into account.</li>
  </ol>
  
  <p>Related to this, when I check the features' importances of the random forest after training it I see the importances of the non-TF-IDF features being very very low (although I am not sure how reliable indicator are the feature importances especially with TF-IDF features included).</p>
  
  <p><strong>Can you explain differently the decrease in accuracy at my classifier?</strong></p>
  
  <p><strong>In any case, what would you suggest doing?</strong></p>
  
  <p>Some other ideas of combining the TF-IDF and non-TF-IDF features are the following.</p>
  
  <p>One option would be to have two separate (random forest) models - one for the TF-IDF features and one for the non-TF-IDF features.
  Then the results of these two models will be combined either by (weighted) voting or meta-classification.</p>