<p>For categorical features target based statistics were used. This is currently the best way to preprocess categorical features for GBDT, works better than one-hot. This is similar to target encoding but uses a permutation to not have overfitting.
Details and comparisons about this approach can be found in NIPS 2018 paper "CatBoost: unbiased boosting with categorical features" (<a href="https://arxiv.org/abs/1706.09516" rel="nofollow noreferrer">https://arxiv.org/abs/1706.09516</a>).</p>
What kind of preprocessing was used to encode categorical variables in CatBoost benchmarks?
<p>I've recently started to use CatBoost for rapid prototyping of machine learning models, inspired by the outstanding <a href="https://github.com/catboost/benchmarks/tree/master/quality_benchmarks" rel="nofollow noreferrer">performance benchmarks</a> of CatBoost compared to XGBoost, LightGBM and h2o.</p>

<p>Since XGBoost can only accept numeric features, a comparison between CatBoost and XGBoost needs a common preprocessing of categorical features. It is not entirely clear to me what kind of preprocessing was used to encode categorical features in the benchmarking experiments, and the rationale for not using a simple one-hot encoding.</p>

<p>I've tried to read the <a href="https://github.com/catboost/benchmarks/blob/master/quality_benchmarks/comparison_description.pdf" rel="nofollow noreferrer">documentation</a> of the experiments. As far as I understand it, the procedure to encode categorical feature <code>j</code> is about equivalent to the following:</p>

<ol>
<li>On the <code>train</code> set, group the response <code>y</code> by <code>j</code>, aggregating with the <code>mean</code> function . Let's call the result <code>df_agg_j</code></li>
<li>Left join the <code>train</code> set and <code>df_agg_j</code> on the categorical column <code>j</code>, drop the original categorical column <code>j</code> and use the new numeric column instead</li>
<li>Left join the <code>valid</code> set and <code>df_agg_j</code> on the categorical column <code>j</code>, drop the original categorical column <code>j</code> and use the new numeric column instead</li>
</ol>

<p>Still I don't understand the need for "a random permutation of the objects for j-th categorical feature and i-th object", and for adding 1 at the numerator and 2 to the denominator in the final formula under the section "Preparation of Splits" of the <a href="https://github.com/catboost/benchmarks/blob/master/quality_benchmarks/comparison_description.pdf" rel="nofollow noreferrer">documentation</a>.</p>

<p>The code for splitting and preprocessing the data can be found <a href="https://github.com/catboost/benchmarks/blob/master/quality_benchmarks/experiment.py" rel="nofollow noreferrer">here</a>.</p>

<p>Is there an explanation (or some reference in the literature) about the method used to encode categorical features in this experiment, and a comparison between this method and one-hot encoding?</p>