Automatic learning algorithm selection for classification via
  convolutional neural networks
As in any other task, the process of building machine learning models can
benefit from prior experience. Meta-learning for classifier selection gains
knowledge from characteristics of different datasets and/or previous
performance of machine learning techniques to make better decisions for the
current modeling process. Meta-learning approaches first collect meta-data that
describe this prior experience and then use it as input for an algorithm
selection model. In this paper, however, we propose an automatic learning
scheme in which we train convolutional networks directly with the information
of tabular datasets for binary classification. The goal of this study is to
learn the inherent structure of the data without identifying meta-features.
Experiments with simulated datasets show that the proposed approach achieves
nearly perfect performance in identifying linear and nonlinear patterns,
outperforming the traditional two-step method based on meta-features. The
proposed method is then applied to real-world datasets, making suggestions
about the best classifiers that can be considered based on the structure of the
data.
Sebastian Maldonado, Carla Vairetti, Ignacio Figueroa
# Automatic learning algorithm selection for classification via convolutional neural networks

###### Abstract

As in any other task, the process of building machine learning models can benefit from prior experience. Meta-learning for classifier selection gains knowledge from characteristics of different datasets and/or previous performance of machine learning techniques to make better decisions for the current modeling process. Meta-learning approaches first collect meta-data that describe this prior experience and then use it as input for an algorithm selection model. In this paper, however, we propose an automatic learning scheme in which we train convolutional networks directly with the information of tabular datasets for binary classification. The goal of this study is to learn the inherent structure of the data without identifying meta-features. Experiments with simulated datasets show that the proposed approach achieves nearly perfect performance in identifying linear and nonlinear patterns, outperforming the traditional two-step method based on meta-features. The proposed method is then applied to real-world datasets, making suggestions about the best classifiers that can be considered based on the structure of the data.

Meta-learning, Meta-features, Algorithm selection, Convolutional networks.

## 1 Introduction

Deep neural networks (DNNs) have shown extraordinary advances in recent years due to their ability to collect and process large volumes of data [12, 33]. Their well-deserved popularity has led to remarkable methodological developments and a broad spectrum of new applications [4, 12].

In this paper, we propose the use of deep neural networks for meta-learning. This task is typically described as "learning to learn" in the sense that knowledge is extracted from datasets and machine learning algorithms, and then used to alleviate the challenges that face the current learning process [3, 19, 25].1
Meta-learning is a broad field of machine learning with its origin some decades before the rise of deep learning. Some primary studies in this field learned from task properties, constructing "meta-features" that represent the datasets. The goal is then to transfer insights from the most similar tasks to a new task [3]. The reasoning behind meta-learning has been extended to other relevant challenges associated with deep neural networks, such as transfer learning or multitask learning [30]. These challenges, however, are not related to this paper, which focuses on learning algorithm selection for classification [3, 10].

This paper proposes the automatic extraction of meta-features using convolutional neural networks (CNNs). These techniques are well-known DNN architectures that are designed to learn from images, among other data sources [4, 12]. Instead of inputting images to a CNN, we consider simulated tabular datasets with distinguishable linear and nonlinear patterns, defining a multiclass task.

The proposed strategy represents a novel approach to meta-learning and algorithm selection. Specifically, it involves leveraging deep learning techniques to learn meta-features from simulated patterns, which can then be applied in the algorithm selection process. To the best of our knowledge, this is the first time such an approach has been proposed. We are confident that this methodology constitutes a significant contribution to the field, opening an interesting research line. This study seeks to answer the following questions: (1) can we learn linear and nonlinear data structures from raw tabular datasets using deep learning? and (2) if yes, can we leverage this knowledge to make suggestions on the most suitable machine learning method or methods for a specific task?

Our study is consistent with the reasoning behind the "no free lunch" (NFL) theorem in the sense that it may not be effective to gain knowledge from tasks to apply it to a completely unrelated dataset. However, existing meta-learning studies have shown the advantages of learning from prior experience [27].

This paper is organized as follows. Section 2 provides an overview of relevant meta-learning studies. The proposed meta-learning framework is presented in Section 3. Section 4 discusses the results obtained using simulated and benchmark datasets. Finally, Section 5 summarizes the primary conclusions and presents possible directions for future developments.

## 2 Prior Work on Meta-learning for Classifier Selection

Meta-learning is an umbrella term that encompasses several approaches that gain experience across tasks. The goal of meta-learning is to avoid time-consuming "trial and error" processes and make better choices when estimating machine learning models. Some examples of these approaches include the following (see [27] for a comprehensible literature review):

* _Learning from previous model estimations and evaluations:_ Given a set of possible configurations \(\Theta\), the goal of this approach is to train a meta-learner based on previous model evaluations to make recommendations on a new task. In a hyperparameter search task, for example, the accuracy of a classifier can be estimated on different datasets (or variants of the same datasets using bootstrapping or cross-validation), and a meta-learner can be used to suggest an optimal hyperparameter configuration (or a set of candidate configurations, called a _portfolio_[27]). Note that a configuration can be a set of hyperparameters and network architectures or pipeline components [27].
* _Transfer learning:_ The reasoning behind this approach is to consider models that are trained with data from one or more sources as a starting point for developing a new model on a similar task. Although this idea has been applied to traditional machine learning methods, this approach has been particularly successful in deep learning (DL) [27]. In tasks such as object recognition and natural language processing (NLP), DL requires a large amount of data to achieve superior performance in relation to other methods. A _pretrained model_ can be constructed with publicly available data (e.g., Wikipedia or books in the case of NLP tasks [20] or ImageNet in the case of visual object recognition [18]), which is then adapted to the new task via _fine-tuning_.
* _Learning from meta-features:_ The goal of this approach is to make better decisions during machine learning by learning from meta-features or properties that describe the datasets. These characterizations can be used for hyperparameter selection [16, 19] or classifier selection [3, 10], among other tasks. A task similarity metric can be defined to transfer knowledge from one task to a new similar task, or a meta-learner can be constructed [2]. For example, in [3], a decision tree learner is trained on several meta-features that were constructed on more than 100 datasets from the UCI Repository [5] and other sources. A multiclass classification problem with five classes related to five different classification algorithms is defined. Alternatively, information on dataset characteristics can be useful to infer the performance of feature selection methods [21].
The method proposed in this study is related to the latter approach in the sense that we learn from characterizations of other tasks to make recommendations of suitable classifiers. However, instead of using a two-step approach by first constructing meta-features and then estimating a meta-learner, the proposed meta-learner is fed with tabular datasets directly.

We can distinguish different purposes in the creation of meta-features in the sense that specific measures aim to identify specific patterns, such as feature interdependence, class overlap, or task similarity:

* _Feature normality and dispersion:_ Statistical measures used to describe a distribution can be considered to assess normality in the covariates. Some common examples are skewness \(\frac{E(X-\mu)^{3}}{\sigma^{3}}\) and kurtosis (\(\frac{E(X-\mu)^{4}}{\sigma^{4}}\)), with \(\mu\) and \(\sigma\) being the mean and standard deviation of the variable, respectively [28]. Other measures that can be related to feature normality are the interquartile range (IQR, the difference between quartile 3 and quartile 1) and the value of the 90th quantile [3]. Other statistics that can assess tendency and dispersion are the arithmetic mean, the geometric mean, the harmonic mean, the trimmed mean, the standard deviation, and the mean absolute deviation (MAD, \(\frac{\sum|x_{i}-\overline{x}|}{n}\)) [3]. Finally, the Index of Dispersion (ID) indicates whether the data points are scattered or clustered [3].
* _Complexity:_ Simple measures can be an indicator of the complexity of the learning tasks, such as the size of the dataset (number of rows and columns), and the number of classes. These measures can be related to the expected training times [17]. Alternatively, the percentage of missing values and outliers can be related to the complexity of the preprocessing step. [3, 24].
* _Feature interdependence:_ The level of redundancy in a dataset can be assessed using the Pearson correlation (\(\rho=\frac{\sigma_{xy}}{\sigma_{x}\sigma_{y}}\)) or by analyzing the eigenvalues that result from applying the principal component analysis (PCA) method for feature extraction [17]. Some PCA-based meta-features are canonical correlations (square root of the eigenvalues), the first and last PC [3], and the skewness and kurtosis of the first PC [6].
* _Class overlap and feature relevance:_ The meta-learning literature has reported different metrics to evaluate overall feature relevancy, such as the center of gravity (Euclidean norm between minority and majority classes) [3], the entropy of classes (\(H(C)=-\sum_{i}\pi_{i}\log\pi_{i}\)), the mean mutual information (MMI), the equivalent number of variables (ENV, ratio between the class entropy and the MMI), and the noise-signal ratio (\(NSR=\frac{H(X)-M(C,X)}{M(C,X)}\)) [3]. A large value for the latter metric measure suggests that the dataset contains several irrelevant noisy variables, and therefore, its dimensionality can be reduced without affecting the classification accuracy [3].
* _Landmarks:_ This strategy computes measures designed to assess task similarity, using the learners themselves to describe the dataset. The idea is to compare differences in terms of performance between configurations and/or datasets using simple classifiers, such as 1-NN or naive Bayes [7].

It is important to notice that, although meta-learning have been usually applied for algorithm recommendation in classification [29, 35]. Other machine learning tasks in which meta-learning has shown good results include regression [15], time series analysis [13], data stream mining [23], and clustering [7, 11, 22].

## 3 Proposed Classifier Selection Framework via CNNs

This paper proposes a novel meta-learning framework in which tabular datasets are introduced in a learning algorithm directly, which differs from the traditional two-step approach that involves the construction of meta-features. In this sense, each tabular dataset is treated as an "image", using CNNs for model training. The reasoning behind this approach is to avoid the loss of information that occurs when meta-features are constructed and subsequently used as inputs for a traditional classifier, such as decision trees [3].

Our framework consists of four steps:

1. **Simulation of patterns:** We define different training patterns that can be related to the performance of learning algorithms. These patterns can be simulated under different noise and class overlap conditions, and then used to feed a CNN in a supervised manner. For example, a logistic regression or a linear support vector machine (SVM) can be recommended when dealing with a tabular dataset with a linear pattern. The most suitable classifier is a hyperplane, while kernel-based SVM, NN, or random forests can be recommended when the dataset at hand follows a nonlinear pattern.
2. **Dealing with inputs of different sizes:** Because the proposed goal is to make recommendations for real-world tabular datasets, we expect that datasets are all of different sizes. Therefore, we must reshape the training and
test samples to a homogeneous size. We propose simple approaches for dealing with images of different sizes with CNNs, such as padding and principal component analysis (PCA).
3. **CNN training:** Often used for image-related tasks such as pattern recognition, segmentation, or object detection, CNNs are among the most popular deep learning variants [31]. CNNs are used in this framework because they outperform other methods for image recognition due to their ability to manage tensor data without the need for an additional feature extraction step [9, 34].
4. **Application to real-world datasets:** The final step consists of applying the CNN model to real-world datasets to identify one of the simulated patterns that was considered during training. We can link the pattern found to one or more suitable classifiers for this pattern. It is important to assess the confidence of the classifier on its choice, and therefore, we must analyze the probabilistic output of the network using a softmax function. For example, we consider a learning task with five simulated patterns. If the largest predicted probability for a given real-world dataset is 0.3, then the model is undecided, and no recommendation can be made because this probability is too near a random choice (0.2 for a 5-class task). In contrast, the model is certain when the probability of a given class is near 1, leading to a trustworthy recommendation.

The first step is arguably the most challenging because only a comprehensible set of patterns under different conditions of noise would lead to an adequate application on real-world datasets. The success of the approach strongly relies on this step. As a first attempt, we discuss five different patterns that are well known in the machine-learning literature, such as the two moons or XOR patterns [32]. We believe that this paper opens an exciting new line of research in which different sets of simulated patterns can be designed.

Regarding the second step, padding is used to resize tabular datasets that are smaller than a target size by extending the area across which a CNN processes. This process is performed by introducing additional rows or columns of zeros [1]. In contrast, PCA can reduce tabular datasets to a target size by finding (orthogonal) linear combinations of the original variables in such a way that the variance is maximized. Thus, we can shrink the datasets while keeping their inherent structure [14].

For the third step, we consider standard CNN architectures with Conv2D layers, which computes two-dimensional convolutions between the inputs and a matrix of weight vectors called kernels. The rectified linear unit (ReLU) is used as the activation function, which allows adequate convergence for the weight updating process, reducing the risk of the vanishing/exploding gradient problem [1]. Additionally, dropout layers are used in the network to prevent overfitting by removing hidden units from a specific layer section, setting them to zero [1].

After the convolutional and dropout layers, a flattened layer was considered to collapse the two-dimensional inputs into a one-dimensional vector. Dense (fully connected) layers were subsequently included, finishing with the softmax output layer [31]. The optimization process considered \(l_{1}\) regularization for the weights together with the minimization of the loss function (multiclass cross-entropy loss).

We emphasize on the fact that the approach provides an **automatic learning algorithm selection model for classification** in the sense that, once the CNN model is constructed, it can be applied directly to any tabular dataset to suggest a classifier based on its characteristics. The first three steps are performed only once, and are not required for the application of the model. We plan to make the CNN model publicly available in order that the community can use it and improve it, similar to other pre-trained deep learning models.

## 4 Experimental Results

To validate the proposed meta-learning framework, we first trained CNN models with simulated data that had different linear and nonlinear patterns, and compared the performances with the traditional approach based on meta-features proposed in [3]. The results of these experiments are shown in Section 4.1. The application of CNNs with real-world tabular datasets is reported in Section 4.2.

### Construction of the Classifier Selection Model

We simulated a total of 50,000 tabular datasets with five different patterns (10,000 datasets each). Each dataset consisted of tuples in the form \(\{(\mathbf{x}_{1},y_{1}),\ldots,(\mathbf{x}_{m},y_{m})\}\), where \(\mathbf{x}_{i}\in\mathbb{R}^{n}\) and \(y_{i}=\{-1,1\}\) for \(i=1,\ldots,m\). The supervised learning task consists of predicting the right pattern of the datasets (i.e., a multiclass problem), and the inputs are two-class datasets of size \(m=1000\), in which the two classes are perfectly balanced.

Five two-dimensional patterns that act as classes for the meta-learner were defined: a linear pattern (C1), an XOR pattern (C2), a two-moons pattern (C3), a "sandwich" pattern (C4), and a quadratic pattern (C5). The following two-dimensional patterns were defined:
- linear pattern:** We combine two Gaussian functions, one for each class, varying the mean and standard deviation of each Gaussian to create different overlap and noise conditions.
* XOR pattern:** We combine four Gaussian functions, two for each class, varying the mean and standard deviation of each Gaussian to create different overlap and noise conditions. The four Gaussians shape the well-known XOR pattern related to the exclusive disjunction logical operation. This nonlinear pattern is also known as "checkerboard".
* "Two moons" pattern:** This well-known synthetic dataset constructs a swirl pattern that is shaped like two moons. We vary the conditions in terms of noise and overlap.
* "Sandwich" pattern:** We simulate samples using one Gaussian function interspersed with two other Gaussians related to a second class. This results in a nonlinear pattern in the form of a "sandwich". We vary the mean and standard deviation of each Gaussian distribution to create different overlap and noise conditions.
* Quadratic pattern:** We simulate samples based on two quadratic functions, one for each class, with a marginal overlap. We vary the conditions in terms of noise and overlap.

Figure 1 shows the five simulated patterns.

To highlight the performance differences of the various patterns, we used several well-known classifiers, evaluating performance on a test set of simulated tabular datasets. The following classification methods and their respective hyperparameter values were considered:

* _Logistic regression (Logit):_ This statistical method is suitable for a linear pattern because it constructs a separating hyperplane. The slopes of this function are given by a vector of coefficients \(\beta\).

Figure 1: Simulated patterns for the meta-learning framework.
* _Decision tree (DT)_: This method performs a branching process in a hierarchical manner until a pruning criterion is met. The Gini index was considered for splitting, while three different values were considered for the complexity parameter (default pruning parameter values of the 'part' R implementation) \(cp=\{0,0.03,0.81\}\).
* _\(k\)-nearest neighbor (\(k\)-NN)_: This classifier predicts new samples according to the labels of a neighborhood of size \(k\) that were obtained from the training set, with \(k=\{5,7,9\}\).
* _Random forest (RF)_: This method constructs an ensemble of decision trees via bagging. The same criteria for the DT classifier were considered, constructing 500 trees with two randomly selected variables.
* _Artificial neural network (ANN)_: A shallow ANN classifier with one hidden layer was constructed. We explored 1, 3, and 5 hidden units and the following values for the weight decay parameter: \(\epsilon=\{0,0.0001,0.1\}\).
* _Support vector machine (SVM)_: We considered the standard kernel-based SVM model with Tikhonov regularization and hinge loss. We explored the following values for the regularization parameter: \(C=\{0.25,0,5,1,2,4\}\), while the kernel width \(\sigma\) was held constant at a value of 1.004454.

Table 1 shows the area under the curve (AUC) measure in the test set for all the methods and simulated patterns (C1 to C5). The largest AUC value is highlighted in bold type, while the second largest AUC appears underlined. Because all the patterns are binary classification problems, AUC is a suitable performance metric that does not rely on a threshold and can balance type-I and -II errors adequately [8].

Table 1 shows that the top classification approaches vary depending on the simulated pattern:

* _C1_: Logit is the best classifier, followed by ANN.
* _C2_: \(k\)-NN is the best classifier, followed by RF.
* _C3_: SVM is the best classifier, followed by RF.
* _C4_: SVM is the best classifier, followed by \(k\)-NN.
* _C5_: SVM is the best classifier, followed by RF.

At this stage, the proposed approach can match the expected classifiers in the sense that the logistic regression performs best for the sole linear pattern (C1), while nonlinear methods (kernel-based SVM, random forest, and \(k\)-NN) perform best with the remaining four nonlinear patterns. For these patterns, the logit classifier does not perform well. Although it is clear that "there is no free lunch", we can still make recommendations for classifiers if the patterns are identified correctly.

We define two different sets of experiments based on the five patterns for model training. We first consider two-dimensional matrices of similar size (i.e., \(m=1000\) and \(n=2\)). We refer to this meta-learning approach as CNN1. We also consider heterogeneous datasets in terms of the number of samples and variables: we generated the patterns using a random number of samples between 800 and 1400 and introduced 0 to 4 irrelevant variables (Gaussian noise), adding them to the two-dimensional patterns (i.e., \(m=[800,1400]\) and \(n=[2,6]\)). We refer to this approach as CNN2.

Note that the goal of training CNNs with a heterogeneous dataset in terms of size is to consider the fact that the goal is to apply the final model to real-world data, which are clearly heterogeneous. Training with tabular datasets of different sizes (CNN2) is then a more realistic approach to meta-learning. However, transformations are required before model training.

As mentioned in the previous section, we combined two simple strategies to manage "images" of different sizes (simulated tabular datasets in the proposed case): (1) PCA to shrink the datasets that are larger than a target average sample size (1099 observations for each tabular dataset, where a total of 1099 components are selected); and (2) padding to create additional space within a dataset smaller than the target average size. Padding is also used to achieve the

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline Classifier & C1 & C2 & C3 & C4 & C5 \\ \hline Logit & **0.938** & 0.590 & 0.954 & 0.609 & 0.904 \\ DT & 0.859 & 0.803 & 0.890 & 0.845 & 0.903 \\ \(k\)-NN & 0.909 & **0.885** & 0.965 & 0.966 & 0.970 \\ RF & 0.910 & 0.884 & 0.966 & 0.962 & 0.981 \\ ANN & 0.926 & 0.577 & 0.953 & 0.691 & 0.867 \\ SVM & 0.920 & 0.882 & **0.968** & **0.981** & **0.985** \\ \hline \hline \end{tabular}
\end{table}
Table 1: Predictive performance for the five simulated patterns.
target size of 7, which is the maximum size of a tabular input \(\{\mathbf{X},\mathbf{y}\}\) with the two-dimensional pattern, four irrelevant variables, and the label vector \(\mathbf{y}\).

The network used is a straightforward two-stage architecture. The first stage is used for feature extraction and learning and consists of two Conv2D layers with 32 filters and a kernel size of 5x5, followed by a dropout layer with a rate of 0.25 and then two Conv2D layers with 64 filters each and a kernel size of 3x3. The first two Conv2D layers used a kernel size of 5x5 for more general feature extraction, while the third and fourth used 3x3 kernels to extract more detailed features from the data. Combinations of kernel sizes in CNNs have yielded successful results in object recognition tasks [34, 9].

The second stage performs the classification task and begins with a flattened layer, followed by a dense layer with 256 neurons, a dropout layer with a rate of 0.5, another dense layer with 5 neurons, and finally a softmax layer with five outputs for the 5 simulated patterns proposed in this experimental setting.

The network was trained with a batch size of 1000 tabular datasets and 10 epochs. We used Adam as the optimizer, which has shown excellent empirical results in terms of efficiency, scalability, and performance (i.e., faster convergence) [1]. Adam is a first-order gradient-based solver stochastic optimization with adaptive learning rates, allowing a more effective and smooth learning process.

The following values were considered for the optimization process with Adam: \(\alpha=1e-03\) (initial learning rate), \(\beta_{1}=0.9\) and \(\beta_{2}=0.999\) (parameters that control the exponential decay rates of the moving averages), and \(\epsilon=1e-07\) (conditioning parameter). These parameter values were selected according to the original Adam paper [1]. The proposed methodology was implemented in the TensorFlow Python library.

Table 2 shows the performance of five different meta-learning strategies by considering various multiclass metrics: micro and macro averages for the precision, recall, and f1 (harmonic mean of the precision and recall). We consider the proposed four CNN variants and the meta-learning approach suggested in [3] (MF+DT), which consists of constructing several meta-features and using a decision tree for model training with the meta-features as inputs. This latter strategy includes several meta-learning studies given the set of meta-features considered in this study (see [3, 6, 17, 24]). We consider the heterogeneous dataset used to train CNN2 to construct the meta-features.

Table 2 shows that the two proposed CNN variants achieved similar performances with nearly perfect classification. We can conclude that the proposed approach can learn from various simulated patterns and identify them in other unseen tabular datasets with marginal variations. In contrast, the two-step approach that constructs meta-features and then implements a classifier is clearly inferior in terms of its predictive capabilities, with an accuracy of approximately 75% on average.

### Application on benchmark datasets

Next, we explore the ability of the proposed model to make suggestions for suitable classifiers. We compare the proposed CNN2 approach with the alternative strategy based on meta feta-features (MF+DT) on well-known binary classification datasets from the UCI Repository [5]. CNN2 is chosen because its construction is designed to manage heterogeneous datasets, although CNN1 achieved marginally better classification results. Also, CNN2 and MF+DT consider the same input information and are therefore comparable.

Table 3 shows relevant metadata for all the benchmark datasets, including the number of examples \(m\), the number of variables \(n\), and the percentage of examples in each class (min.,maj. ), and the imbalance ratio (IR), computed as the fraction of the majority class and minority class samples.

The tabular datasets exhibit important differences in terms of size and imbalance ratio, which is important to assess meta-learning approaches properly (see Table 3). Note that these datasets have no "class" in the sense that they have no known linear/nonlinear pattern. Therefore, multiclass classification performance cannot be computed. However, we can first compute the binary classification performance for each tabular dataset and then compare it with the patterns suggested with both CNN and MF+DT.

\begin{table}
\begin{tabular}{l l l l} \hline Perf. Measure & CNN1 & CNN2 & MF+DT \\ \hline precision (macro) & 0.996 & 0.986 & 0.790 \\ recall (macro) & 0.996 & 0.985 & 0.752 \\ f1-score (macro) & 0.996 & 0.985 & 0.745 \\ f1-score (micro) & 0.996 & 0.985 & 0.752 \\ \hline \end{tabular}
\end{table}
Table 2: Predictive performance for the various meta-learning approaches.
The results of the various classifiers on the 23 datasets are shown in Table 4. The best classifier for each dataset is highlighted in bold type.

\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline ID & Logit & DT & \(k\)-NN & RF & ANN & SVM \\ \hline ds1 & 0.777 & 0.82 & **0.892** & 0.864 & 0.793 & 0.822 \\ ds2 & 0.865 & 0.863 & 0.844 & 0.865 & 0.858 & **0.866** \\ ds3 & 0.982 & 0.977 & 0.997 & 0.986 & **1** & **1** \\ ds4 & 0.544 & **0.644** & 0.56 & 0.613 & 0.604 & 0.5 \\ ds5 & 0.51 & 0.594 & **0.719** & 0.671 & 0.676 & 0.702 \\ ds6 & 0.647 & 0.514 & 0.616 & **0.666** & 0.527 & 0.617 \\ ds7 & 0.701 & 0.755 & 0.594 & **0.79** & 0.759 & 0.775 \\ ds8 & 0.5 & 0.519 & 0.625 & **0.745** & 0.5 & 0.5 \\ ds9 & 0.919 & 0.96 & **0.994** & 0.987 & 0.978 & 0.981 \\ ds10 & 0.504 & 0.556 & **0.65** & 0.611 & 0.559 & 0.536 \\ ds11 & 0.761 & 0.786 & 0.886 & **0.935** & 0.912 & 0.932 \\ ds12 & 0.771 & 0.822 & 0.93 & **0.941** & 0.906 & 0.899 \\ ds13 & 0.5 & 0.5 & **0.577** & 0.5 & 0.5 & 0.5 \\ ds14 & 0.667 & 0.753 & **0.842** & 0.804 & 0.745 & 0.792 \\ ds15 & 0.538 & 0.498 & 0.591 & **0.634** & 0.599 & 0.585 \\ ds16 & **0.824** & 0.82 & 0.813 & 0.821 & 0.821 & 0.821 \\ ds17 & 0.769 & 0.817 & **0.952** & 0.929 & 0.793 & 0.826 \\ ds18 & 0.68 & 0.681 & 0.85 & **0.771** & 0.757 & 0.68 \\ ds19 & 0.743 & 0.723 & 0.787 & 0.783 & **0.827** & 0.807 \\ ds20 & 0.692 & 0.675 & **0.697** & 0.677 & 0.695 & 0.695 \\ ds21 & **0.867** & 0.803 & 0.84 & 0.862 & 0.866 & **0.867** \\ ds22 & 0.876 & 0.93 & 0.941 & **0.946** & 0.865 & 0.913 \\ ds23 & 0.957 & 0.976 & 0.979 & **0.981** & 0.975 & 0.972 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Predictive performance for the various benchmark datasets.

\begin{table}
\begin{tabular}{c l c c c} \hline \hline ID & dataset & \(m\) & \(n\) & \%class(min,maj) & IR \\ \hline ds1 & abalone7 & 4177 & 8 & (9.3, 90.7) & 9.7 \\ ds2 & australian-credit & 690 & 14 & (44.5,55.5) & 1.2 \\ ds3 & banknote-auth & 1372 & 4 & (44.5,55.5) & 1.2 \\ ds4 & breast-cancer & 569 & 30 & (37.3,62.7) & 1.7 \\ ds5 & bupa-liver & 345 & 6 & (42, 58) & 1.4 \\ ds6 & german-credit & 1000 & 24 & (30.0,70.0) & 2.3 \\ ds7 & hearth-statlog & 270 & 13 & (44.4,55.6) & 1.3 \\ ds8 & horse-colic & 300 & 27 & (33.0,67.0) & 2.0 \\ ds9 & image-1 & 2310 & 19 & (14.3, 85.7) & 6 \\ ds10 & image-5 & 2310 & 19 & (38.1,61.9) & 1.6 \\ ds11 & ionosphere & 351 & 34 & (35.9,64.1) & 1.8 \\ ds12 & monk-2 & 432 & 6 & (47.2,52.8) & 1.1 \\ ds13 & oil-spill & 937 & 49 & (4.4,95.6) & 21.9 \\ ds14 & phoneme & 5404 & 19 & (29.3,70.7) & 2.4 \\ ds15 & pima-diabetes & 768 & 8 & (34.9,65.1) & 1.9 \\ ds16 & ring & 7400 & 20 & (49.5,50.5) & 1.0 \\ ds17 & solar-flares-M & 1389 & 10 & (4.9,95.1) & 19.4 \\ ds18 & sonar & 208 & 60 & (46.6,53.4) & 1.4 \\ ds19 & splice & 1000 & 60 & (48.3,51.7) & 1.1 \\ ds20 & titanic & 2201 & 3 & (32.3,67.7) & 2.1 \\ ds21 & waveform & 5000 & 21 & (33.1,66.9) & 2.0 \\ ds22 & yeast02579vs368 & 1004 & 8 & (9.9, 90.1) & 9.1 \\ ds23 & yeast5 & 1484 & 8 & (3.0, 97.0) & 32.78 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Descriptive statistics for all the benchmark datasets.
Table 4 shows that no method outperforms others in terms of performance. In contrast to the performance of these classifiers with simulated data (see Table 1), random forest and \(k\)-NN achieved a better performance than SVM. These results confirm the advantages of RF when facing noisy mixed-type data (numerical and dummy variables).

Next, the results of the proposed CNN approach for meta-learning are reported in Table 5. For each dataset, the predicted probability of belonging to a given training pattern is shown. The largest predicted probability for a given dataset is highlighted in bold type. The best classifier (bc) found in Table 4 is also reported, and whether the recommendation is successful or not (column "hit").

A "hit" occurs when one of the two classifiers recommended by the predicted class coincides with the best classifier bc. For ds1, for example, C2 is the predicted class with a probability of 0.978, and therefore, the recommendations for this dataset based on the simulated data are first \(k\)-NN and then RF (see Table 1). The best classifier is \(k\)-NN, and therefore, the recommendation is valid. A hit is highlighted with the symbol \(\checkmark\), also including the relative position of the recommended classifier (1 or 2). In contrast, for dataset ds4, the recommendations are also \(k\)-NN and RF (class C2, with a confidence of 0.947), but the best classifier is DT, and therefore, the recommendation is not successful (hit=\(\checkmark\)).

Note that there is a tie for bc in three of the 23 datasets. In such cases, the best classifier that is considered a hit appears underlined in case one of the two best classifiers is indeed recommended by the CNN.

Table 5 demonstrates the advantages of the CNN: recommendations are successful 78.2% of the time (18 of the 23 datasets). Because a random recommender would achieve a 33.3% success rate at suggesting two of the six classifiers, the proposed meta-learning approach clearly outperforms a random model. Of the 19 hits, 13 correspond to the second-best model, while only six correspond to the best model. However, at least two classifiers should be recommended because the performance of the top method is typically similar to that of the second-best method.

Table 5 also shows the confidence of the CNN in predicting the training pattern: the largest probability is below 0.8 only in three of the 23 datasets. This result demonstrates that the method indeed identifies one of the various patterns in the datasets.

Finally, we compare the performance of the proposed method with the traditional two-step approach. Table 6 shows the results when a DT is applied on meta-features. The identified training pattern is highlighted with the symbol \(\clubsuit\). In the case of ties, the best classifier (bc) that is recommended by the meta-learning method appears underlined. Note that landmarks are not included because they consider information that is not available for the proposed approach (e.g., the performance of the various traditional classifiers).

\begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline ID & C1 & C2 & C3 & C4 & C5 & bc & hit \\ \hline ds1 & 0.022 & **0.978** & 0 & 0 & 0 & k-NN & \(\checkmark\)(1) \\ ds2 & **0.965** & 0.026 & 0 & 0 & 0.009 & SVM & \(\checkmark\) \\ ds3 & 0.044 & 0 & 0 & 0.057 & **0.898** & ANN/SVM & \(\checkmark\)(1) \\ ds4 & 0.041 & **0.947** & 0.001 & 0.001 & 0.01 & DT & \(\checkmark\) \\ ds5 & 0.163 & **0.679** & 0.02 & 0.026 & 0.111 & k-NN & \(\checkmark\)(1) \\ ds6 & 0.028 & 0.007 & 0.008 & 0.142 & **0.815** & RF & \(\checkmark\)(2) \\ ds7 & 0.09 & **0.869** & 0.009 & 0.008 & 0.025 & RF & \(\checkmark\)(2) \\ ds8 & 0 & 0 & 0.006 & 0.078 & **0.916** & RF & \(\checkmark\)(2) \\ ds9 & 0.002 & 0.15 & 0 & **0.848** & 0 & k-NN & \(\checkmark\)(2) \\ ds10 & 0 & 0.004 & 0 & **0.996** & 0 & k-NN & \(\checkmark\)(2) \\ ds11 & 0.007 & **0.992** & 0 & 0 & 0.001 & RF & \(\checkmark\)(2) \\ ds12 & 0.002 & **0.998** & 0 & 0 & 0 & RF & \(\checkmark\)(2) \\ ds13 & 0 & 0 & 0 & **0.994** & 0.006 & k-NN & \(\checkmark\)(2) \\ ds14 & 0 & **1** & 0 & 0 & 0 & k-NN & \(\checkmark\)(1) \\ ds15 & 0 & 0 & 0 & 0 & **1** & RF & \(\checkmark\)(2) \\ ds16 & 0 & **0.98** & 0.02 & 0 & 0 & logit & \(\checkmark\) \\ ds17 & 0.028 & 0.444 & 0 & **0.528** & 0 & k-NN & \(\checkmark\)(2) \\ ds18 & 0.126 & **0.559** & 0.075 & 0.143 & 0.097 & RF & \(\checkmark\)(2) \\ ds19 & 0.038 & 0.003 & 0.004 & 0.016 & **0.939** & ANN & \(\checkmark\) \\ ds20 & 0.008 & 0.147 & 0 & **0.845** & 0 & k-NN & \(\checkmark\)(2) \\ ds21 & 0 & 0.069 & **0.931** & 0 & 0 & logit/SVM & \(\checkmark\)(1) \\ ds22 & 0.019 & 0.01 & **0.895** & 0.064 & 0.012 & RF & \(\checkmark\)(2) \\ ds23 & **0.99** & 0.008 & 0 & 0.002 & 0 & RF & \(\checkmark\) \\ \hline \hline \end{tabular}
\end{table}
Table 5: Predictions for the proposed CNN approach.
Table 6 shows that the alternative meta-learning approach is not as successful as the proposed CNN: recommendations are successful 65.2% of the time (15 of the 23 datasets). However, this approach clearly outperforms a random recommender.

The proposed method makes successful recommendations 78.2% of the time compared to 65.2% with the standard two-step approach (15 of the 23 datasets). From the 15 hits, 11 correspond to the second best model. In this sense, the alternative method is worse than the proposed CNN at identifying the best performing classifier.

## 5 Discussion and Conclusions

This paper proposes a novel meta-learning approach in which CNNs are used to learn directly from tabular datasets without the need to construct meta-features. The goal is to avoid any loss of information that results from a two-step approach. The proposed model can make accurate classifier recommendations for this type of dataset, outperforming a two-step approach based on meta-features.

The primary challenge is to define a comprehensible set of simulated patterns that can be extrapolated to real-world datasets. Results show that CNNs that are used in a supervised manner can achieve nearly perfect identification of simulated patterns; however, the capacity that the proposed model has to extrapolate this knowledge to real-world datasets requires further research. Fortunately, the experiments in this study show promising results.

There are some methodological challenges that can be improved. For example, we can use more sophisticated strategies to match datasets of different sizes on a CNN, such as spatial pyramid pooling (SPP). This method consists of a special type of pooling layer that allows the inclusion of images of heterogeneous sizes. An SPP layer pools the features in an image and generates outputs that feed the dense layers [26]. Also, different CNN architectures could be explored. These challenges, however, would have a minor impact in the proposed framework because the classification accuracy is already near 100% on the simulated datasets.

This study should be seen as a first attempt in the development of a recommender system that can defy the NFL theorem. Therefore, this study opens interesting lines for future research. Apart from the identification of new patterns that can be simulated and subsequently linked to classifiers, the proposed method can be used to learn from more sophisticated data structures, such as text or images. The proposed model can also be tailored to the class-imbalance problem, in which

\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline ID & C1 & C2 & C3 & C4 & C5 & bc & hit \\ \hline ds1 & & & & & \(\bigstar\) & k-NN & ✗ \\ ds2 & & & & & SVM & ✓(1) \\ ds3 & ✗ & & & & ANN/SVM & ✓(2) \\ ds4 & & & & & DT & ✗ \\ ds5 & & ✗ & & & k-NN & ✓(1) \\ ds6 & & & & & \(\bigstar\) & RF & ✓(2) \\ ds7 & & & & & \(\bigstar\) & RF & ✓(2) \\ ds8 & & & & & \(\bigstar\) & RF & ✓(2) \\ ds9 & & & & & \(\bigstar\) & k-NN & ✗ \\ ds10 & & & & & & k-NN & ✗ \\ ds11 & ✗ & & & & & RF & ✗ \\ ds12 & & ✗ & & & & RF & ✓(2) \\ ds13 & & & & & & k-NN & ✗ \\ ds14 & & & & & & k-NN & ✓(2) \\ ds15 & & & & & & RF & ✓(2) \\ ds16 & ✗ & & & & & logit & ✓(1) \\ ds17 & & & & & & k-NN & ✗ \\ ds18 & & & & & & RF & ✓(2) \\ ds19 & ✗ & & & & & ANN & ✓(2) \\ ds20 & ✗ & & & & & k-NN & ✓(1) \\ ds21 & ✗ & & & & & logit/SVM & ✗ \\ ds22 & & & & & & RF & ✓(2) \\ ds23 & & & & & & RF & ✓(2) \\ \hline \hline \end{tabular}
\end{table}
Table 6: Predictions for the tree-based meta-learning approach trained on meta-features.
classifiers and resampling strategies can be recommended. Finally, generative models can be considered to provide a wider diversity of training patterns.

## Acknowledgements

The authors gratefully acknowledge financial support from ANID, PIA/PUENTE AFB220003 and FONDECYT-Chile, grants 1200221 and 11200007.
