Neural Network (NN) has achieved state-of-the-art performances in many tasks within image, speech, and text domains. Such great success is mainly due to special structure design to fit the particular data patterns, such as CNN capturing spatial locality and RNN modeling sequential dependency. Essentially, these specific NNs achieve good performance by leveraging the prior knowledge over corresponding domain data. Nevertheless, there are many applications with all kinds of tabular data in other domains. Since there are no shared patterns among these diverse tabular data, it is hard to design specific structures to fit them all. Without careful architecture design based on domain knowledge, it is quite challenging for NN to reach satisfactory performance in these tabular data domains. To fill the gap of NN in tabular data learning, we propose a universal neural network solution, called TabNN, to derive effective NN architectures for tabular data in all kinds of tasks automatically. Specifically, the design of TabNN follows two principles: \emph{to explicitly leverages expressive feature combinations} and \emph{to reduce model complexity}. Since GBDT has empirically proven its strength in modeling tabular data, we use GBDT to power the implementation of TabNN. Comprehensive experimental analysis on a variety of tabular datasets demonstrate that TabNN can achieve much better performance than many baseline solutions. Recent years have witnessed the extraordinary success of Neural Networks (NN), especially Deep Neural Networks, in achieving state-of-the-art performances in many domains, such as image classification BID27 , speech recognition BID25 , and text mining BID22 . Beside enlarged model capacity, such great achievement of NN is mainly due to the deliberate design of its structures derived from prior knowledge over the certain domain data. For example, Convolutional Neural Networks (CNN) BID40 have become the standard solution to address image classification since it can capture the spatial locality by using "Local Receptive Field" BID40 , which is a common pattern in image data. Recurrent Neural Networks (RNN) BID29 , as another example, has been widely-used on speech recognition and language modeling because its recurrent structure can effectively model the sequential dependency among speech and text data.In contrast to most of tasks in image, speech, or text domains whose input yields natural spatial or temporal dimension, many other real-world applications, e.g., click through rate prediction BID24 , time series forecasting BID49 BID11 , web search ranking BID0 BID8 , etc, bear structured input consisting of multi-dimension meaningful features. Typically, such input data can be generalized as the tabular data, as each row of the tabular corresponds to one data example and each column denotes an individual meaningful feature. Despite the success of CNN and RNN over computer vision, speech recognition, and natural language process, adopting NN over tabular data receives far less attention and yet remains quite challenging. In particular, as illustrated in previous studies BID18 , it usually leads to unsatisfactory performance on tabular data by directly using Fully Connected Neural Network (FCNN), because its fully connected model structure leads to very complex optimization hyper-planes with a high risk of falling into local optimums. Moreover, since different applications usually indicate various effective feature combinations within their respective tabular data, it is quite beneficial to recognize such feature combinations and take advantage of them to design the effective NN model on their tabular data, which however has not been well studied yet.To address these challenges, we identify two principles for the purpose of designing effective NN models on tabular data: (1) To explicitly leverage expressive feature combinations. Rather than blindly pouring all features together into FCNN and learning via back-propagation to discover the implicit feature combinations, it will be beneficial to let NN explicitly leverage the expressive feature combinations. (2) To reduce model complexity. Contrary to highly-complex FCNN with too many parameters leading to higher risk of over-fitting or falling into local optimums, it is vital to reduce the complexity of NN models by removing unnecessary parameters and encouraging parameter sharing.Inspired by these two principles, we propose a universal neural network solution, called TabNN, to derive effective NN architectures for tabular data in all kinds of tasks automatically, by leveraging the knowledge learned by GBDT model (Gradient Boosting Decision Tree) BID19 BID15 BID12 , which has empirically proven its strength in modeling tabular data BID12 . More specifically, the GBDT-powered TabNN consists of four major steps: (1) Automatic Feature Grouping (AFG) automatically discovers feature groups implying effective partial combinations based on GBDT-powered knowledge. (2) Feature Group Reduction (FGR) attempts to further cluster feature groups in order to encourage parameter sharing within the same clusters, which can accordingly reduce the complexity of the resulting NN models. (3) Recursive Encoder with Shared Embedding (RESE) aims at designing a both effective and efficient NN architecture over clustered tabular feature groups, based on the results of FGR and the feature group importance powered by GBDT. (4) Transfer Structured Knowledge from GBDT (TSKG) further leverages structured knowledge within GBDT model to provide an effective initialization for the obtained NN architecture.To illustrate the effectiveness of the proposed TabNN solution, we conduct extensive experiments on various publicly available datasets with tabular data. Comprehensive experimental analysis has shown that TabNN cannot only create effective NN architectures for various tabular data but also achieves much better performance than other solutions.In summary, the contributions of this paper are multi-fold:• We identify two principles for the purpose of designing effective NN models on tabular data.• We propose TabNN, a general solution for deriving effective NN models for tabular data by leveraging the data knowledge learned by GBDT.• Extensive experiments show that the proposed method is an off-of-shelf model, which can be ready to use in any kinds of tabular data efficiently and achieves state-of-the-art performance. To fill the gap of NN in tabular data learning, we propose a universal neural network solution, called TabNN, which can derive the effective neural architectures automatically for tabular data. The design of TabNN follows two principles, one as explicitly leveraging expressive feature combinations and the other as reducing model complexity. Since GBDT is proven to be effective in tabular data, we leverage GBDT to power the implementation of TabNN. Specifically, TabNN first leverages GBDT to automatically identify expressive feature groups and then clusters feature groups into sets to encourage parameter sharing. After that, TabNN utilizes tree importance knowledge from GBDT to construct recursive NN architectures. To enhance the training efficiency and learning performance, tree structural knowledge is also utilized to provide an effective initialization for the derived architecture. Extensive experiments on various tabular datasets show the advantages of TabNN in modeling tabular data and demonstrate the necessity of designed components in TabNN.
We propose a universal neural network solution to derive effective NN architectures for tabular data automatically. A new Neural Network training procedure, designed for tabular data, that seeks to leverage feature clusters extracted from GBDTs. Proposal for a hybrid machine learning algorithm using Gradient Boosted Decision Trees and Deep Neural Networks, with intended research direction on tabular data.