Deep neural models, such as convolutional and recurrent networks, achieve phenomenal results over spatial data such as images and text.
 However, when considering tabular data, gradient boosting of decision trees (GBDT) remains the method of choice.
 Aiming to bridge this gap, we propose \emph{deep neural forests} (DNF)
 --  a novel architecture that combines elements from decision trees as well as dense residual connections. 
 We present the results of extensive empirical study in which we examine the performance of GBDTs, DNFs and (deep) fully-connected networks. 
 These results indicate that DNFs achieve comparable results to GBDTs on tabular data, and open the door to end-to-end neural modeling of multi-modal data. To this end, we present a successful application of DNFs as part of a hybrid architecture for a multi-modal driving scene understanding classification task. While deep neural models have gained supremacy in many applications, it is often the case that the winning hypothesis class in learning problems involving tabular data is decision forests. Indeed, in Kaggle competitions, gradient boosting decision trees (GBDTs) (Chen & Guestrin, 2016; Friedman, 2001) are often the superior model. 1 Decision forest techniques have several distinct advantages: they can handle heterogeneous feature types, they are insensitive to feature scaling, and perhaps, most importantly, they perform a rudimentary kind of "feature engineering" automatically by considering conjunctions of decision stumps. These types of features may be a key reason for the relative success of GBDTs over tabular data. In contrast, deep neural models (CNNs, RNNs) have become the preeminent favorites in cases where the data exhibit a spatial proximity structure (namely, video, images, audio, and text) . In certain problems, such as image classification, by restricting the model to exploit prior knowledge of the spatial structure (e.g., translation and scale invariances), these models are capable of generating problem dependent representations that almost completely overcome the need for expert knowledge. However, in the case of tabular data, it is often very hard to construct (deep) neural models that achieve performance on the level of GBDTs. In particular, the "default" fully connected networks (FCNs), which do not reflect any specific inductive bias toward tabular data, are often inferior to GBDTs on these data. There have been a few works aiming at the construction of neural models for tabular data (see Section 2). However, for the most part, these attempts relied on conventional decision tree training in their loop and currently, there is still no widely accepted neural architecture that can effectively replace GBDTs. This deficiency prevents or makes it harder to utilize neural models in many settings and constitutes a lacuna in our understanding of neural networks. Our objective in this work is to create a neural architecture that can be trained end-to-end using gradient based optimization and achieve comparable or better performance to GBDTs on tabular data. Such an architecture is desirable because it will allow the treatment of multi-modal data involving both tabular and spatial data in an integrated manner while enjoying the best of both GBDTs and deep models. Moreover, while GBDTs can handle medium size datasets ("Kaggle scale"), they do not scale well to very large datasets ("Google scale"), where their biggest computational disadvantage is the need to store (almost) the entire dataset in-memory 2 (see Appendix C for details as well as a real-life example of this limitation). A purely neural model for tabular data, which is trained with SGD, should be scalable beyond these limits. A key-point in successfully applying deep models is the construction of architectures that contain inductive bias relevant to the application domain. This quest for appropriate inductive bias in the case of tabular data is not yet well understood (not to mention that there can be many kinds of tabular data). However, we do know that tree and forest methods tend to perform better than vanilla FCNs on these data. Thus, our strategy is to borrow properties of decision trees and forests into the network structure. We present a generic neural architecture whose performance can be empirically similar to GBDTs on tabular data. The new architecture, called Deep Neural Forest (DNF), combines elements from both decision forests and residual/dense nets. The main building block of the proposed architecture is a stack of neural branches (NBs), which are neural approximations of oblique decision branches that are connected via dense residual links (Huang et al., 2017) . The final DNF we propose is an ensemble of such stacks (see details in Section 3). We present an empirical study where we compare DNFs to the FCNs and GBDTs baselines, optimized over their critical parameters. We begin with a synthetic checkerboard problem, which can be viewed as a hypothetical challenging tabular classification task. We then consider several relatively large tasks, including two past Kaggle competitions. Our results indicate that DNFs consistently outperform FCNs, and achieve comparable performance to GBDTs. We also address applications of DNFs over multi-modal data and examine an integrated application of DNFs, CNNs and LSTMs over a multi-modal classification task for driving scene understanding involving both sensor recording and video (Ramanishka et al., 2018) . We show that the replacement of the FCN component by DNF in the hybrid deep architecture of Ramanishka et al. (2018) , which was designed to handle these multi-modal data, leads to significant performance improvement.
An architecture for tabular data, which emulates branches of decision trees and uses dense residual connectivity  This paper proposes deep neural forest, an algorithm which targets tabular data and integrates strong points of gradient boosting of decision trees. A novel neural network architecture mimicking how decision forests work to tackle the general problem of training deep models for tabular data and showcasing effectiveness on par with GBDT.