Homological Convolutional Neural Networks
Deep learning methods have demonstrated outstanding performances on
classification and regression tasks on homogeneous data types (e.g., image,
audio, and text data). However, tabular data still pose a challenge, with
classic machine learning approaches being often computationally cheaper and
equally effective than increasingly complex deep learning architectures. The
challenge arises from the fact that, in tabular data, the correlation among
features is weaker than the one from spatial or semantic relationships in
images or natural language, and the dependency structures need to be modeled
without any prior information. In this work, we propose a novel deep learning
architecture that exploits the data structural organization through
topologically constrained network representations to gain relational
information from sparse tabular inputs. The resulting model leverages the power
of convolution and is centered on a limited number of concepts from network
topology to guarantee: (i) a data-centric and deterministic building pipeline;
(ii) a high level of interpretability over the inference process; and (iii) an
adequate room for scalability. We test our model on 18 benchmark datasets
against 5 classic machine learning and 3 deep learning models, demonstrating
that our approach reaches state-of-the-art performances on these challenging
datasets. The code to reproduce all our experiments is provided at
https://github.com/FinancialComputingUCL/HomologicalCNN.
Antonio Briola, Yuanrong Wang, Silvia Bartolucci, Tomaso Aste
# Homological Convolutional Neural Networks

###### Abstract

Deep learning methods have demonstrated outstanding performances on classification and regression tasks on homogeneous data types (e.g., image, audio, and text data). However, tabular data still poses a challenge with classic machine learning approaches being often computationally cheaper and equally effective than increasingly complex deep learning architectures. The challenge arises from the fact that, in tabular data, the correlation among features is weaker than the one from spatial or semantic relationships in images or natural languages, and the dependency structures need to be modeled without any prior information. In this work, we propose a novel deep-learning architecture that exploits the data structural organization through topologically constrained network representations to gain spatial information from sparse tabular data. The resulting model leverages the power of convolutions and is centered on a limited number of concepts from network topology to guarantee (i) a data-centric, deterministic building pipeline; (ii) a high level of interpretability over the inference process; and (iii) an adequate room for scalability. We test our model on \(18\) benchmark datasets against \(5\) classic machine learning and \(3\) deep learning models demonstrating that our approach reaches state-of-the-art performances on these challenging datasets. The code to reproduce all our experiments is provided at [https://github.com/FinancialComputingUCL/HomologicalCNN](https://github.com/FinancialComputingUCL/HomologicalCNN).

## 1 Introduction

We are experiencing a tremendous and inexorable progress in the field of deep learning. Such a progress has been catalyzed by the availability of increasing computational resources and always larger datasets. The areas of success of deep learning are heterogeneous. However, the three application domains where superior performances have been detected are the ones involving the usage of images [1; 2], audio [3; 4] and text [5; 6; 7] data. Despite their inherent diversity, these data types share a fundamental characteristic: they exhibit homogeneity with notable inter-feature correlations and evident spatial or semantic relationships. On the contrary, tabular data represent the "unconquered castle" of deep neural network models [8]. Tabular data are heterogeneous and present a mixture of continuous, categorical, and ordinal values which can be either independent or correlated. They are characterized by the absence of any inherent positional information, and tabular models have to handle features from multiple discrete and continuous distributions. Tabular data
are the most common data format and are ubiquitous in many crucial applications, such as medicine [9, 10, 11, 12], finance [13, 14, 15, 16], recommendation systems [17, 18, 19, 20], cybersecurity [21, 22], anomaly detection [23, 24, 25, 26] and so forth. During the last decade, traditional machine learning methods dominated tabular data modeling, and nowadays, tree ensemble algorithms (e.g. XGBoost, LightGBM, CatBoost) are considered the recommended option to solve real-life problems of this kind [27, 28, 29].

In this paper, we introduce a novel deep learning architecture for tabular numerical data classification and we name it "Homological Convolutional Neural Network" (HCNN). The building process is entirely centered on the structural organization of input data obtained through network representations that allow gaining spatial information from tabular data. A network (or graph) represents components of a system as nodes (or vertices) and interactions among them as links (or edges). The number of nodes defines the size of the network, and the number of links determines the network's sparsity (or, conversely, density). Reversible interactions between components are represented through undirected links, while non-reversible interactions are represented as direct links [30]. In this research work, we exploit a class of information filtering networks [31], namely the Triangulated Maximally Filtered Graph [32], to model the inner sparsity of tabular data and obtain a geometrical organization of input features. The choice of the network representation is not binding, even if limited to the family of so-called simplicial complexes [33]. Simplicial complexes are generalized network structures that allow capturing many-body interactions between the constituents of complex systems [34]. They are formed by sets of simplices such as nodes, links, triangles, tetrahedra, and so on, glued to each other along their faces forming higher-order graphs [33]. These graphs connect not only vertices (\(0\)-dimensional simplices) with edges (\(1\)-dimensional simplices) but also higher-order simplices (e.g. triangles, \(2\)-dimensional simplices, and tetrahedra, \(3\)-dimensional simplices). The study of networks in terms of the relationship between structures at different dimensionality is a form of "homology" and HCNNs keep into account higher-order interactions in data dependency structure as homological priors. During the neural network's building process, given a proper network representation of input data, we isolate all the simplicial structures with dimension \(\geq 1\) and we process them at two granularity levels: (i) across single representatives of each simplicial structure (i.e. convolution over each edge, triangle, tetrahedron); and (ii) across representatives of each simplicial structure (i.e. convolution over all the transformed edges, all the transformed triangles, and all the transformed tetrahedra). In doing so, we capture both the simplicial and the homological structure of input data also searching for non-trivial structural data relationships. This methodology allows to find localities in tabular data and leverages the power of Convolutional Neural Networks (CNNs) to effectively model their sparsity. Compared to its state-of-the-art (SOTA) machine learning alternatives, our method (i) maintains an equivalent level of explainability; (ii) has a comparatively low level of computational complexity; and (iii) can be scaled to a higher number of learning tasks (e.g. time series forecasting) without structural changes. Compared to its SOTA deep-learning alternatives, our method (i) is data-centric (i.e. the architecture depends on the data describing the system under analysis), (ii) presents an algorithmic data-driven building pipeline, (ii) it has a lower complexity replacing complex architectural modules (e.g. attention-based mechanisms) with elementary computational units (e.g. convolutional layers). We provide a comparison between HCNNs, simple-to-advanced machine learning algorithms and SOTA deep tabular architectures using a heterogeneous battery of small-sized numerical benchmark datasets. We observe that HCNN always ties SOTA performances on the proposed tasks, providing, at the same time, structural and computational advantages.

The rest of the paper is organized as follows. In Section 2 we review the previous research on information filtering networks, sparsity handling in deep learning, and automated learning for tabular data. In Section 3.1 we discuss the data acquisition and transformation pipeline. In Section 3.2 we introduce the basic concepts about network science and information filtering networks. In Section 3.3 we provide the background for Homological Neural Networks. In Section 3.4 we present the working mechanism of Homological Convolutional Neural Networks. In Section 3.5, we provide the mathematical justification for the proposed methodology. In Section 4, we explore the effectiveness of Homological Convolutional Neural Networks compared to SOTA machine learning and deep learning models. Finally, in Section 5, we interpret our results and discuss future research lines in this area.
Related Work

**Information Filtering Networks**. The search for increasingly sophisticated sparse network representations of heterogeneous data types is an active area of research. During the past three decades, Information Filtering Networks (IFNs) [35; 36; 31; 32; 37] emerged as an effective tool in this research field. Their effectiveness has been demonstrated in many application domains, including but not limited to finance [38; 39; 40; 41; 42], psychology [43; 44], medicine [45; 46] and biology [47; 48]. However, in many cases, the power of IFNs has been limited to descriptive tasks. More recently, considerable efforts have been spent to make them active modeling tools. In this sense, the work by [49] suggests to use IFNs to perform topological regularization in multivariate probabilistic modeling with both linear and non-linear multivariate probability distributions; the work by [50] proposes a new unsupervised feature selection algorithm entirely based on the study of the relative position of nodes inside the above mentioned constrained network representations; while the work by [39] suggests a first integration of IFNs into articulated pipelines involving also complex deep learning architectures. The latest milestone is represented by the introduction of Homological Neural Networks (HNN) [51], where the authors propose a pioneering methodology to extract a versatile computational unit directly from the IFNs' network representation.

**Sparsity in Deep Learning**. Recent advances in many deep learning related fields [52; 53; 54; 55] came with an increasing demand for computational resources. The growing energy costs have driven the community to search for new models with reduced size which heavily rely on selective pruning of redundant connections. Indeed, sparse neural networks have been found to generalize just as well (sometimes even better) than original dense networks while reducing the memory footprint and shortening training time [56]. Even if large, the landscape of approaches to sparsify deep neural network models can be schematically organized considering six main categories: (i) down-sizing models [57; 58; 59]; (ii) operator factorization [60; 61; 62]; (iii) value quantization [63; 64; 65], (iv) value compression [66; 67]; (v) parameter sharing [68]; and (vi) sparsification [69; 70; 71; 72]. All these approaches intend sparsity as a concept to refer to the proportion of neural network weights that are zero-valued. Higher sparsity corresponds to fewer weights, and smaller computational and storage requirements. Based on this, the weights pruning phase can be at (i) at initialization [73]; (ii) after training [74]; or (iii) while training [75]. The current research work introduces a unique approach to neural network sparsification, which emphasizes the pruning of weak relationships during the data modeling stage. This approach involves constructing a lightweight neural network architecture that adapts its structure to a sparse representation of input data. In this sense, the sparsification process occurs before the initialization stage. The most similar solution to this is represented by Simplicial NNs [76] and Simplicial CNNs [77]. Indeed, these architectures constitute the very first attempt to exploit the topological properties of sparse graph representations to capture higher-order data relationships. Despite their novelty, the design of these neural network architectures limits them to pre-designed network data, without the possibility to easily scale to more general data types (e.g., tabular data).

**Tabular Learning**. Traditionally the field of tabular data learning has been widely dominated by classic machine learning methods. Among them, ensembles of decision trees (DTs), such as GBDT (Gradient Boosting Decision Tree) [27; 78], represent the top choice for both practitioners and academics. The prominent strength of DTs is the efficient picking of global features with a high rate of statistical information gain [79], while their ensemble guarantees a generalized performance improvement by reducing variance [80]. GBDT is an algorithm in which new week learners (i.e. decision trees) are created from previous models' residuals and then combined to make the final prediction. Several GBDT variations exist, including XGBoost [78], LightGBM [81] and CatBoost [28]. Extended studies demonstrated how, despite their differences, the performance of these algorithms on many tasks is statistically equivalent [28]. In the last decade, several studies proposed novel deep learning architectures explicitly designed to solve tabular problems [82; 83; 84; 85; 86; 87; 80; 88]. These models can be roughly categorized into five groups: (i) differentiable trees; (ii) attention-based model; (iii) explicit modeling of multiplicative interactions; (iv) regularization methods; and (v) convolutions-based approaches. Differentiable tree models leverage the power of classical decision trees proposing smoother decision functions, which make them differentiable [83; 86; 87]. Attention-based models suggest exploiting the power of attention mechanisms [88; 89] by integrating it into tabular deep learning architectures [80; 84; 90; 91]. Methods that explicitly model multiplicative interactions try to incorporate feature products into Multilayer Perceptron models [92; 93; 94]. Regularization methods leverage large-scale hyper-parameter tuning schemes to learn a "regularization strength" for every
neural weight [95; 8; 96]. Finally, convolutions-based approaches leverage the power of CNNs in tabular learning problems. The two most significant attempts in this sense are the work by [97], where tabular data are reshaped directly into a multi-channel image forman letting the model learn the correct features sorting through back-propagation, and the work by [98], where tabular data are transformed into images by minimizing the difference between the ranking of distances between features and the ranking of distances between their assigned pixels in the image. Despite these attempts, there is still an active debate over whether or not deep neural networks generally outperform gradient-boosted decision trees on tabular data, with multiple works arguing either for [80; 95; 87; 99] or against [11; 100; 101; 29] neural networks [102].

## 3 Data and Methods

### Data

To provide a fair comparison between HCNN and SOTA models, we use a collection of \(18\) tabular numerical datasets (see Appendix A) from the open-source "OpenML-CC18" benchmark suite [103]. Following the selection criteria in [104], all the datasets contain up to \(2000\) samples, \(100\) features, and \(10\) classes. A deep overview on the properties of this first set of data is provided in Appendix A. Training/validation/test split is not provided. For all the datasets, the \(50\%\) of the raw dataset is used as a training set, the \(25\%\) as validation set, and the remaining \(25\%\) as a test set. To prove the statistical significance of results presented in the current research work, all the analyses are repeated on \(10\) different combinations of training/validation/test splits. The reproducibility of results is guaranteed by a rigorous usage of seeds (i.e. \([12,190,903,7687,8279,9433,12555,22443,67822,9822127]\)). Following [101], we focus on small datasets because of two main reasons: (i) small datasets are often encountered in real-world applications [105] and (ii) existing deep learning methods are limited in this domain.

It is worth noting that, differently from other deep learning architectures (e.g. [104; 80]), the applicability of HCNNs is not limited to small tabular data problems and can easily scale to medium-to-large problems. To provide evidence of this, we use a collection of 9 numerical tabular datasets (see Appendix A) from the "OpenML tabular benchmark numerical classification" suite [101]. All these datasets violate at least one of the selection criteria in [104] (i.e. they are characterized by a number of samples \(>2000\) or they are characterized by a number of features \(>100\)). A more detailed overview of the properties of this second set of data is provided in Appendix A.

### Information Filtering Networks

The HCNN's building process is entirely centered on the structural organization of data emerging from the underlying network representation. The choice of the network representation is not binding even if limited to the family of simplicial complexes [33]. In this paper, we exploit the power of a class of information filtering networks (IFNs) [35; 36; 31; 32; 37], namely the Triangulated Maximally Filtered Graph (TMFG) [32], to model the inner sparsity of tabular data and obtain a structural organization of input features. IFNs are an effective tool to represent and model dependency structures among variables characterizing complex systems while imposing topological constraints (e.g. being a tree or a planar graph) and optimizing specific global properties (e.g. the likelihood) [49]. Starting from a system characterized by \(n\) features and \(T\) samples, arranged in a matrix \(\mathbf{X}\), this methodology builds a \(n\times n\) similarity matrix \(\hat{\mathbf{C}}\) which is filtered to obtain a sparse adjacency matrix \(A\) retaining only the most structurally significant relationships among variables.

The introduction of TMFG is a milestone in the IFNs' research area. The building process of TMFG (see Appendix B) is based on a simple topological move that preserves planarity (i.e. a graph is planar if it can be embedded on the surface of a sphere without edges crossing): it adds one node to the center of three-nodes cliques by using a score function that maximizes the sum of the weights of the three edges connecting the existing vertices. This addition transforms three-node cliques (i.e. triangles) into four-node cliques (i.e. tetrahedra) characterized by a chord (i.e. an edge that is not part of the cycle but connects two vertices of the cycle itself) that is not part of the clique but connects two nodes in the clique, forming two triangles and generating a chordal network (a graph is said to be chordal if all cycles made of four or more vertices have a chord, reducing the cycle to a set of triangles [106]) [43]. As with all chordal graphs, the TMFG fulfills the independence assumptions of
Markov and Bayesian networks [107; 43]. It has \(n\) nodes, where \(n\) is the cardinality of the set of input features and \(3n-6\) edges. A nested hierarchy emerges from its cliques [108]: compared to the fully connected graph represented by \(\hat{\mathbf{C}},\boldsymbol{A}\)'s density is reduced in a deterministic manner while the global hierarchical structure of the original network is retained. The TMFG presents three main advantages: (i) it can be used to generate sparse probabilistic models as a form of topological regularization [49]; (ii) it is computationally efficient and (iii) allows to find maximal cliques in polynomial time although the problem is NP-complete for general graphs. On the other hand, the two main limitations of chordal networks are that (i) they may add unnecessary edges to satisfy the property of chordality; and (ii) their building cost can vary based on the chosen optimization function.

Working with numerical-only, tabular data, in the current paper, \(\hat{\mathbf{C}}\) corresponds to a matrix of squared correlation coefficients. It is worth noting that, while characterizing cross-correlations, one could face statistical uncertainty due to many reasons including, but not limited to the noise in the data and the intrinsic complexity of interactions among variables of the system. Attempts to overcome these problems may require filtering out statistically reliable information from the correlation matrix. Spectral analysis [109; 110; 111], clustering [112] and graph theory [113] demonstrated to be fruitful approaches to efficiently handle this problem [114; 35; 115]. In line with the work by [116], in the current paper, we use the bootstrapping approach [117; 118]. This technique requires to build a number \(r\) of replicas \(X_{i}^{*}\), \(i\in 1,\ldots,r\) of the data matrix \(\mathbf{X}\). Each replica \(X_{i}^{*}\) is constructed by randomly selecting \(T\) rows from the matrix \(\mathbf{X}\) allowing for repetitions. For each replica \(X_{i}^{*}\) the correlation matrix \(\hat{\mathbf{C}}_{i}^{*}\) is then computed. We highlight that (i) the bootstrap approach does not require the knowledge of the data distribution and (ii) it is particularly useful to deal with high dimensional systems where it is difficult to infer the joint probability distribution from data. Once obtained replicas-dependent correlation matrices, we treat them in two different ways:

* We compute \(\hat{\mathbf{C}}\) as the entry-wise mean of correlation matrices \(\hat{\mathbf{C}}_{i\in 1,\ldots,r}^{*}\).
* Based on each replica-dependent correlation matrix \(\hat{\mathbf{C}}_{i}^{*}\), we compute a TMFG\({}_{i}^{*}\) and we obtain the final TMFG by taking only the links that appear in all the TMFGs with a frequency higher than a specified threshold.

In the rest of the paper, we refer to the first configuration as MeanSimMatrix and to the second one as BootstrapNet. These two approaches lead to widely different results. In the former case, the final TMFG will be a sparse, connected graph that necessarily maintains all the topological characterization of the family of IFNs it belongs to (i.e. planarity and chordality). In the latter case, instead, there will be no guarantee on the connectedness of the graph. Indeed, the chosen threshold could lead to disconnected components and to the removal of edges assuring the graph's chordality.

### Homological Neural Networks

The main idea behind IFNs is to explicitly model higher-order sub-structures, which are crucial for the representation of the underlying system's interactions. In the case of TMFG, a simple higher-order representation can be obtained by adding triplets (triangles) and quadruplets (tetrahedra) to the set of nodes in the network. However, the associated higher-order graph is hard to be handled both visually and computationally. As a solution to this problem, in the work by [51], the authors start from a layered representation (i.e. the Hasse diagram), which explicitly takes into account higher-order sub-structures and their interconnections, and show how to easily convert this representation into a stand-alone computational unit named Homological Neural Network (HNN). Specifically, to represent the complexity of a higher order network (i.e. a TMFG), the authors propose to adopt a layered structure. As shown in Figure 1 nodes in layer \(d\) represent \(d\)-dimensional simplices (i.e. \(0\)-dimensional simplices are nodes, \(1\)-dimensional simplices are edges, \(2\)-dimensional simplices are triangles, \(3\)-dimensional simplices are tetrahedra). The structure starts with the vertices in layer \(0\); couples of vertices connect to edges, which are represented in layer \(1\); edges connect to triangles, which are represented in layer \(2\); triangles connect to tetrahedra, which are represented in layer \(3\), and so on. The resulting deep neural network is a sparse Multilayer Perceptron (MLP) with a one-to-one correspondence with the original network representation, explicitly retaining the simplices and their interconnection in the structure. All information about the network at all dimensions is explicitly encoded in this representation, including elements such as maximal cliques, separators, and their multiplicity.
### Homological Convolutional Neural Networks

Despite the undeniable advantages deriving from the sparse structure provided by HNNs, results in [51] suggest that the choice of the Multilayer Perceptron as deep learning architecture to process the information encoded in the underlying network representation is sub-optimal (especially for tabular data problems). In addition to this, HNNs impose the chordality of the underlying network and the building process of the deep neural network architecture implies the usage of non-native components inducing a substantial computational overhead. In this research work, we propose an alternative computational architecture that aims to solve these issues and we name it "Homological Convolutional Neural Network" (HCNN).

Given the adjacency matrix \(\mathbf{A}\) constructed using IFNs (see Section 3.2), to model the complexity embedded in the network representation, we isolate \(3\) different simplicial families: (i) maximal cliques with size \(4\) (i.e. the \(3\)-dimensional simplices or tetrahedra), (ii) maximal cliques with size \(3\) (i.e. the \(2\)-dimensional simplices or triangles) and (iii) maximal cliques with size \(2\) (i.e. the \(1\)-dimensional simplices or edges). When using the TMFG as network representation, these \(3\) structures are sufficient to capture all the higher-order dependency structures characterizing the underlying system. Each input of the novel deep learning architecture is hence represented by \(3\) different \(1\)-\(d\) vectors that we call \(H\) (i.e. realizations of the input features belonging to at least one tetrahedron), \(R\) (i.e. realizations of the input features belonging to at least one triangle), \(E\) (i.e. realizations of the input features belonging to at least one edge).

As a first step, in HCNN, we perform a \(1\)-\(d\) convolution across each set of features defining a realization of a simplicial family. We use a kernel size and a stride equals to \(d+1\) (i.e. the dimension of the simplicial structure itself), and a number of filters \(\zeta\in[4,8,12,16]\). This means that, given the three input vectors \(H\), \(R\) and \(E\) representing the three simplicial families characterizing a TMFG, we compute a \(1\)-\(d\) convolution with a kernel size and a stride of \(2\), \(3\) and \(4\) respectively for edges, triangles, and tetrahedra. The usage of stride is necessary to prevent the "parameter sharing". While generally considered an attractive property as fewer parameters are estimated and overfitting is avoided, in our case parameter sharing leads to inconsistencies. Indeed, geometrical structures belonging to the same simplicial family (i.e. edges, triangles, and tetrahedra) but independent in the hierarchical dependency structure of the system would share parameters, which is obviously wrong. After the \(1^{st}\)-level

Figure 1: Pictorial representation of an HNN and its building pipeline. From left to right, (i) we start from a chordal graph representing the dependency structures of features in the underlying system, (ii) we re-arrange the network’s representation to highlight the underlying simplicial complex structures (i.e. edges, triangles, tetrahedra), and (iii) we finally report a layered representation, which explicitly takes into account higher order sub-structures and their interconnections, and can be easily converted into a computational unit (i.e. a sparse MLP).
convolutions, which extract element-wise information from geometrical structures belonging to the same simplicial family, we apply a \(2^{nd}\)-level convolutions extracting homological insights. Indeed, the convolution is applied to the output of the first layer, extracting information related to entities belonging to the same simplicial family, which are not necessarily related in the original network representation. In this case, we use a kernel with a size equal to the cardinality of the simplicial family (i.e. \(|E|\), \(|R|\), \(|H|\) respectively) and a number of filters \(\xi\in[32-64]\) with a skip factor of \(4\). The final layer of the HCNN architecture is linear and maps the outputs from the \(2^{nd}\)-level convolutions to the output. It is worth noting that each level of convolution is followed by a regularization layer with a dropout rate equal to \(0.25\) and the non-linear activation function is the classic Rectified Linear Unit (ReLU).

Even if HNN and HCNN are built on the concept of homology and exploit it in the construction of a data-centric neural network unit, it is worth noting that their design aims to capture different data relationships. In the case of HNN different aggregation layers aim to capture relationships between increasingly complex geometrical structures, which are linked together through at least one edge in the network representation of the system under analysis. If two geometrical structures are not linked, then any potential relationship is missed. This architectural philosophy is maintained in HCNN and is fully captured in the \(1^{st}\)-level of convolution, where we model interactions embedded in unitary geometrical structures. In so doing, we capture the information contained in all the representatives of each simplicial family since the convolution is iterated for each size of higher-order structures. This step is highly eased if the input network is chordal. Indeed, this allows to have increasingly complex structures containing all the possible substructures. The chordality property has also additional advantages. The building process of this first layer of the data-centric unit can be built in polynomial time. This property is based on the fact that the recognition of maximal cliques of different sizes and the maximal clique of a chordal graph can be found in polynomial time although the problem is NP-complete for general graphs [119]. In the second and third layer of aggregation, in HCNNs, we aim to capture homological relationships characterizing the underlying system. Specifically, they

Figure 2: Pictorial representation of an HCNN and its building pipeline. From left to right, (i) we start from a chordal graph representing the dependency structures of features in the underlying system (the choice of the network representation is not binding), (ii) we isolate the maximal cliques corresponding to \(1\)-, \(2\)- and \(3\)-dimensional simplices (i.e. edges, triangles, tetrahedra) and we group them into \(1-d\) vectors containing features’ realizations, (iii) we compute a \(1\)-\(d\) convolution which extracts simplicial-wise non-linear relationships, (iv) we compute a \(2^{nd}\)-level convolution, which operates on the output of the previous level of convolution across all the representatives of each simplicial family extracting a first class of non-trivial homological insights, (v) we finally apply a linear map from the \(2^{nd}\)-level convolutions to the output extracting a second class of cross-network homological insights.
allow us to overcome the limits imposed by any network structure, by capturing potential hidden data dependency structures.

### On the learning process of network's representation

In the problem setting described in Section 3.4, we are dealing with a computational system \(\mathcal{M}_{\mathcal{G}}\), the HCNN, which depends on a network representation \(\mathcal{G}\). To discover the best network representation, in principle, one needs to explore the ensemble of all possible networks and identify the one that makes the model perform best. This problem is known to be NP-hard [119]. However, one can restrict the search space and identify a priori the kind of optimal network by analysing the dependency structure of the features of the system under analysis. From an information theoretic perspective, the general problem consists in finding the multivariate probability density function with representation structure \(\mathcal{G}\), \(\hat{f}(\mathbf{X}|\mathcal{G})\), that best describes the "true" underlying distribution \(f(\mathbf{X})\) (which is unknown). To quantify the distance between a model, \(\hat{f}(\mathbf{X}|\mathcal{G})\), and the true distribution, \(f(\mathbf{X})\), one can use the Kullback-Leibler divergence [120]

\[D_{KL}(f\parallel\hat{f})=\mathbb{E}(\log f(\mathbf{X}))-\mathbb{E}(\log\hat {f}(\mathbf{X}|\mathcal{G})), \tag{1}\]

which must be minimized. The first term of Equation 1 is independent of the model and therefore its value is irrelevant to the purpose of discovering the representation network. The second term, \(-\mathbb{E}(\log\hat{f}(\mathbf{X}|\mathcal{G}))\) (note the minus), instead depends on \(\mathcal{G}\) and must be minimized. This term is the estimate of the entropy of the multivariate system of variables \(\mathbf{X}\) by using the model \(\hat{f}(\mathbf{X}|\mathcal{G})\):

\[\hat{H}(\mathbf{X}|\mathcal{G})=-\mathbb{E}(\log\hat{f}(\mathbf{X}|\mathcal{G })) \tag{2}\]

and corresponds to the so-called cross-entropy. Given that the true underlying distribution is unknown, the expectation cannot be computed exactly, however, it can be estimated with arbitrary precision using the sample mean. Such a sample mean approximates the expected value of the negative log-likelihood of the model \(\hat{f}(\mathbf{X}|\mathcal{G})\). Therefore, the construction of the representation network must aim to maximize the likelihood of the model, which is indeed a typical quantity that is maximized when training a model.

The network associated with the largest model's likelihood can be constructed step-by-step by joining disconnected parts that share the largest mutual information. Indeed, in a graph, the gain achieved by joining two variables \(X_{a}\) and \(X_{b}\), is approximately given by the mutual information shared by the two variables \(\simeq I(X_{a};X_{b})\). In turn, at the second-order approximation, the mutual information is approximated by the square of the correlation coefficient between the two variables. Therefore, the gain in the model's likelihood is \(I(X_{a};X_{b})\simeq\rho_{a,b}^{2}\)[121], and the TMFG construction with \(\rho^{2}\) weights implies a graph that aims to maximize the model's likelihood itself.

## 4 Experiments

In this section, we compare the performance of HCNN classifier in its MeanSimMatrix and BootstrapNet configuration (see Section 3.2) against \(8\) machine learning and deep learning SOTA classifiers under homogeneous evaluation conditions. We consider LogisticRegression, RandomForest, XGBoost, LightGBM and CatBoost as representatives of machine learning classifiers, and MLP, TabNet and TabPFN as representatives of deep learning classifiers. For each of them, the inference process is structured into two different phases: (i) the hyper-parameters search stage and (ii) the training/test stage with optimal hyper-parameters. Both stages are repeated \(10\) times with fixed seeds that guarantee a full reproducibility of results. For each run, we allow for a maximum of \(500\) hyper-parameters search iterations allocating \(8\) CPUs each with \(2\)GB memory and a time budget of \(48\) hours. Experiments are entirely run on the University College London HPC CS Cluster [122]. The hyper-parameters search phase consists of a Sequential Model-Based Optimization with the Tree Parzen Estimator [123], where we maximize the F1_score on each validation set. In Appendix C, we describe the hyper-parameters' search space for each classifier. We use three metrics to evaluate the classifiers on out-of-sample datasets: the F1_score, the Accuracy, and the Matthews Correlation Coefficient (MCC) [124; 125]. The results obtained are statistically validated using the Wilcoxon significance test, a standard metric for comparing classifiers across multiple datasets [126].
As second stage of the analysis, we investigate the scalability of each model in tackling extensive numerical tabular classification tasks. In so doing, we use an ad-hoc suite of datasets (see Section 3.1), while maintaining the inference process described earlier in this section. A model converges (i.e. it is able to scale to larger classification problems) once completing the learning task using the given computational resources in the allocated time budget for all the \(10\) seeds.

### Small tabular classification problems

Table 4.1 reports a cross-datasets, out-of-sample comparison of classifiers previously listed in this Section. For each model, we provide (i) the average and (ii) the best/worst ranking position considering three different evaluation metrics, (iii) the average value for each evaluation metric, and (iv) the time required for the hyper-parameters tuning and for the training/test run with optimal hyper-parameters.

On average, the TabPFN model occupies a ranking position higher than the one of HCNN both in its MeanSimMatrix and in BootstrapNet configuration. However, it is worth noting that when we evaluate models' performance through F1_Score and MCC (i.e. the two performance metrics that are less prone to bias induced by unbalanced datasets), the HCNN in the MeanSimMatrix configuration occupies a ranking position for the worst performance, which is better than the one of its immediate competitor (i.e. \(7\) and \(6\) of HCNN MeanSimMatrix vs \(10\) and \(8\) of TabPFN). The same happens in the case of HCNN BootstrapNet with F1_score. These findings highlight an evident robustness of the HCNN model, which is superior not only to TabPFN model but also to all the other deep learning and machine learning alternatives. More generally, both TabPFN and HCNN show superior performance compared to the other two deep learning models (i.e. MLP and TabNet), which occupy an average ranking position equal to \(~{}7\) and \(~{}9\) respectively on all the three different evaluation metrics. Among machine learning models, CatBoost achieves the highest performance with an average ranking position equal to \(~{}4\) considering the F1_Score and the MCC, and equal to \(~{}5\) considering the Accuracy (in this case the position number \(4\) is occupied by LogisticRegression). All these findings can be visualized in Figure 4.1. Specifically, the highest robustness of HCNN model in MeanSimMatrix configuration compared to TabPFN model can be observed in Figure 3(a) and Figure 3(c). They represent the ranking position of each model on each dataset using the F1_Score and the MCC as performance metrics respectively. In the first case, we notice that the ranking position of the worst performance by HCNN is \(7\) when dealing with dataset "climate-model-simulation-crashes" (OpenML ID \(40994\)), while the one occupied by TabPFN is \(10\) with dataset "pc_1" (OpenML ID \(1068\)). In the second case, we notice that the worst performance by HCNN has ranking \(6\) when dealing with datasets "mfeat-karhunen" (OpenML ID \(16\)), "steel-plates-fault" (OpenML ID \(40982\)) and "climate-model-simulation-crashes" (OpenML ID \(40994\)), while the one occupied by TabPFN is \(8\) with dataset "pc_1" (OpenML ID \(1068\)). Except for "mfeat-karhunen" dataset (OpenML ID \(16\)), all the datasets listed before are strongly unbalanced.

Models' numerical performances for each evaluation metric enforce all the findings discussed above. It is, however, clear that the differences in performance are very reduced. This evidence suggests a potential statistical equivalence of the models and this hypothesis is verified through a specific statistical test discussed later in this Section.

The final comparison to be performed is the one related to the models' running time. In this sense, machine learning models still represent the SOTA with CatBoost being an exception. Among deep

\begin{table}
\begin{tabular}{c c c c c c c c c c c} \hline \hline  & **LogileRegression** & **RandomForest** & **XCBoost** & **LightGBM** & **Cellboard** & **MLP** & **TabNet** & **TabPFN** & **BICNN** \\ \hline
**3.7.max T1.Score** & 5.533 & 5.333 & 5.380 & 5.277 & 4.500 & 9.500 & 7.584 & 2.538 & 4.588 & 3.722 \\
**3.7.max Accuracy** & 4.838 & 5.972 & 6.914 & 5.916 & 5.388 & 9.646 & 7.166 & 1.833 & 4.694 & 3.277 \\
**3.7.max MCC** & 5.166 & 6.388 & 5.611 & 5.646 & 4.833 & 9.500 & 7.313 & 2.166 & 4.777 & 3.556 \\ \hline
**3.7.max T1.Score** & 1.79 & -1.00 & 1.79 & -1.8 & 1.5 & 8.10 & -2.10 & -1.70 & -1.79 & -2.7 \\
**2.8/8.5** & -1.10 & -1.0 & 3.9 & -2.8 & -1.8 & 8.10 & -2.10 & -1.5 & -1.9 & -1.6 \\
**3.7.max MAC** & -1.10 & -2.00 & -1.9 & 1.8 & 1.8 & 8.10 & -2.10 & -1.8 & 1.9 & -2.6 \\ \hline
**3.7.max T1.Score** & 0.791 & 0.771 & 0.771 & 0.771 & 0.771 & 0.771 & 0.771 & 0.771 & 0.771 & 0.511 & 0.301 & 0.301 & 0.301 \\
**3.7.max T1.Score** & 0.861 & 0.861 & 0.879 & 0.874 & 0.862 & 0.861 & 0.
learning models, however, it is worth noting that HCNN has a running time that is comparable with the one of TabPFN and much lower than the other attention-based model, TabNet. This result is relevant since it legitimates the architecture proposed in this paper as a strong competitor of TabPFN. Indeed, the proposed architecture reaches comparable results without pre-training, with a higher level of explainability in the architectural building process and with a much lower number of parameters. Also among deep learning models, there is an exception represented by the MLP: its SOTA running time heavily depends on the number of layers and on the number of neurons per layer emerging from the hyper-parameter search.

Figure 4.1 reports the relationship between the number of features and the total number of parameters in the HCNN MeanSimMatrix configuration, the relationship between the number of features and the total number of parameters in the HCNN BootstrapNet and the relationship between the difference in the number of features and the difference in the total number of parameters in the two configurations. Looking at Figure 4(a), it is possible to conclude that a strong linear relationship exists between the number of features and the total number of parameters of the HCNN model in the MeanSimMatrix. This finding was expected since the proposed model's architecture totally depends on the complete homological structure of the underlying system. This means that each time a new feature is introduced, we could potentially observe an increase in the number of edges, triangles, and tetrahedra which in turn determines a proportional increase in the number of parameters of the HCNN itself.

Figure 3: Out-of-sample model- and dataset-dependent average ranking considering (a) F1_Score (b) Accuracy (c) MCC evaluation metrics. This representation allows to clearly assess the higher robustness of HCNN model to datasets’ unbalance, over all its deep learning and machine learning competitors.
On this point, we need to underline that the magnitude of the slope of the regression line heavily depends on the optimal hyper-parameters describing the number of filters in the two convolutional layers. Looking at Figure 4(b), we observe again a relatively strong linear relationship between the number of features and the total number of parameters of the HCNN model in the BootstrapNet. The difference in r_value between the two configurations is equal to \(0.19\) and depends on the fact that, in the second case, the optimal threshold value, which maximizes the model's performance, is different across datasets, it does not depend on the number of inputs features and determines an ablation of features that has no dependence on any other factor. More generally, in the BootstrapNet configuration, we observe a number of parameters that is, on average, one order of magnitude below the one in the HCNN MeanSimMatrix configuration. To better study this finding, in Figure 4(c) we report, on the \(x\)-axis the difference in the number of features \(\Delta_{f}\) and on \(y\)-axis the difference in the number of parameters \(\Delta_{p}\). As one can see, the linear relationship is strong only when the two deltas are low. For higher deltas, specifically for the three datasets "mfeat-fourier" (OpenML ID \(14\)), "mfeat-karhunen" (OpenML ID \(16\)), and "analcaddata_authorship" (OpenML ID \(458\)), even if the decrement is significant for both parameters, the relationship is not linear.

To assess the statistical significance of the difference in models' performance, we use the Critical Difference (CD) diagram of the ranks based on the Wilcoxon significance test (with \(p\)-values below \(0.1\)), a standard metric for comparing classifiers across multiple datasets [126]. The overall empirical

Figure 4: Study of the relationship between the number of total features (\(x\)-axis) and a number of total parameters (\(y\)-axis) for the (a) HCNN MeanSimMatrix and (b) HCNN BootstrapNet configuration. Figure (c) reports the relationship structure among the difference in the number of features (\(x\)-axis) and the difference in the number of total parameters (\(y\)-axis) when using the two above-mentioned configurations.
comparison of the methods is given in Figure 4.1. We notice that the performance of HCNN and TabPFN is not statistically different. This finding is coherent across the three different evaluation metrics. This result is particularly relevant because makes these deep learning architectures the only two which are really comparable with the SOTA machine learning ones. Indeed, MLP and TabNet are statistically different from other models in the majority of cases. These findings legitimate the methodology proposed in the current research work as a SOTA one both in terms of performance and in terms of computational complexity (i.e. number of parameters). We cannot assert the same for the TabPFN, which is among the SOTA models in terms of performance but the worst model in terms of computational (and architectural) complexity.

### Models' scalability to larger tabular numerical classification problems

All the models considered in the current research work are primarily designed to handle small tabular classification problems. As described in [104], a dataset is defined as "small" if contains up to \(2000\) samples and \(100\) features. In this section, we explore the ability of the models to scale to larger problems. In so doing, we use benchmark datasets characterized, in turn, by a number of samples greater than \(2000\) or a number of features greater than \(100\).

In Table 2, we mark the success in solving the corresponding tabular classification task with a () symbol, while a failure to solve the problem is denoted by an () symbol. As one can notice, the proposed datasets are sufficient in underlining the criticalities of two models: the TabPFN model and the HCNN model in its MeanSimMatrix configuration. In the first case, the model is unable to scale to problems with a larger number of samples and features. This limitation was already pointed out in the original work by [104] and directly depends on the model's architecture, which strongly leverage the power of attention-based mechanisms. Indeed, the runtime and memory usage of the TabPFN architecture scales quadratically with the number of inputs (i.e. training samples passed) and the fitted model cannot work with datasets with a number of features \(>100\). The authors propose a potential solution to these problems by recommending the incorporation of attention mechanisms that exhibit linear scalability with the number of inputs [127; 128], while simultaneously maintaining satisfactory

Figure 5: Critical Difference plots on out-of-sample average ranks with a Wilcoxon significance analysis. In (a) the test is run considering the F1_Score, in (b) the test is run considering the Accuracy, in (c) the test is run considering the MCC.
performance outcomes. However, no evidence is presented to support this suggestion. In the case of HCNN MeanSimMatrix, instead, the proposed architecture demonstrates a limit in handling problems characterised by a large number of features (but not samples). Also in this case, the reason of the failure should be searched in the model's architectural design choices. Indeed, as underlined in Figure 4(a), there is a strong linear relationship between the number of features and the number of parameters, meaning that when the first parameter is large, convolving across all representatives of each simplicial complex family becomes computationally demanding. A solution to this problem can be found in employing the BootstrapNet configuration, which disrupts the linear relationship discussed earlier, resulting in a significant reduction in the number of parameters when dealing with a large number of features. While this approach demonstrates considerable efficacy, it remains reliant on a threshold parameter (see Section 3.2), suggesting the need for more advanced and parameter-free alternatives.

For the seek of completeness, in Appendix E, we partially repeat the analyses presented in Section 4.1 on the newly introduced datasets. Because of the fragmentation caused by the increased size, we report only the dataset-dependent analyses, excluding cross-datasets ones.

## 5 Conclusion

In this paper, we introduce the Homological Convolutional Neural Network (HCNN), a novel deep learning architecture that revisits the simpler Homological Neural Network (HNN) to gain abstraction, representation power, robustness, and scalability. The proposed architecture is data-centric and arises from a graph-based higher-order representation of dependency structures among multivariate input features. Compared to HNN, our model demonstrates a higher level of abstraction since we have higher flexibility in choosing the initial network's representation, as we can choose from the universe simplicial complexes and we are not restricted to specific sub-families. Looking at geometrical structures at different granularity levels, we propose a clear-cut way to leverage the power of convolution on sparse data representations. This allows to fully absorb the representation power of HNN in the very first level of HCNN, leaving room for additional data transformations at deeper levels of the architecture. Specifically, in the current research work we build the HCNN using a class of information filtering networks (i.e. the TMFG) that uses squared correlation coefficients to maximize the likelihood of the underlying system. We propose two alternative architectural solutions: (i) the MeanSimMatrix configuration and (ii) the BootstrapNet configuration. Both of them leverage the power of bootstrapping to gain robustness toward data noise and the intrinsic complexity of interactions among the underlying system's variables. We test these two modeling solutions on a set of tabular numerical classification problems (i.e. one of the most challenging tasks for deep learning models and the one where HNN demonstrates the poorest performances). We compare HCNN with different machine- and deep-learning architectures, always teeing SOTA performances and demonstrating superior robustness to data unbalances. Specifically, we demonstrate that HCNN is not only able to compete with the latest transformer architectures (e.g. TabPFN) by using a considerably lower and easily controllable number of parameters (especially in the BootstrapNet configuration), guaranteeing a higher level of explicability in the neural network's building process and having a comparable running time without the need for pre-training. We finally propose a study on models' scalability to dataset with increasing size. We underline the fragility of transformer models and we also demonstrate that HCNN in its MeanSimMatrix configuration is unable to manage datasets

\begin{table}
\begin{tabular}{c c c c c c c c c c c c} \hline \hline
**OpenML ID** & **\# Samples** & **\# Features** & & & & & & & & & \\  & & **LogisticRegression** & **RandomForest** & **XGBon** & **LogicGBM** & **MLP** & **ThaNet** & **ThaPFN** & **HCNN** & **HCNN** \\ \hline
361055 & 16714 & 10 & ✓ & ✓ & ✓ & ✓ & ✓ & ✓ & ✗ & ✓ & ✓ \\
361052 & 10825 & 26 & ✓ & ✓ & ✓ & ✓ & ✓ & ✓ & ✗ & ✓ & ✓ \\
361053 & 13488 & 16 & ✓ & ✓ & ✓ & ✓ & ✓ & ✓ & ✓ & ✓ & ✓ \\
361055 & 13276 & 10 & ✓ & ✓ & ✓ & ✓ & ✓ & ✓ & ✓ & ✓ & ✓ \\
361046 & 10578 & 27 & ✓ & ✓ & ✓ & ✓ & ✓ & ✓ & ✓ & ✓ \\
361275 & 13272 & 209 & ✓ & ✓ & ✓ & ✓ & ✓ & ✓ & ✓ & ✓ & ✓ \\
361277 & 3664 & 8 & ✓ & ✓ & ✓ & ✓ & ✓ & ✓ & ✓ & ✓ & ✓ \\
361278 & 10000 & 22 & ✓ & ✓ & ✓ & ✓ & ✓ & ✓ & ✓ & ✓ & ✓ \\ \hline \hline \end{tabular}
\end{table}
Table 2: Study on models’ ability to scale to larger problems. Considered datasets belong to the OpenML benchmark suite “Tabular benchmark numerical classification” [101]. For each of them, we report the OpenML ID, the number of samples, and the number of features. We indicate the success in solving the corresponding tabular classification task with a \(\bigcirc\) symbol, while a failure to solve the problem is denoted by an \(\bigcirc\) symbol.
characterized by a large number of input features. On the other side, we show that the design choice adopted for the BootstrapNet configuration offers a parametric solution to the problem.

Despite significant advances introduced by HCNNs, this class of neural networks remains in an embryonic phase. Further studies on underlying network representations should propose alternative metrics that replace squared correlation coefficients for mixed data-types (i.e. categorical and numerical or categorical only data-types), and further work is finally required to better understand low-level interactions captured by the proposed neural network model. This final point would certainly lead to a class of non-parametric parsimonious HCNN.
