Global Hierarchical Neural Networks using Hierarchical Softmax
This paper presents a framework in which hierarchical softmax is used to
create a global hierarchical classifier. The approach is applicable for any
classification task where there is a natural hierarchy among classes. We show
empirical results on four text classification datasets. In all datasets the
hierarchical softmax improved on the regular softmax used in a flat classifier
in terms of macro-F1 and macro-recall. In three out of four datasets
hierarchical softmax achieved a higher micro-accuracy and macro-precision.
Jetze Schuurmans, Flavius Frasincar
# Global Hierarchical Neural Networks using Hierarchical Softmax

###### Abstract.

This paper presents a framework in which hierarchical softmax is used to create a global hierarchical classifier. The approach is applicable for any classification task where there is a natural hierarchy among classes. We show empirical results on four text classification datasets. In all datasets the hierarchical softmax improved on the regular softmax used in a flat classifier in terms of macro-F1 and macro-recall. In three out of four datasets hierarchical softmax achieved a higher micro-accuracy and macro-precision.

Hierarchical Classification, Hierarchical Softmax, Global Hierarchy +
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [
+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [
+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [
+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [
+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote †: [

+
Footnote: [

+
Footnote: [

+
Footnote †: [

+
Footnote: [

+
Footnote: [

+
Footnote: [

+
Footnote: [

+
Footnote: [

+
Footnote: [

+
Footnote: [

+
Footnote: [

+
Footnote: [

+
Footnote: [

+
Footnote: [

+
Footnote: [

+
Footnote: [

+
Footnote: [

+
Footnote: [

+
Footnote: [

+
Footnote: [
+
Footnote:
this last method might be easier to explain than the one from a local based approach (Kang et al., 2017). During both training and testing probabilities of all classes can be assessed.

### Hierarchical Softmax

Hierarchical softmax was first described by (Han et al., 2017). In the context of neural network language models, hierarchical softmax was first introduced in (Kang et al., 2017). Other versions of hierarchical softmax are proposed in (Kang et al., 2018) and (Kang et al., 2019). (Bahdan et al., 2017) used the specification from (Kang et al., 2017) in their FastText classifier. While most of these methods use a binary tree to speed up training and inference time, we try to exploit the natural hierarchy found in the taxonomy of classes for improving the performance. In this taxonomy a node can have more than two child nodes.

### Hierarchical Text Classification

Hierarchical classification was first used for text classification by (Kang et al., 2017). They used a local classifier per parent node for training, at each node selecting a subset of features relevant for that step in the classification process. A similar hierarchical structure with an SVM at every node was used by Kang et al. (Kang et al., 2017) for speech-act classification. Ono et al. (Ono et al., 2018) used a form of local classifier per level, where they tried the lowest level (leaf nodes) first. If the uncertainty was too high, they moved up in the hierarchical level.

## 3. Methodology

Hierarchical classification can be considered as a classification that takes the hierarchical structure of the taxonomy of classes into account, as opposed to a flat classifier, which only takes the final classes into account. By imposing the hierarchical structure, the model does not need to learn the separation between a large number of classes. It can now focus on classifying categories, or subclasses within a category. The taxonomy can be formalised as a tree or a DAG. We consider here the case where the taxonomy is a tree. A taxonomy represented by a Tree is easier to construct, as each child node only has one parent.

A local hierarchical neural network would be infeasible. The network has many parameters and having to learn a new neural network from scratch at every parent node would result in too many parameters, and considerably longer training and inference times. Therefore, we consider making a global classifier by using a hierarchical softmax. Hierarchical softmax easily extends the neural networks by replacing the regular softmax. In this section we discuss the general case of the global hierarchical classifier and the specific case for the hierarchical softmax.

### Global Hierarchy

Global classifiers take advantage of the whole hierarchical structure in the classes at once (Kang et al., 2017). Each node in this hierarchical structure is associated with the probability of the path from the root to that node. We illustrate this in Figure 1. If the node is at depth \(l\) with parents \(n_{1},\ldots,n_{l-1}\), the probability of arriving in this node is:

\[P(n_{l}) =P(n_{l}|n_{l-1})P(n_{l-1}) \tag{2}\] \[=\prod_{j=1}^{l}P(n_{j}|n_{j-1}). \tag{1}\]

### Hierarchical Softmax

In order to calculate the conditional probabilities the hierarchical softmax uses a softmax at every node. The softmax used in the hierarchical softmax to calculate the conditional probability of belonging to node \(m=n_{l}\) conditional on being in the parent node of \(p=n_{l-1}\) becomes:

\[P(m|p)=\frac{\exp(\nu_{pm}^{T}h)}{\sum_{j=1}^{p}\exp(w_{pj}^{T}h)}, \tag{3}\]

where \(w_{pm}^{T}\) is the weight vector corresponding to parent node \(p\) and child node \(m\). The weight vectors \(w\) include the bias terms. \(h\) is therefore the last hidden state concatenated with a one, and provides the same input for each parent node, independent of the depth \(l\). The number of weight vectors \(J_{p}\) is equal to the number of child classes of parent node \(p\).

Compared to a flat classifier with a regular softmax (\(P=1\)), the total number of weights does increase with \((P-1)*(h_{dim}+1)\), where \(P\) is the number of parent nodes, \(h_{dim}\) is the dimension of the hidden dimension and one is added to account for all the additional bias terms. Although the total number of weights is increasing, this is considerably less then if we would consider a new neural network at every parent node, as it is done in the local classifier per parent node.

Each weight vector now has a new purpose. In a flat classifier with the regular softmax, \(\exp(h^{T}w_{j})\) attributes the evidence of class \(j\) compared to all leaf nodes \(\sum_{c=1}^{C}\exp(w_{c}^{T}h)\), where \(C\) equals the number of classes, or leaf nodes. While in the hierarchical softmax, the importance of node \(m\), \(\exp(w_{pm}^{T}h)\), is compared to a subset of nodes \(j=1,\ldots,J_{p}\), \(\sum_{j=1}^{J_{p}}\exp(w_{pj}^{T}h)\). This gives the hierarchical softmax the potential advantage of only having to make the distinction between smaller subsets. In other words, the additional \((P-1)*(h_{dim}+1)\) parameters empower the \(C*(h_{dim}+1)\) parameters to specialise in discriminating within their respective subgroups.

#### 3.2.1. Training of the Hierarchical Softmax

In order to understand the training of a network with a hierarchical softmax component we need to calculate the gradients of the loss function with respect to the parameters, \(w_{pj}\) and \(h\). This will also show that the hierarchical softmax is truly a global classifier, as the whole network is updated based on the performance of all relevant parent nodes.

The loss function we use is the Cross Entropy function. For observation \(i\) the loss is calculated as a function of the estimated
class probabilities \(P_{i}(c)\):

\[E_{i} =-\sum_{c=1}^{C}y_{i,c}\log P_{i}(c) \tag{4}\] \[=-\log P_{i}(m)\] (5) \[=-\log\prod_{q\in Q}P(m_{q}|q)\] (6) \[=-\sum_{q\in Q}\log P(m_{q}|q) \tag{7}\]

The indicator \(y_{i,c}\) is 1 if observation \(i\) belongs to class \(c\), therefore the element of the sum that remains is the negative log probability of the correct class \(m\). We then substitute (2) in the loss (6), where \(Q\) is the set of all parent nodes that lead to the correct class, and \(m_{q}\) the correct child of parent \(q\). In (7) we rearrange the log, making it easier to calculate the derivatives we are looking for:

\[\frac{\partial E_{i}}{\partial w_{p}^{T}} =\mathbb{1}_{p\in Q}(P(j|p)-\delta_{jm_{p}})h \tag{9}\] \[\frac{\partial E_{i}}{\partial h} =\sum_{q\in Q}\sum_{j=1}^{J_{q}}(P(j|q)-\delta_{jm_{q}})w_{jq}^{T} \tag{8}\]

where the indicator function \(\mathbb{1}_{p\in Q}\) equals one if \(p\) is in \(Q\) and zero otherwise. Likewise, the Kronecker delta is defined as \(\delta_{jm}=1\) if \(j=m\), and 0 otherwise. The derivations of (8) and (9) are given in Appendix A.

These gradients can be used in the Stochastic Gradient Decent algorithm. More importantly, (9) shows the update of the hidden state (and therefore the rest of the network) is a combination of the performances across all child nodes that belong to the parent nodes that make up the path to the correct class. This shows that a neural network with a hierarchical softmax is truly a global hierarchical classifier.

## 4. Datasets

We consider four text classification datasets, in which we can find a hierarchical structure in the classes. In Appendix B we present the exact taxonomy used.

### Trec

First, we consider the TREC 10 Question Answering Track Corpus (Trec et al., 2015), abbreviated as TREC. This dataset consists of 5952 questions (5452 train, 500 test), each belonging to one of the 50 classes. These classes are split up between 6 categories. Figures 2 shows that the distribution between categories is highly unbalanced. The TREC training set is highly unbalanced as well. The number of training observations per class range from 4 till 962.

### 20NewsGroups

The second dataset is the 20NewsGroups dataset (Trec et al., 2015). We find 6 categories, on top of the 20 classes. In Figure 3 the distribution of the classes between the categories can be seen. This distribution is relatively balanced. This dataset contains 11293 training observations and 7527 test observations. The training set is relatively balanced. Most classes have between 500 and 600 observations. There is one outlying class with only 377 training observations.

### Reuters-21578

As third and fourth dataset we use two configurations of the Reuters-21578 dataset (Ferrone et al., 2015). These are Reuters-8 and Reuters-52.

#### 4.3.1. Reuters-8.

The Reuters-8 dataset consists of the 8 most frequent classes, based on the number of observations in the training set. These are distributed between 4 categories. The classes are distributed evenly between the categories, as shown in Figure 4. The observations of this dataset are split between train and test set with 5485 in train and 2189 in test. The training set is very unbalanced, with observations per class ranging from 41 to 2840.

#### 4.3.2. Reuters-52.

Respectively the Reuters-52 dataset contains the 52 most frequent classes, also distributed between 4 categories. Figure 5 shows the distribution of classes among the categories is highly unbalanced. The Reuters-52 dataset contains 6532 training and 2568 test observations. This dataset is by definition more unbalanced than Reuters-8. The minimum number is only a single training observation.

## 5. Experiments

In our experiments we employ an LSTM (He et al., 2017) with hierarchical softmax and compare the results with an LSTM with regular softmax.

Figure 1. Hierarchical structure of a two level global classifier.
LSTMs are a popular and well performing architecture for text classification, because of their ability to process sequential data (Bir
between recall and precision indicate a worse trade-off in the flat model. With a difference of 0.2 percentage points, the micro-accuracy of the regular softmax is not much higher than the micro-accuracy of the hierarchical softmax. The flat and hierarchical models have a larger gap between the micro-average and macro measures, due to the higher class unbalance. Finally, we note that both and are close to the state-of-the-art (Frieder et al., 2017).

## 7. Conclusion

We conclude that the hierarchical softmax makes a good candidate for making a neural network a global hierarchical classifier. We show that it can improve performances of a recurrent network on four different text classification datasets, in terms of macro-F1 and macro-Recall. The performances on the different datasets show that the hierarchical softmax can handle different types of class taxonomies, balanced and unbalanced, in terms of both training observations per class, as well as child nodes per parent node.

With regard to the state-of-the-art, it is not our goal to improve the SOTA, instead we show that changing a regular softmax with a hierarchical softmax in a dataset with a natural hierarchy in the classes leads to an improvement. Although we did not improve the state-of-the-art, we do come close with the hierarchical softmax on a parsimonious model. We also note that the SOTA models are a different model for each dataset, while we consistently perform well on all datasets with the same model. Future work can study if the state-of-the-art might be improved if the hierarchical softmax is used in the respective model.

Furthermore, we consider a two-level hierarchical taxonomy, by introducing one level of parent nodes in between the root and the leaves. In future work, the taxonomy could be extended with an additional hierarchical layer, i.e. by grouping parent nodes.

The hierarchy is currently determined based on the hierarchy in the class taxonomy. Alternatively the construction and evaluation of different hierarchical structures could be automated.

The performance of the hierarchical softmax depends on the probability estimates of the conditional probabilities of moving from a parent node to a child node. Better estimates might be obtained by using Bayesian neural networks, as their probability estimates are significantly better (Frieder et al., 2017; Goyal et al., 2017; Goyal et al., 2017).

The hierarchical softmax is not only applicable for text classification. In theory it can replace a softmax in any classification task. It would also be interesting to see how this approach fares in other classification tasks, for example in image classification.
