Do Deep Neural Networks Capture Compositionality in Arithmetic
  Reasoning?
Compositionality is a pivotal property of symbolic reasoning. However, how
well recent neural models capture compositionality remains underexplored in the
symbolic reasoning tasks. This study empirically addresses this question by
systematically examining recently published pre-trained seq2seq models with a
carefully controlled dataset of multi-hop arithmetic symbolic reasoning. We
introduce a skill tree on compositionality in arithmetic symbolic reasoning
that defines the hierarchical levels of complexity along with three
compositionality dimensions: systematicity, productivity, and substitutivity.
Our experiments revealed that among the three types of composition, the models
struggled most with systematicity, performing poorly even with relatively
simple compositions. That difficulty was not resolved even after training the
models with intermediate reasoning steps.
Keito Kudo, Yoichi Aoki, Tatsuki Kuribayashi, Ana Brassard, Masashi Yoshikawa, Keisuke Sakaguchi, Kentaro Inui
# Do Deep Neural Networks Capture Compositionality

###### Abstract

Compositionality is a pivotal property of symbolic reasoning. However, how well recent neural models capture compositionality remains underexplored in the symbolic reasoning tasks. This study empirically addresses this question by systematically examining recently published pre-trained seq2seq models with a carefully controlled dataset of multi-hop arithmetic symbolic reasoning. We introduce a _skill tree_ on compositionality in arithmetic symbolic reasoning that defines the hierarchical levels of complexity along with three compositionality dimensions: systematicity, productivity, and substitativity. Our experiments revealed that among the three types of composition, the models struggled most with systematicity, performing poorly even with relatively simple compositions. That difficulty was not resolved even after training the models with intermediate reasoning steps.1

Footnote 1: Our code and data are available at [https://github.com/keitokudo/dentaku_skill_tree](https://github.com/keitokudo/dentaku_skill_tree).

## 1 Introduction

Integrating symbolic reasoning capabilities into neural models has been a crucial goal of artificial intelligence Marcus (2003); d'Avila Garcez and Lamb (2020). With this in mind, many researchers investigated how well modern neural models achieve symbolic reasoning Lake and Baroni (2018). However, recent studies have reported conflicting results on this; some suggest that neural models can solve complex multi-hop reasoning Clark et al. (2020), while others claim that models struggle even with performing simple symbolic operations Qian et al. (2022).

As a step toward further understanding neural models' symbolic reasoning ability, this study systematically analyzes recently published pre-trained seq2seq models using a carefully controlled dataset of multi-hop arithmetic symbolic reasoning. Specifically, our study empirically evaluates the models' ability to generalize the _compositionality_ underlying arithmetic reasoning, where we explore three dimensions of compositionality: (i) systematicity, (ii) productivity, and (iii) substitutivity, as illustrated in Figure 1. Capturing compositionality is crucial in performing symbolic reasoning since compositionality is a pivotal property of generalizability over training instances.

To systematically explore the models' composition ability, we introduce a _skill tree on compositionality_ that defined the hierarchical levels of complexity in arithmetic symbolic reasoning, as illustrated in Figure 2. Using this hierarchy as a lens, we identify the limitations of the neural seq2seq models in capturing the compositionality in arithmetic symbolic reasoning. Our major findings can be summarized as follows:

* Among the three types of composition, the models struggled most with **systematicity**, performing poorly even with relatively simple compositions.
* The major difficulty in systematicity was in the access to intermediate information that is not stated in input but produced during the reasoning.
* Capturing systematicity remained hard for the models trained with the information of the intermediate reasoning steps.

Figure 1: Three dimensionalities of compositionality in arithmetic symbolic reasoning
Skill tree in arithmetic reasoning

We take arithmetic reasoning as the domain for our exploration because it allows us to synthesize questions systematically, as we show in this paper, which helps examine a model's composition ability in a controlled manner. Furthermore, the arithmetic reasoning ability of neural models has gained much attention as modern large language models still struggle with this problems (Rac et al., 2021).

Specifically, we use multi-hop arithmetic reasoning problems as follows:

_Question:_ A=1, B=2, C=A+2, C=?

_Answer:_ 3

Here, the value assigned to the variable C is asked.

### Compositionality in multi-hop symbolic reasoning

In this study, we specifically focused on three dimensions: systematicity, productivity, and substitutivity (Hupkes et al., 2020). According to these, we evaluate how well neural models achieve compositional generalization.

**Systematicity** refers to the ability of combining known different concepts into a more complex concept, i.e., structural composition. To evaluate this ability in models, we first trained with several types of primitive operations (e.g., addition; A=1+2,A=? and selection; A=1,B=2,B=?). Then, we measured the performance in solving problems consisting of combinations of primitives (e.g., A=1+2,B=2+3,B=?).

**Productivity** refers to the ability to solve longer/complex problems based on shorter/simpler ones. To evaluate this ability in models, we first trained with a short version of a formula (e.g., A=1+2,B=2+3,B=?). Then, we measured the performance in solving longer problems (e.g., A=1+2,B=2+3,C=3+4,C=?).

**Substitutivity** refers to the ability to keep the performance even if a particular constituent in a problem is replaced with another (unseen) constituent (i.e., lexical composition). To evaluate this ability in models, we conduct several experiments changing the variable characters between training and test (e.g., train with A=1+2,A=?; then evaluated with \(\boldsymbol{\alpha}\)=1+2, \(\boldsymbol{\alpha}\)=?).

### Dataset configurations

Typical symbolic reasoning (e.g., procedural programming, assembly language) consists of at least three primitive symbol manipulations: assignment (a=2), arithmetic operation (1+2), and reference (a=?). With this in mind, our dataset is generated by combining the following five basic formulas: (i) A=1 (assignment), (ii) A=B (reference & assignment), (iii) A=1+2 (arithmetic operation & assignment), (iv) A=B+2 (arithmetic operation & assignment & reference), (v) A=? (reference). The detailed properties are explained in Section 3.

### Skill tree evaluations

We preliminarily observed that compositionally generalizing complex multi-hop arithmetic reasoning was difficult for neural seq2seq learners (the 1,2,6\(\rightarrow\)9 setting in Section 4). Building on this fact, this study questions what type of composition made it hard for the neural models. To answer this, we designed a _skill tree on compositionality_ that organizes the (hierarchical) complexity levels of symbolic reasoning.2 Evaluating the models using problems with different complexity of composition in a step-wise manner, we elucidate the exact weakness of neural seq2seq models in multi-hop symbolic reasoning.

Footnote 2: The term “skill tree” refers to a visualization method of step-by-step learning in the field of pedagogy (Tondello and Nacke, 2019), distinct from the “tree” in graph theory.

Specifically, we designed ten versions of symbolic reasoning problems. The hierarchical relationship of their task levels is illustrated in Figure 2; in this skill tree, each vertex, i.e., domain, corresponds to different task settings with different

Figure 2: Skill tree to evaluate compositional generalization. The data format of primitive operations is gray and others (complex formulas composed of combinations of primitive operations) are blue.
complexity, and edges represent the hierarchical complexity levels.

By adequately selecting a particular combination of training and test domains, we evaluated the compositional generalization ability of the models from various perspectives. Here, the arithmetic expressions used in the test domain are a combination of those in training domains, creating a semi-order relationship in the skill tree. For example, using the settings 1 (A=1+2,A=?) and 2 (A=1,B=2,B=?) as a training set, and 4 (A=1+2,B=2+3,B=?) as a test set, one can evaluate the model's systematicity generalization towards the arithmetic operations (a+b) and assignments (A=\(i\),B=\(j\),B=?).

## 3 Experimental settings

### Data

**Dataset:** In each experimental setting, we refer to the training domains as \(\mathcal{D}_{\mathrm{train}}=\{d_{\mathrm{train}1},\cdots d_{\mathrm{train}k}\}\) and the test domain as \(d_{\mathrm{test}}\). Each domain has 100,000 training data and 3,200 test data; these are randomly generated, and there is no overlapped instance. When the training domain consisted of multiple domains, we used the union of the training data in \(\mathcal{D}_{\mathrm{train}}\). In addition, when the training domain is not primitive operations (1, 2, 3, and 6 in Figure 2), we further added the primitive operation data related to the training domain (Appendix A) into the training data.

**Arithmetic expressions:** As introduced in Section 2, the input is a sequence of arithmetic expressions. Formally, each expression is in the format of a=\(n\) or a=\(n\){+,-,max,min}\(m\) except that the final expression asks the number assigned to a specified variable (b=?). Here, a and b are a member of a variable name set \(\Sigma\); \(n\) and \(m\) are a member of the variable name set or number set \(\Sigma\cup\mathcal{N}\). Specifically, \(\Sigma\) consists of 21 alphabets, and \(N\) consists of the integer from 0 to 99. The symbol = indicates that the result of the left-hand side is substituted into the right-hand side. The operations (+, -, max, and min) correspond to arithmetic addition, subtraction, max (returning the larger of its left and right numbers), and min (returning the smaller of its left and right numbers).

The questions are designed so that the answer is unique, and depending on the problem set-up, may include mathematical expressions that are not directly related to the final answer, i.e., distractors. The order of the equations is arbitrary; the first equation should not necessarily be calculated first.

**Substittivity test:** In each experimental setting, we evaluate the substitativity generalization performance of the model under the situation where the variable names are replaced with unseen ones, e.g., training with a=1+2,a=?; then evaluating with \(\alpha\)=2+4, \(\alpha\)=?. In this setting, we replaced each variable name in the test set with one of five alphabets that do not overlap with the training ones.

### Trainig and test

**Training:** The training stops when the accuracy on the validation dataset does not increase in successive five epochs or until the validation accuracy reaches 100%. Checkpoints with the highest accuracy in the validation dataset are used for evaluation. Note that among the experiments, the accuracy in the training domain reached at least 99.5%; this indicates that the primitive operations were learnable for the models. Detailed settings for training are described in Appendix A.

**Evaluation metrics:** The accuracy is calculated by the test data in the test domain \(d_{\mathrm{test}}\). Here, we used two metrics: (i) zero-shot accuracy (ZA) and (ii) weighted average of accuracies (WA) to measure the efficiency of learning (Talmor et al., 2020). In measuring WA, a model was further trained using the training set in the test domain \(d_{\mathrm{test}}\); then, the

\begin{table}
\begin{tabular}{l l c c c c c c} \hline \hline \multirow{2}{*}{Task} & \multirow{2}{*}{Type} & \multicolumn{2}{c}{base} & \multicolumn{2}{c}{large} & \multicolumn{2}{c}{x-large} \\  & & \multicolumn{1}{c}{ZA} & WA & \multicolumn{1}{c}{ZA} & WA & \multicolumn{1}{c}{ZA} & WA \\ \hline
1,2 & sys. & 42.0 & 82.1 & 35.2 & 89.8 & 51.7 & 96.7 \\ \(\rightarrow\)4 & +subst. & 39.1 & 80.4 & 32.8 & 89.4 & 50.7 & 96.7 \\ \hline
2,3 & sys. & 33.6 & 75.4 & 32.1 & 85.6 & 35.9 & 94.7 \\ \(\rightarrow\)5 & +subst. & 33.6 & 77.0 & 31.5 & 87.2 & 36.5 & 94.9 \\ \hline
2,3,6 & sys. & 40.8 & 76.5 & 39.1 & 87.7 & 40.5 & 94.6 \\ \(\rightarrow\)8 & +subst. & 39.3 & 74.7 & 37.5 & 86.4 & 39.2 & 94.9 \\ \hline
2,3,6 & sys. & 56.5 & 79.7 & 51.0 & 83.7 & 58.1 & 94.2 \\ \(\rightarrow\)7 & +subst. & 57.6 & 80.2 & 51.1 & 85.8 & 56.7 & 95.1 \\ \hline
1,2,6 & sys. & 24.1 & 28.2 & 23.1 & 29.3 & 27.5 & 32.6 \\ \(\rightarrow\)9 & +subst. & 25.8 & 28.0 & 24.1 & 31.7 & 28.5 & 34.7 \\ \hline
7,8 & sys. & 23.6 & 21.3 & 25.3 & 25.9 & 22.3 & 28.2 \\ \(\rightarrow\)10 & +subst. & 22.3 & 21.6 & 24.4 & 26.4 & 23.0 & 30.2 \\ \hline
1 & prod. & 100.0 & 100.0 & 100.0 & 100.0 & 100.0 & 100.0 \\ \(\rightarrow\)3 & +subst. & 100.0 & 100.0 & 100.0 & 100.0 & 100.0 \\ \hline
4 & prod. & 100.0 & 100.0 & 100.0 & 100.0 & 100.0 & 100.0 \\ \(\rightarrow\)5 & +subst. & **99.9** & 99.0 & 100.0 & 100.0 & 100.0 & 99.9 \\ \hline
9 & prod. & 57.0 & 59.3 & 61.7 & 63.8 & 60.6 & 62.7 \\ \(\rightarrow\)10 & +subst. & 58.4 & 60.9 & 62.2 & 64.1 & 59.5 & 64.5 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Average accuracies in the experiment with 2 different seeds. The “Task” column exhibits (train\(\rightarrow\)test) domains corresponding to the skill-tree (Figure 2). The “Type” column shows the targeted compositionality type in each setting; here, “sys.,” “prod,” and “subst.” denote the systematicity, productivity, and substituttivity generalizations, respectively.
weighted average of accuracies at every update was calculated (details are in Appendix B).

### Models:

We used three different sizes (base, large, and xl) of T5 (Raffel et al., 2020), which is a widely used pre-trained seq2seq model in numerical reasoning tasks (Pal and Baral, 2021; Chung et al., 2022; Yang et al., 2021). Note that we began our training using the models with learned parameters. We also evaluated BART variants and randomly initialized models in Appendix C.

## 4 Experiments and results

We adopted nine combinations of training and test domains as shown in the first column of Table 1 (training domains\(\rightarrow\)test domain). Six of them test the systematicity generalization and the other three test productivity generalization. In each setting, we further tested substitutivity generalization ability using the test domain data with a different variable name set (e.g., \(\boldsymbol{\alpha}\) instead of A) to that used in the training domain. Table 1 shows the overall results. We observed the following four trends:

* Systematicity generalization was more difficult than productivity generalization.
* Even in the simple composition (the setting 1,2\(\rightarrow\)4), the models struggle with generalization from zero or few examples.
* Models achieved substitutivity generalization.
* Model size did not incur substantial performance difference.

We identified that the systematicity generalization of reference and arithmetic operations (setting \(2,3\to 5\); from A=1,B=2,C=3,C=7 and A=1+2,A=7 to A=1+2,B=2+3,C=4+5,C=7) was a simple setting, yet difficult to solve (refer to Appendix D for results on other tasks.). To better understand why neural models struggle with this setting, we decomposed the complexity of this setting and analyzed the model performance. Note that Kim and Linzen (2020) also suggested that neural models lack systematicity generalization ability in the context of semantic parsing; our results corroborate their findings from the context of arithmetic multi-hop reasoning.

Is this difficulty specific to _arithmetic_ symbolic reasoning?We experimented with the same setting except that the four arithmetic operations are replaced with string operations (join, reserveJoin, strSub, and stackJoin; details are in Appendix E.1). The notable difference between arithmetic and string operations is that the string operation could be achieved by only copying selective elements in the input (e.g., 12+34=1234), while arithmetic operation requires the models to access the arithmetic knowledge stored in their internals (e.g., 1+2=3) and generate new information not stated in the input context (e.g., 3).

Larger models tended to overcome the weakness in composition with string operations (e.g., the accuracy of 86.9 in zero-shot evaluation with the x-large model), while they struggled with arithmetic operations. This suggests that the major difficulty in systematicity was in **the access to the arithmetic knowledge (e.g., 1+1=2)**.

Does scratchpad training alleviate the difficulty?Existing studies suggested that showing the intermediate step (scratchpad-style training/inference) improves the multi-hop reasoning ability of neural models (Wei et al., 2022). We tested whether such an explicit generation of intermediate information alleviates the difficulty faced in the previous analysis. Specifically, we trained models with intermediate steps (e,g. A=1+2,B=2+3,B=7; B=2+3,B=5. Details are in Appendix E.2) during training.

The accuracy was calculated by the exact match of the answer and intermediate steps (the steps are designed to be uniquely determined). The performance gain due to explicating the intermediate steps was limited (Table 2), at least with our T5-based models. This shows that, in our carefully controlled setting, merely employing the scratchpad-style generation is not substantially effective.

## 5 Analysis

We conduct a more in-depth analysis of compositional generalization difficulties from another perspective--complexity of arithmetic expressions.

\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline  & \multicolumn{2}{c}{base} & \multicolumn{2}{c}{large} & \multicolumn{2}{c}{x-large} \\ Setting & ZA & WA & ZA & WA & ZA & WA \\ \hline
2,3\(\rightarrow\)5 & 33.6 & 75.4 & 32.1 & 85.6 & 35.9 & 94.7 \\ \hline String & 37.3 & 94.1 & **66.1** & 98.4 & **86.9** & 99.3 \\ Steps & 26.2 & 82.1 & 36.1 & 89.4 & 33.7 & 96.4 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Ablation study with the 2,3\(\rightarrow\)5 (vanilla) setting. “String” refers to the setting where string operations are used instead of arithmetic operations. “Step” denotes the setting generating intermediate steps.
Specifically, for each pair of training and test domains listed in Table 1 (e.g., 1,2\(\rightarrow\)4), we quantified the increase of the complexity of arithmetic formulas from several aspects, e.g., how much the formula's number of variables increased in the test domain (setting of 4) compared to the training domain (setting of 1 and 2). Specifically, we focused on the increase of the number of variables (\(\Delta\)#variables), numbers (\(\Delta\)#numbers), operations (\(\Delta\)#operations), and references (\(\Delta\)#references) from the training to test domains. Here, "#reference" denotes the number of access to a particular variable on the right-hand of equations. For example, the \(\Delta\)#references is 1 if the training data format is A=\(n\), B=A+\(m\) and the test format is A=\(n\), B=A+\(m\), C=B+\(l\). Then, we identified which dimension strongly relates to the compositional generalization difficulty.

We analyzed the macro trends between formula's complexity increase and the difficulty of generalization across the experimental settings. Table 3 shows Spearman's rank correlation coefficient between each complexity and the test-domain accuracy. We found a notable negative correlation in the \(\Delta\)#reference; that is, the more references in the test domain compared to the training domain, the more difficult the compositional generalization becomes (the cases of 1,2,6\(\rightarrow\)9 and 9\(\rightarrow\)10 settings). Simply put, this reveals the difficulty of compositional generalization with _multi-hop_ reasoning--retaining the results of a calculation and accessing them again for another calculation.

## 6 Related work

The analysis of the compositional generalization ability of neural models and arithmetic multi-hop reasoning problems have typically been studied separately; this study has merged these two directions. As for composition generalization analysis, several studies analyzed neural models using datasets such as SCAN Lake and Baroni (2018), COGS Kim and Linzen (2020), and CFQ Keysers et al. (2020). These mainly focused on compositionality in the context of semantic parsing; the composition ability toward symbol manipulations (e.g., multi-hop arithmetic reasoning) is typically out of focus. As for arithmetic reasoning, neural models' abilities have been analyzed typically using benchmarks such as DROP Dua et al. (2019). It has recently been reported that such dataset has superficial cues Al-Negheimish et al. (2021), which made it unclear how much arithmetic reasoning neural model achieves; our study using a carefully controlled dataset contributed to the exact weakness of neural models in this context.

## 7 Conclusion

In this study, we have empirically investigated the arithmetic multi-hop reasoning ability of modern neural models through the lens of compositional generalization ability. To systematically analyze neural models' ability, we have defined a skill tree that organizes the (hierarchical) complexity levels of the multi-hop symbolic reasoning dataset.

Our experiments have revealed that the major weakness lies in systematicity, even with a relatively simple composition. Through the ablation studies, we also have found that difficulty in systematicity is pronounced in accessing knowledge that is not written in input but stored in models. Furthermore, even in training models with intermediate steps that explicate the composition, they struggle to capture systematicity. We also found the difficulty of multi-hop reasoning in compositional generalization. These highlight the exact weakness of neural models and encourage studies to overcome such limitations.

\begin{table}
\begin{tabular}{l r r r r r r r} \hline \hline Complexity & \multicolumn{2}{c}{base} & \multicolumn{2}{c}{large} & \multicolumn{2}{c}{x-large} & \multicolumn{1}{c}{Average} \\ dimensions & ZA & WA & ZA & WA & ZA & WA & Average \\ \hline \(\Delta\)\#variables & 0.098 & -0.098 & 0.488 & -0.293 & -0.098 & -0.488 & -0.065 \\ \(\Delta\)\#numbers & 0.059 & 0.265 & -0.088 & 0.647 & 0.206 & 0.677 & 0.294 \\ \(\Delta\)\#operations & -0.507 & -0.338 & -0.338 & 0.169 & -0.338 & 0.169 & -0.197 \\ \(\Delta\)\#reference & -0.655 & -0.655 & -0.393 & -0.655 & -0.655 & -0.655 & **-0.611** \\ \hline \hline \end{tabular}
\end{table}
Table 3: Spearman’s rank correlation coefficient between the increase of training–test arithmetic complexity and the compositional generalization performance (accuracy) across the nine settings listed in Table 1. A negative score indicates that the greater the training–test discrepancy in its dimension, the more difficult compositional generalization is. In the case that there are multiple training domains, the maximum value among them is used.
### Limitations

In this work, we explored neural networks' ability to capture compositionality in symbolic arithmetic reasoning in hopes that it may lead to future improvements in more general reasoning. However, arithmetic reasoning may not necessarily generalize to natural language tasks. Furthermore, we explored several aspects of multi-hop arithmetic reasoning, but these were chosen from a relatively human-centric perspective, and models may suffer from unforeseen other difficulties. Finally, while we found several patterns in how model performance degrades, it is difficult to aggregate this into a full picture of what a model can and cannot do. Further experiments are needed to gain a more complete understanding of model performance

## Acknowledgements

We thank four anonymous reviewers who provided valuable feedback. We would like to also appreciate the member of Tohoku NLP Group for their cooperation in conducting this research. This work was supported by JST CREST JPMJCR20D2 and JSPS KAKENHI Grant Number JP22H00524, 21K21343.
