# Learning to Generalize Compositionally by
Transferring Across Semantic Parsing Tasks

Wang Zhu1, Peter Shaw2, Tal Linzen3, Fei Sha2
1University of Southern California, 2Google Research, 3New York University

###### Abstract

Neural network models often generalize poorly to mismatched domains or
distributions. In NLP, this issue arises in particular when models are
expected to generalize compositionally, that is, to novel combinations of
familiar words and constructions. We investigate learning representations that
facilitate transfer learning from one compositional task to another: the
representation and the task-specific layers of the models are strategically
trained differently on a pre-finetuning task such that they generalize well on
mismatched splits that require compositionality. We apply this method to
semantic parsing, using three very different datasets, COGS, GeoQuery and
SCAN, used alternately as the pre-finetuning and target task. Our method
significantly improves compositional generalization over baselines on the test
set of the target task, which is held out during fine-tuning. Ablation studies
characterize the utility of the major steps in the proposed algorithm and
support our hypothesis.

##  1 Introduction

Recent work has spotlighted significant shortcomings of neural network
approaches to NLP in coping with compositional generalization (CG) (Lake &
Baroni, 2018; Finegan-Dollak et al., 2018; Keysers et al., 2020; Kim & Linzen,
2020; Shaw et al., 2021). In those studies, the test set combines in
unfamiliar ways linguistic elements that may themselves be familiar from the
training set; for example, sentences in the test set may be longer than those
observed in training, or may use familiar words in syntactic roles in which
they did not occur in training. The performance of popular neural
architectures in contemporary NLP models such as Transformers drops
considerably when they are expected to generalize compositionally. Several
approaches have been proposed to address this issue, including specialized
architectures with compositional inductive biases (Furrer et al. 2020) and
compositional data augmentation (Andreas, 2020), but as of yet the problem is
far from solved.

In this paper, we study this challenge from the perspective of transferability
of compositional generalization skills: is possible for a neural model to
transfer compositional generalization skills acquired from one task to another
task that also requires compositional generalization skills?

We ground our inquiries with the semantic parsing tasks on three very
different datasets: geoquery (Zelle, 1995), cogs (Kim & Linzen, 2020) and scan
(Lake & Baroni, 2018). For each task, we use existing compositional
training/test splits or create new ones. We propose a learning algorithm that
can extract a compositional inductive bias from one task — a stage we refer to
as pre-finetuning 111We borrow this terminology from the NLP research
community. A large number of research papers have explored the idea of
transfer learning, starting from a pre-trained language model, followed by
training on a set of pre-finetuning tasks, and then fine-tuning on the target
or downstream tasks (Vu et al., 2020; Gururangan et al., 2020; Pruksachatkun
et al., 2020; Chen et al., 2020a; Aghajanyan et al., 2021).  — and transfers
that bias to a target task, improving the models’ compositional generalization
behavior on that task.

To extract the inductive bias so as to transfer, we introduce a new training
algorithm duel. In summary (cf. Fig 1), duel is designed to be compatible with
pre-trained neural encoder-decoder models, such as T5 (Raffel et al., 2020).
We view the encoder as learning a representation for the inputs and the
decoder as “a task head” that is specialized to different tasks. In pre-
finetuing, framing the task’s two (compositional) splits as deriving
representation from one split for zero-shot learning to the other split, duel
trains the encoder and the decoder using different splits. In contrast to
using standard supervised learning over both splits as a whole for pre-
finetuning, duel encourages the encoder to learn to represent the input texts
in a way that facilitates greater compositional generalization, and that this
transfers across domains and persists through fine-tuning on the target task,
as shown by our empirical studies.

The rest of the paper is organized as follows. In §2, we detail our setup. We
describe our approach in §4, followed by empirical studies §5. We discuss
related work in §6 and conclude in §7.

##  2 Problem Setup

Our neural learner is given a pre-finetuning task and a target task. For the
pre-finetuning task, we have a training data distribution  $s$ and a
test/eval data distribution  ${\tilde{s}}$ that may deviate from  $s$.
Likewise, for the target task, we have  $q$ and  ${\tilde{q}}$. In the case
of compositional generalization, the difference between  $s$ and
${\tilde{s}}$ (or  $q$ versus  ${\tilde{q}}$) is defined in compositional
terms: the ways in which elements combine differ systematically between the
two parts of each datasets (Keysers et al., 2020; Kim & Linzen, 2020; Lake &
Baroni, 2018; Hupkes et al., 2020).

Our goal is to train a neural model on ( $s$,  ${\tilde{s}}$) and then fine-
tune it on  $q$ so that the final model performs well on  ${\tilde{q}}$,
namely, attaining strong compositional generalization performance on the
target task.

Our assumption is that the difference between  $s$ and  ${\tilde{s}}$ is
similar to the difference between  $q$ and  ${\tilde{q}}$, such that an
inductive bias acquired from ( $s$,  ${\tilde{s}}$) can be transferred to
the target task. A motivating example is that  ${\tilde{s}}$ contains longer
texts than  $s$ and so does  ${\tilde{q}}$ than  $q$: in this case, we
would investigate whether learning to generalize from short to long texts is a
transferrable skill.

##  3 Task, Data Sets and Their Compositional Splits

#### Data Splits

In this work, we focus on semantic parsing, the task of translating a natural
language text into a formal representation of its semantics. Standard data
splits for supervised learning partition a dataset randomly into a training
set and testing set. As such, the examples in both portions are drawn from the
same distribution. By contrast, compositional splits partition a dataset in a
way that creates a distributional discrepancy between the training and the
test set, such that strong performance on the test set requires compositional
generalization from the training set.

Compositional splits are straightforward to create for datasets that are
generated from grammars or templates, such as SCAN (Lake & Baroni, 2018) and
COGS (Kim & Linzen, 2020), described in detail in the next section. For
instance, the templates used for the test set can include specific words in
syntactic roles they did not appear in in the training set, or they can nest
syntactic constituents more deeply than in the training set, creating
differences in lengths of the input texts.

An alternative mechanism creates—based on an existing dataset for which the
generative process is not available (for example, real-world user
queries)—test sets that require compositional generalization by on splitting
the dataset so as to obtain the maximum compound divergence (MCD) between the
training and test examples (Keysers et al., 2020). Here, compounds are complex
expressions formed by composing atoms, and compound divergence is computed as

$\mathcal{D}_{C}(V\|W)=1\,-\,C_{0.1}(\mathcal{F}_{C}(V)\,\|\,\mathcal{F}_{C}(W))$

where $\mathcal{F}$ is the distribution over compounds, and
$C_{\alpha}(P\|Q)=\sum_{k}p_{k}^{\alpha}\,q_{k}^{1-\alpha}$. The intuition
behind this method is that given an existing dataset it is difficult to limit
the test examples to compounds that are completely absent from the training
set, but by making the compound distributions as distinct as possible, we are
primarily measuring the model’s performance on compounds are at least very
infrequent in training.

We use the MCD split generated for SCAN from Keysers et al. (2020) and the
Target Maximum Compound Divergence (TMCD) splits generated for GeoQuery from
Shaw et al. (2021). TMCD splits extend the MCD methodology to create
compositional splits of non-synthetic datasets and define atoms and compounds
based on the known syntactic structure of the output logical forms.

#### Datasets and Tasks

Table 1 summarizes the 5 compositional splits we use in this paper. We also
compared with standard splits (_std_) where the training and test come from
the same distribution.

As an example for our setting in §2, taking cogs as our pre-finetuning task
and geoquery as our target task, we learn from the CG General split of cogs
and transfer to the CG General split of geoquery. In this example, cogs
${}_{\textsc{base}}$ is the distribution  $s$ and cogs ${}_{\textsc{cg}}$
is the distribution  ${\tilde{s}}$. geo ${}_{\textsc{tmcd1}}$ is our fine-
tuning distribution  $q$ and geo ${}_{\textsc{tmcd2}}$ is our final test
distribution  ${\tilde{q}}$.

Table 1: The datasets and their compositional splits used in this paper. We classify compositional splits into two categories: general CG splits where the training and test sets differ in compound distribution (cd) or in a range of compositional properties (cg), and a special type of length CG splits where the training and test sets differ in sentence length (len). Category | Dataset | Shorthand | Train | Test
---|---|---|---|---
CG General | cogs |  cogs ${}_{\text{cg}}$ |  cogs ${}_{\textsc{base}}$ |  cogs ${}_{\textsc{cg}}$
geoquery |  geo ${}_{\text{cd}}$ |  geo ${}_{\textsc{tmcd1}}$ |  geo ${}_{\textsc{tmcd2}}$
scan |  scan ${}_{\text{cd}}$ |  scan ${}_{\textsc{mcd1}}$ |  scan ${}_{\textsc{mcd2}}$
CG Length |  geoquery |  geo ${}_{\text{len}}$ |  geo ${}_{\textsc{short}}$ |  geo ${}_{\textsc{long}}$
scan |  scan ${}_{\text{len}}$ |  scan ${}_{\textsc{short}}$ |  scan ${}_{\textsc{long}}$

cogs (Kim & Linzen, 2020) is a synthetic semantic parsing dataset generated
from templates. The inputs are English sentences and the outputs are
corresponding logical forms inspired by $\lambda$-calculus, _e.g_., $\textit{A
dog ate the
cake}\rightarrow\text{*cake}(x_{4});\text{dog}(x_{1})\;\textsc{and}\;\text{eat.agent}(x_{2},x_{1})\;\textsc{and}\;\text{eat.theme}(x_{2},x_{4})$.

cogs has two components: a training dataset, which we refer to as cogs
${}_{\textsc{base}}$, and a compositional generalization (CG) dataset cogs
${}_{\textsc{cg}}$. Kim & Linzen (2020) show that the performance of neural
models trained on cogs ${}_{\textsc{base}}$ degrades significantly when they
are applied to cogs ${}_{\textsc{cg}}$. cogs ${}_{\textsc{cg}}$ includes two
kinds of CG challenges: lexical and structural. In _lexical CG_ , familiar
words need to be interpreted in new syntactic positions; for example, in cogs
${}_{\textsc{base}}$ the word _hedgehog_ may occur only in the subject
position, in cogs ${}_{\textsc{cg}}$ it would need to be interpreted as as
object. By contrast, _structural CG_ involves new combinations of familiar
syntactic structures. For example, in cogs ${}_{\textsc{base}}$,
prepositional phrases only modify objects (_Noah ate the cake on the plate_),
whereas in cogs ${}_{\textsc{cg}}$ they modify subjects (_The cake on the
plate burned_). Likewise, in cogs ${}_{\textsc{base}}$ prepositional phrases
can only be nested once (_Ava saw the ball in the bottle on the table_), but
in cogs ${}_{\textsc{cg}}$ they are nested multiple times (_Ava saw the ball
in the bottle on the table on the floor_), creating longer sentences. cogs
${}_{\textsc{base}}$ has 24,155 examples and cogs ${}_{\textsc{cg}}$ has
21,000 examples, of which 18,000 are instances of lexical CG and 3,000 are
instances of structural CG.

The geoquery dataset (Zelle, 1995; Tang & Mooney, 2001) contains 880 natural
language questions about US geography (e.g., _What states border Texas?_). We
use the same pre-processing as in Shaw et al. (2021), replacing entity
mentions with placeholders in the Functional Query Language (FunQL; Kate et
al. 2005) output representations. The input _what states border m0_ , for
example, is associated with the output answer(intersection(state,
next_to_2(m0))). Atoms and compounds are defined over the FunQL expressions.
We adopt the length and TMCD splits from Shaw et al. (2021). For the length
split geo ${}_{\text{len}}$, the training set geo ${}_{\textsc{short}}$
consists of examples with shorter inputs than the test set geo
${}_{\textsc{long}}$. We refer to the training and test set for the TMCD
split as geo ${}_{\textsc{tmcd1}}$ and geo ${}_{\textsc{tmcd2}}$. Both
splits divide geoquery equally into two sets of 440 examples.

The scan dataset (Lake & Baroni, 2018) consists of over 20,000 natural
language navigation commands and corresponding “action sequences”, _e.g_.
_jump twice_ $\rightarrow$ JUMP JUMP. We adopt the length split (scan
${}_{\text{len}}$) and MCD split (scan ${}_{\text{cd}}$) introduced by
Keysers et al. (2020).

##  4 Proposed Method

#### Intuition and Main Idea

As mentioned in §2, our goal is to improve compositional generalization on our
target task, where the training distribution is  $q$ and the evaluation
distribution  ${\tilde{q}}$. The challenge is to leverage the information in
 $s$ and  ${\tilde{s}}$ during the pre-finetuning stage such that the model,
when fine-tuned on  $q$, generalizes better on  ${\tilde{q}}$. This
contrasts with a standard supervised learning approach, which would collapse
together the examples from  $s$ and  ${\tilde{s}}$ during pre-finetuning,
discarding the information about how examples are divided between  $s$ and
${\tilde{s}}$—information that, we hypothesize, can be useful for encouraging
compositional generalization on the target task.

We assume a neural encoder-decoder architecture (Cho et al., 2014). Our
proposed method, duel, iteratively updates the parameters of the encoder on
examples from  ${\tilde{s}}$, while keeping the decoder fixed, and then
updates the decoder parameters on examples from  $s$, while keeping the
encoder fixed. The encoder updates are then retained prior to fine-tuning on
$q$, while the decoder updates are discarded (as the pre-finetuning and the
target tasks are different). In contrast to standard fine-tuning, duel
encourages the encoder to learn to represent the input sequences in a way that
facilitates compositional generalization, which we hypothesize will transfer
across domains and persist through fine-tuning.

#### Details

For encoder-decoder architectures, the model’s parameters naturally partition
into two sets, $\theta$ for the encoder and $\phi$ for the decoder, where
$\theta$ parametrizes the representation $f(x;\theta)$ of the input $x$ and
$\phi$ parametrizes the “task head” $g(\cdot)$, which uses that representation
to accomplish a task such as classification or semantic parsing. Taken
together, the model’s output is $p(y|x;\theta,\phi)=g(f(x;\theta);\phi)$.

Given a pre-finetuning task where both  $s$ and  ${\tilde{s}}$ are given,
our goal is to drive the learner to learn the optimal representation and task
head parameters $(\theta^{*},\phi^{*})$ such that
$g(f(x;\theta^{*});\phi^{*})$ performs the best on both distributions. To
achieve this, we design an iterative dueling game that the learner plays with
the two distributions.

Suppose the learner’s parameters are currently $(\theta,\phi)$. To perform
optimally on  ${\tilde{s}}$, we would like to use the task head
$g(\cdot;\phi)$ without further training it, _i.e_., perform well in an “zero-
shot” setting; this setting directly anticipates how learning would proceed
for the target task, where we will be training only on  $q$, without exposure
to  ${\tilde{q}}$. Since we are holding the task head $g(\cdot;\phi)$ fixed,
we need to optimize the representation $f(\cdot;\theta)$ such that it models
${\tilde{s}}$ appropriately. To achieve this desiderata, we update the
representation parameters iteratively for at most $T_{\text{inner}}$ steps,
according to

$\theta\leftarrow\theta+\alpha\frac{1}{N}\nabla_{\theta}\sum\nolimits_{(x,y)\in
B_{\text{\\!~{}${\tilde{s}}$}}}\log p(y|x;\theta,\phi)$ (1)

where $B_{\text{\\!~{}${\tilde{s}}$}}$ is a batch of $N$ samples from
${\tilde{s}}$ and $\alpha$ is the step size.

Updates to $\phi$ follow the reverse logic. The representation function that
results from updating $\theta$ defines a prior on how  $s$ should be
represented. When updating the task head parameters $\phi$, our goal is to
have the task head perform optimally conditioned on this prior. Concretely, we
hold $\theta$ fixed and apply iteratively at most $T_{\text{inner}}$ steps to
$\phi$:

$\phi\leftarrow\phi+\alpha\frac{1}{N}\nabla_{\phi}\sum\nolimits_{(x,y)\in
B_{\text{\\!~{}$s$}}}\log p(y|x;\theta,\phi)$ (2)

where $B_{\text{\\!~{}$s$}}$ is a batch of samples from  $s$. Crucially, we
alternate between these two types of updates, such that  ${\tilde{s}}$
informs the learner about the compositional inductive bias that should be
incorporated into the representation $f$, and  $s$ teaches the learner how to
use that representation to perform the task (here semantic parsing).

We contrast our approach with the standard supervise learning approach, where
the two distributions are merged together as
$\text{\\!~{}$s$}\cup\text{\\!~{}${\tilde{s}}$}$, ignoring the compositional
split:

$\displaystyle(\theta,\phi)\leftarrow(\theta,\phi)+\eta\frac{1}{N}\nabla_{(\theta,\phi)}\sum\nolimits_{(x,y)\sim
B_{\text{\\!~{}$s$}\cup\text{\\!~{}${\tilde{s}}$}}}\,\log p(y|x;\theta,\phi)$
(3)

Figure 1: The duel updates. The parameters in the representation function
$f(\cdot)$ and the task head $g(\cdot)$ are updated using different data
distributions in pre-finetuning.

duel is used for pre-finetuning. When fine-tuning the model on the target task
 $q$, we retain the representation component $f(\cdot;\theta)$ and re-
initialize the task head $g(\cdot;\phi)$. Both $\theta$ and $\phi$ are then
updated, as is standard, to optimize the loss function on  $q$.

duel is illustrated schematically in the pre-finetuning panel of Fig. 1. The
pseudocode is listed in Algorithm 1. The algorithm contains one outer loop for
$T_{\text{outer}}$ rounds (with possible early stopping), which alternates
between inner loops updating $\theta$ and $\phi$. The early-stopping
monitoring function AccuracyDecreases takes as arguments the current model
parameters $\theta$ and $\phi$, a data distribution to use for evaluation ($s$
or $\tilde{s}$), and the maximum patience $T_{\textup{patience}}$ for early-
stopping. This function returns true if accuracy on the dataset in question
has not improved for $T_{\textup{patience}}$ consecutive steps. To terminate
the outer loop (thus, the algorithm), we keep track how many steps the inner
update for $\theta$ take. If the number steps is smaller than a preset
threshold $T_{\text{min}}$, we conclude that the algorithm has converged to
the desired representation since the difference in representing  $s$ and
${\tilde{s}}$ is small, requiring little adaptation. We do not use the same
logic to limit how long it takes to optimize the task head parameters $\phi$,
conditioned on the representation, as the goal is to use the derived
representation to arrive at a strong performance.

Require: Data sets  $s$,  ${\tilde{s}}$; Learning rate $\alpha$; Batch size
$N$; Outer loop iterations $T_{\textup{outer}}$; Inner loop iterations
$T_{\textup{inner}}$; Inner loop early stopping criteria
$T_{\textup{patience}}$; Outer loop early stopping criteria
$T_{\textup{min}}$;

Initialize model parameters $\theta,\phi$

$i\leftarrow 0$

while  _$i <T_{\textup{outer}}$_ do

$j\leftarrow 0$

while  _$j <T_{\textup{inner}}$_ do

Sample a batch $B_{s}$ from  $s$

Compute loss: $\mathcal{L}(\theta,\phi,B_{s})=\frac{1}{N}\sum_{(x,y)\in
B_{s}}-\log p(y\mid x;\theta,\phi)$

Update parameters:
$\phi\leftarrow\phi-\alpha\cdot\nabla_{\phi}\mathcal{L}(\theta,\phi,B_{s})$

$j\leftarrow j+1$

if  _AccuracyDecreases($\theta$, $\phi$,  ${\tilde{s}}$,
$T_{\textup{patience}}$)_ then

Early stop: break

end if

end while

$k\leftarrow 0$

while  _$k <T_{\textup{inner}}$_ do

Sample a batch $B_{\tilde{s}}$ from  ${\tilde{s}}$

Compute loss:
$\mathcal{L}(\theta,\phi,B_{\tilde{s}})=\frac{1}{N}\sum_{(x,y)\in
B_{\tilde{s}}}-\log p(y\mid x;\theta,\phi)$

Update parameters:
$\theta\leftarrow\theta-\alpha\cdot\nabla_{\theta}\mathcal{L}(\theta,\phi,B_{\tilde{s}})$

$k\leftarrow k+1$

if  _AccuracyDecreases($\theta$, $\phi$,  $s$, $T_{\textup{patience}}$)_ then

Early stop: break

end if

end while

if  _$k <T_{\textup{min}}$_ then

Early stop outer loop: break

end if

$i\leftarrow i+1$

end while

Return: Model parameters $\theta$, $\phi$

Algorithm 1 duel

##  5 Experiments

###  5.1 Settings

#### Datasets and Tasks

We use the datasets described in Table 1. The target tasks always come from
one of the compositional splits: cogs ${}_{\text{cg}}$, geo
${}_{\text{cd}}$, scan ${}_{\text{cd}}$, or geo ${}_{\text{len}}$. For the
pre-finetuning tasks, we consider two configurations: (1) standard splits for
cogs, geoquery and scan; (2) compositional splits: cogs ${}_{\text{cg}}$, geo
${}_{\text{cd}}$, scan ${}_{\text{cd}}$, scan ${}_{\text{len}}$.

#### Models

We use two neural architectures, both encoder-decoder models based on the
Transformer architecture (Vaswani et al., 2017). The first one, which we refer
to as bert2seq, uses the BERT encoder bert${}_{\textsc{small}}$ (Devlin et
al., 2019), with 4 Transformer layers, to learn the representation
$f(x;\theta)$, followed by a Transformer-based sequence-to-sequence model with
2 encoder layers and 2 decoder layers, as the task head $g(\cdot;\phi)$.222The
model is larger than the baseline Transformer model from Kim & Linzen (2020),
which has the same architecture as $g(\cdot)$ alone. The model trained by Kim
& Linzen (2020) does not use pre-finetuning and is outperformed by our model
in the same setup from a pre-trained bert${}_{\textsc{small}}$.  We initialize
$f(\cdot)$ with the pre-trained bert${}_{\textsc{small}}$ (Turc et al., 2019).

We also experimented with the much larger T5-Base model (Raffel et al., 2020).
We take the T5 encoder as $f(x;\theta)$ and the T5 decoder as the task head
$g(\cdot;\phi)$; the encoder and encoder have 12 Transformer layers each. We
initialize them with the pre-trained T5 model. In both pre-finetuning and
fine-tuning, we generate task prompts by prepending a task tag to each input
sentence (following Raffel et al. 2020). We provide detailed examples of the
task prompts for each task in Appendix B.

When using bert2seq, our main results are obtained by initializing the task
heads randomly. For T5, we initialize the task heads with the pre-trained T5
decoder, for both pre-finetuning and fine-tuning.

#### Baselines

We compare duel to two baselines: (1) none, where we fine-tune on the target
task (_i.e_.,  $q$) directly, without pre-finetuning; (2) merged, pre-
finetuning on  $s$ $\cup$ ${\tilde{s}}$ using eq. (3) and then fine-tuning
on the target task. As mentioned above, merged ignores the compositional
splits and conducts pre-finetuning using the standard splits (cf. Table 1).

###  5.2 duel learns transferable compositional generalization inductive bias

#### Main Results

Table 2 shows the results of our experiments taking the CG General splits as
the target splits. The metric we report is exact match accuracy over output
sequences. For cogs, we report accuracy separately for the lexical and
structural generalization cases, as our models’ performance differed
dramatically between those two types of cases. On scan, the none baseline (no
pre-finetuning) matches the average accuracy 15.4% reported by Conklin et al.
(2021).

Pre-finetuning substantially improves over the baselines, across the board. In
particular, pre-finetuning on scan ${}_{\text{cd}}$ leads to an accuracy of
57.8% on geo ${}_{\textsc{tmcd2}}$, surpassing — with a neural-only approach
— the previous state-of-the-art of 56.6%, which was established by NQG-T5, a
hybrid model that combines a grammar-based approach with T5 (Shaw et al.,
2021).

Table 2: Accuracy on the target tasks after pre-finetuning on other tasks. Note that the merged baseline is the same for the Standard and CG General splits as it uses the union of two distributions. For COGS, we report lexical and structural CG accuracy separately (lexical/structural). Model | Pre-fine | Category | geo | scan | cogs | scan | cogs | geo
---|---|---|---|---|---|---|---|---
| tuning |  |  $\rightarrow$cogs ${}_{\text{cg}}$ |  $\rightarrow$geo ${}_{\text{cd}}$ |  $\rightarrow$scan ${}_{\text{cd}}$
bert2seq | none | - | 50.8/0.2 | 50.8/0.2 | 34.3 | 34.3 | 1.3 | 1.3
merged | Standard | 71.2/0.7 | 66.9/0.8 | 35.9 | 36.1 | 3.2 | 3.0
duel | Standard | 63.4/1.0 | 61.5/0.3 | 35.2 | 35.7 | 2.9 | 3.1
| duel | CG General | 75.5/1.1 | 73.2/1.1 | 36.9 | 37.7 | 4.8 | 5.5
T5-Base | none | - | 93.7/2.5 | 93.7/2.5 | 52.5 | 52.5 | 15.4 | 15.4
merged | Standard | 95.2/4.2 | 94.3/3.4 | 52.5 | 54.3 | 18.7 | 16.9
duel | Standard | 94.9/3.1 | 94.1/3.2 | 51.1 | 51.6 | 19.2 | 15.7
| duel | CG General | 95.4/4.5 | 94.9/3.9 | 53.8 | 57.8 | 20.2 | 21.3

#### Pre-finetuning on a compositional split is essential

Do the improvements we observe in Table 2 specifically reflect the
contribution of pre-finetuning on compositional splits, or can they be
attributed to pre-finetuning more generally? Table 2 reports the performance
of the merged baseline, _i.e_., pre-finetuning on the merged dataset
$s\cup\tilde{s}$, without a compositional split. Comparing to the none (no
pre-finetuning) rows in Table 2, we observe that pre-finetuning improves
performance on the target tasks even without using duel or the compositional
split. However, we see bigger gains from using duel on compositional splits
(duel rows with CG General category in Table 2), especially for the target
tasks geo ${}_{\text{cd}}$ and scan ${}_{\text{cd}}$. On cogs
${}_{\text{cg}}$, the gain of using duel on compositional splits is less
pronounced. This reflects two facts: first, that lexical generalization is not
very challenging when the encoder uses a pre-trained model with strong lexical
representations, such that performance is close to being saturate even without
pre-finetuning; and second, that the structural generalization cases in cogs
remain too difficult for purely neural models, even with duel.

Within Table 2, it is clear that duel does not make a major difference from
merged when restricted to standard splits . This is reasonably expected as the
duel procedure reduces to the merged when the data distribution  $s$ and
${\tilde{s}}$ are exactly the same for the pre-finetuning task (cf. eq.(3) to
eq. (1,2)).

In summary, the results so far suggest that duel is most effective when used
in conjunction with pre-finetuning tasks with compositional splits.

###  5.3 When will duel works best?

Table 3: Results on length generalization (geo ${}_{\textsc{short}}$ $\rightarrow$geo ${}_{\textsc{long}}$) from different pre-finetuning tasks. Numbers in parentheses show the improvement of duel from merged. Model | Pre-finetuning |  scan ${}_{\text{std}}$ |  scan ${}_{\text{cd}}$ |  cogs ${}_{\text{cg}}$ |  scan ${}_{\text{len}}$
---|---|---|---|---|---
bert2seq | none | 16.1
merged | 16.1 | 16.1 | 16.8 | 16.1
| duel | 15.9 (-0.2) |  16.8 (+0.7) |  17.7 (+0.9) | 18.6 (+2.5)
T5-Base | none | 38.6
merged | 39.1 | 39.1 | 40.9 | 39.1
| duel |  39.3 (+0.2) |  40.0 (+0.9) |  43.0 (+2.1) |  45.0 (+5.9)

We hypothesize duel works best when the pre-finetuning tasks and the target
tasks share strong similarity in their compositional splits. We gain further
insights about how effective duel is by restricting our study to a special
type of compositional generalization - the CG Length generalization. Table 3
reports the results when the compsitional split in the target task geo
${}_{\text{len}}$ is to generalize from shorter texts to longer ones. Without
pre-finetuning, the accuracy on geo ${}_{\text{len}}$ is 38.6% with T5-Base,
and 16.1% with bert2seq. The state-of-the-art results on this task is 52.2%
from the NQG-T5 model (Shaw et al., 2021).

As before, pre-finetuning improves from that baseline. scan ${}_{\text{len}}$
with duel improves the most for both neural models. Intuitively, this is
sensible as scan ${}_{\textsc{short}}$ also consists of examples with shorter
texts than those in the test set scan ${}_{\textsc{long}}$: on average, the
input lengths are 7.0 versus 8.2 on scan ${}_{\textsc{short}}$ and scan
${}_{\textsc{long}}$ respectively, while the output lengths significantly
vary from 10.8 to 30.0. In this case, duel helps extracting the inductive bias
for generalizing on length difference.

Our experiments also show that cogs ${}_{\text{cg}}$ works better than scan
${}_{\text{cd}}$. The compositional splits in cogs ${}_{\text{cg}}$ are
multiple typed and not all about length difference. Nonetheless, in the
strucural challenge portion of the cogs ${}_{\text{cg}}$, the average text
length is 61 in cogs ${}_{\textsc{cg}}$, compared to 22 in cogs
${}_{\textsc{base}}$. On the other hand, scan ${}_{\textsc{mcd1}}$ and scan
${}_{\textsc{mcd2}}$ have average text lengths of 7.0 and 7.7, respectively.
The scan ${}_{\text{std}}$, which is the standard split, does not improve as
it is not a compositional split.

###  5.4 How Much Can We Improve Performance of COGS by Generating Very
Similar Pre-Finetuning Tasks?

While duel performed well on the scan and geoquery target tasks, performance
on cogs was less impressive, especially on the structural generalization cases
(Table 2). It has recently been shown that the challenges presented by cogs
can be addressed using symbolic methods, albeit ones that are informed by
detailed knowledge of the generative process that was used to create the
dataset (Liu et al., 2021). Can we make additional progress on this dataset
using a generic sequence-to-sequence neural model?

We hypothesized that the pre-finetuning splits we used before fine-tuning on
cogs were insufficiently similar to the split required for cogs (unlike, for
example, the scan length split that correspond to the generalization behavior
required for geo ${}_{\text{len}}$). To test this hypothesis, as a lower
bound on duel’s ability to extract inductive bias for compositional
generalization, we designed a new experiment such that the pre-finetuning
tasks will have the same compositional splits as the cogs.

Concretely, we created ten cogs variants with the same type of splits as cogs
${}_{\text{cg}}$; the only difference between the variants was that each of
them used a different lexicon. To create these variants, we first used the
SpaCy Python library to identify 3 part-of-speech classes: PROPN, NOUN and
VERB. For each proper noun, we selected 5 different alternatives. For each
noun and verb, we selected all the synonyms and antonyms from WordNet that had
the same part-of-speech. To create a single cogs variant, for each word in the
cogs vocabulary, we built a 1-to-1 replacement mapping by randomly selecting
one alternative, and generated new sentences by the mapping. The following
pair illustrates the mapping with an example (and more in Appendix C).:

cogs: $\displaystyle\texttt{Emma ate the ring beside a bed.}\longrightarrow$
cogs_var: Trudy consumed the hoop beside a layer.

The lexicons of the variants and the original cogs were completely disjoint in
those part-of-speech classes; other words (in particular function words were
the same across all variants. We use cogs_var ${}_{\text{cg}}$ to denote the
CG split of a single cogs variant. For pre-finetuning, we used 1, 5 or 10
variants.

Table 4: Accuracy of T5-Base on cogs’s structural challenge, with different pre-finetuned tasks  Pre-finetuning |  scan ${}_{\text{cd}}$ |  geo ${}_{\text{cd}}$ | 1$\times$cogs_var ${}_{\text{cg}}$ | 5$\times$cogs_var ${}_{\text{cg}}$ |  $10\times$ cogs_var ${}_{\text{cg}}$
---|---|---|---|---|---
merged | 3.4 | 4.2 | 4.9 | 5.1 | 4.8
duel | 3.9 | 4.5 | 4.9 | 5.2 | 5.2

The results reported in Table 4 suggest that pre-finetuning with the same type
of compositional generalization split only leads to a very minor improvement
in accuracy, from 4.5% to 5.2%. Note that the pre-finetuning tasks have an
average accuracy of 79.3% on the structural generalization portions of those
tasks. Our method is thus surprisingly ineffective in this case, even when the
pre-finetuning task is very similar in structure to the target task. For
future research, we plan to investigate the reasons of the limited improvement
and approaches for extracting stronger inductive bias.

##  6 Related Work

#### Transfer Learning

A large body of work has attemped to leveraged multi-task learning to endow a
model with an inductive bias that improves generalization on a main task of
interest (Caruana, 1998; Bakker & Heskes, 2003; Raffel et al., 2020), with
recent work in NLP sharing our focus on neural networks (Sogaard & Goldberg
2016; Hashimoto et al. 2016; Swayamdipta et al. 2018; for a review, see Ruder
2017). Intermediate training of pre-trained sentence encoders on a task or a
set of tasks that are related to the task of interest has been advocated,
among others, by Phang et al. (2018) and Aghajanyan et al. (2021). Gururangan
et al. (2020) craft a training pipeline where a pre-trained language model is
adapted to domain-specific and task-specific ones. There are also empirical
studies of transferrability among tasks (Vu et al., 2020). Our work differs
from most of the work in this area in that our goal is to learn to generalize
compositionally from a training distribution to a test one; as such we provide
the model with a strategic training and test set split drawn from a pre-
finetuning task, which together illustrate the desired generalization behavior
that we intend for the model to transfer to a target task. This approach bears
some resemblance to work on learning-to-learn or meta-learning (Thrun & Pratt,
1998; Gu et al., 2018), and a recent study on learning low-resource tasks
(Chen et al., 2020a).

#### Compositional Generalization

A number of approaches have been explored to achieve compositional
generalization in neural models for NLP. Specifically, recent work has
proposed new or modified model architectures (Li et al., 2019; Russin et al.,
2019; Gordon et al., 2020; Liu et al., 2020; Nye et al., 2020; Chen et al.,
2020b; Zheng & Lapata, 2020; Oren et al., 2020; Herzig & Berant, 2020; Shaw et
al., 2021; Yin et al., 2021). Furrer et al. (2020) compared several of these
architectures with general-purpose pre-trained models such as T5. Other work
has explored intermediate representations (Herzig et al., 2021) and
compositional data augmentation procedures (Andreas, 2020; Akyürek et al.,
2020).

Our approach is most closely related to work that applies meta-learning to
compositional generalization (Conklin et al., 2021; Oren et al., 2021; Lake,
2019). Conklin et al. (2021) focus on using tasks that are similar to the
target task to regularize the learning of neural models. By systematically
generating compositional splits on the synthetic dataset SCAN, Oren et al.
shows that the success of meta-learning depends on the specific splits that
are presented to the learner (Oren et al., 2021). Our work complements these
studies by showing that it is possible to transfer an inductive bias learned
on one task to a substantially different task.

##  7 Conclusion

In this paper, we studied acquiring compositional generalization skills
through a pre-finetuning task, where a compositional split is available, and
transferring those skills to a target task where such a compositional split
does not exist. We proposed duel, a simple method for this purpose. The
underlying intuition of this method is that we would like to learn an input
representation that incorporates a compositional inductive bias, and train it
separately from a task head that performs a specific task. We demonstrated the
effectiveness of this method on 3 semantic parsing tasks. In particular, using
the COGS and SCAN synthetic tasks as pre-finetuning tasks, we were able to
achieve high compositional generalization accuracy on two splits of the non-
synthetic geoquery dataset: the Target Maximum Compound Divergence split geo
${}_{\text{cd}}$ and the length split geo ${}_{\text{len}}$. For geo
${}_{\text{cd}}$, our approach beats the state-of-the-art, established by a
hybrid neurosymbolic model (Shaw et al., 2021); for geo ${}_{\text{len}}$, we
substantially improved over the baselines.

#### Reproducibility Statement

To ensure reproducibility, we describe the datasets and their train/test
splits for our experiments in Section 3 and Section 5.1. We explain our
algorithm with pseudocode in Section 4 Algorithm 1. We also describe the model
architecture and baselines in Section 5.1. Further, we provide detailed hyper-
parameters and hyper-parameter tuning methods in Appendix A. We give examples
of preprocessed inputs to T5 for each dataset in Appendix B. We will release
the code at https://github.com/Sha-Lab/duel.

## References

  * Aghajanyan et al. (2021) Armen Aghajanyan, Anchit Gupta, Akshat Shrivastava, Xilun Chen, Luke Zettlemoyer, and Sonal Gupta.  Muppet: Massive multi-task representations with pre-finetuning.  _arXiv preprint arXiv:2101.11038_ , 2021.
  * Akyürek et al. (2020) Ekin Akyürek, Afra Feyza Akyurek, and Jacob Andreas.  Learning to recombine and resample data for compositional generalization.  _ArXiv_ , abs/2010.03706, 2020.
  * Andreas (2020) Jacob Andreas.  Good-enough compositional data augmentation.  In _ACL_ , 2020.
  * Bakker & Heskes (2003) Bart Bakker and Tom Heskes.  Task clustering and gating for Bayesian multitask learning.  _Journal of Machine Learning Research_ , 4:83–99, 2003\.  URL http://jmlr.org/papers/v4/bakker03a.html.
  * Caruana (1998) Rich Caruana.  Multitask learning.  In Sebastian Thrun and Lorien Pratt (eds.), _Learning to learn_ , pp. 95–133. Kluwer Academic Publishers, Boston, 1998.
  * Chen et al. (2020a) Xilun Chen, Asish Ghoshal, Yashar Mehdad, Luke Zettlemoyer, and Sonal Gupta.  Low-resource domain adaptation for compositional task-oriented semantic parsing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pp. 5090–5100, Online, November 2020a. Association for Computational Linguistics.  doi: 10.18653/v1/2020.emnlp-main.413.  URL https://www.aclweb.org/anthology/2020.emnlp-main.413.
  * Chen et al. (2020b) Xinyun Chen, Chen Liang, Adams Wei Yu, Dawn Song, and Denny Zhou.  Compositional generalization via neural-symbolic stack machines, 2020b.
  * Cho et al. (2014) Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio.  Learning phrase representations using RNN encoder–decoder for statistical machine translation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pp. 1724–1734, Doha, Qatar, October 2014. Association for Computational Linguistics.  doi: 10.3115/v1/D14-1179.  URL https://aclanthology.org/D14-1179.
  * Conklin et al. (2021) Henry Conklin, Bailin Wang, Kenny Smith, and Ivan Titov.  Meta-learning to compositionally generalize.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pp. 3322–3335, Online, August 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.acl-long.258.  URL https://aclanthology.org/2021.acl-long.258.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pp. 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.  doi: 10.18653/v1/N19-1423.  URL https://www.aclweb.org/anthology/N19-1423.
  * Finegan-Dollak et al. (2018) Catherine Finegan-Dollak, Jonathan K. Kummerfeld, Li Zhang, Karthik Ramanathan, Sesh Sadasivam, Rui Zhang, and Dragomir Radev.  Improving text-to-SQL evaluation methodology.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 351–360, Melbourne, Australia, July 2018. Association for Computational Linguistics.  doi: 10.18653/v1/P18-1033.  URL https://www.aclweb.org/anthology/P18-1033.
  * Furrer et al. (2020) Daniel Furrer, Marc van Zee, Nathan Scales, and Nathanael Schaerli.  Compositional generalization in semantic parsing: Pre-training vs. specialized architectures.  _CoRR_ , abs/2007.08970, 2020.  URL https://arxiv.org/abs/2007.08970.
  * Gordon et al. (2020) Jonathan Gordon, David Lopez-Paz, Marco Baroni, and Diane Bouchacourt.  Permutation equivariant models for compositional generalization in language.  In _International Conference on Learning Representations_ , 2020.  URL https://openreview.net/forum?id=SylVNerFvr.
  * Gu et al. (2018) Jiatao Gu, Yong Wang, Yun Chen, Victor O. K. Li, and Kyunghyun Cho.  Meta-learning for low-resource neural machine translation.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pp. 3622–3631, Brussels, Belgium, October-November 2018. Association for Computational Linguistics.  doi: 10.18653/v1/D18-1398.  URL https://www.aclweb.org/anthology/D18-1398.
  * Gururangan et al. (2020) Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith.  Don’t stop pretraining: Adapt language models to domains and tasks.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pp. 8342–8360, Online, July 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.acl-main.740.  URL https://www.aclweb.org/anthology/2020.acl-main.740.
  * Hashimoto et al. (2016) Kazuma Hashimoto, Caiming Xiong, Yoshimasa Tsuruoka, and Richard Socher.  A joint many-task model: Growing a neural network for multiple NLP tasks.  In _NIPS 2016 Continual Learning and Deep Networks Workshop_ , 2016\.
  * Herzig & Berant (2020) Jonathan Herzig and Jonathan Berant.  Span-based semantic parsing for compositional generalization.  _arXiv preprint arXiv:2009.06040_ , 2020.
  * Herzig et al. (2021) Jonathan Herzig, Peter Shaw, Ming-Wei Chang, Kelvin Guu, Panupong Pasupat, and Yuan Zhang.  Unlocking compositional generalization in pre-trained models using intermediate representations.  _arXiv preprint arXiv:2104.07478_ , 2021.
  * Hupkes et al. (2020) Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni.  Compositionality decomposed: How do neural networks generalise?  _Journal of Artificial Intelligence Research_ , 67:757–795, 04 2020.  doi: 10.1613/jair.1.11674.
  * Kate et al. (2005) Rohit J Kate, Yuk Wah Wong, and Raymond J Mooney.  Learning to transform natural to formal languages.  In _Proceedings of the National Conference on Artificial Intelligence_ , volume 20, pp. 1062. Menlo Park, CA; Cambridge, MA; London; AAAI Press; MIT Press; 1999, 2005.
  * Keysers et al. (2020) Daniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, Dmitry Tsarkov, Xiao Wang, Marc van Zee, and Olivier Bousquet.  Measuring compositional generalization: A comprehensive method on realistic data.  In _ICLR_ , 2020.  URL https://arxiv.org/pdf/1912.09713.pdf.
  * Kim & Linzen (2020) Najoung Kim and Tal Linzen.  COGS: A compositional generalization challenge based on semantic interpretation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pp. 9087–9105, Online, November 2020\. Association for Computational Linguistics.  doi: 10.18653/v1/2020.emnlp-main.731.  URL https://www.aclweb.org/anthology/2020.emnlp-main.731.
  * Lake (2019) Brenden M Lake.  Compositional generalization through meta sequence-to-sequence learning.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), _Advances in Neural Information Processing Systems_ , volume 32, pp. 9791–9801. Curran Associates, Inc., 2019.  URL https://proceedings.neurips.cc/paper/2019/file/f4d0e2e7fc057a58f7ca4a391f01940a-Paper.pdf.
  * Lake & Baroni (2018) Brenden M. Lake and Marco Baroni.  Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks.  In _Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018_ , pp. 2879–2888, 2018.
  * Li et al. (2019) Yuanpeng Li, Liang Zhao, Jianyu Wang, and Joel Hestness.  Compositional generalization for primitive substitutions.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pp. 4284–4293, Hong Kong, China, November 2019. Association for Computational Linguistics.  doi: 10.18653/v1/D19-1438.  URL https://www.aclweb.org/anthology/D19-1438.
  * Liu et al. (2021) Chenyao Liu, Shengnan An, Zeqi Lin, Qian Liu, Bei Chen, Jian-Guang Lou, Lijie Wen, Nanning Zheng, and Dongmei Zhang.  Learning algebraic recombination for compositional generalization, 2021\.
  * Liu et al. (2020) Qian Liu, Shengnan An, Jian-Guang Lou, Bei Chen, Zeqi Lin, Yan Gao, Bin Zhou, Nanning Zheng, and Dongmei Zhang.  Compositional generalization by learning analytical expressions.  _Advances in Neural Information Processing Systems_ , 33, 2020.
  * Müller et al. (2019) R. Müller, Simon Kornblith, and Geoffrey E. Hinton.  When does label smoothing help?  In _NeurIPS_ , 2019.
  * Nye et al. (2020) Maxwell I Nye, Armando Solar-Lezama, Joshua B Tenenbaum, and Brenden M Lake.  Learning compositional rules via neural program synthesis.  _arXiv preprint arXiv:2003.05562_ , 2020.
  * Oren et al. (2020) Inbar Oren, Jonathan Herzig, Nitish Gupta, Matt Gardner, and Jonathan Berant.  Improving compositional generalization in semantic parsing.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pp. 2482–2495, Online, November 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.findings-emnlp.225.  URL https://www.aclweb.org/anthology/2020.findings-emnlp.225.
  * Oren et al. (2021) Inbar Oren, Jonathan Herzig, and Jonathan Berant.  Finding needles in a haystack: Sampling structurally-diverse training sets from synthetic data for compositional generalization, 2021.
  * Phang et al. (2018) Jason Phang, Thibault Févry, and Samuel R Bowman.  Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks.  _arXiv preprint arXiv:1811.01088_ , 2018.
  * Pruksachatkun et al. (2020) Yada Pruksachatkun, Jason Phang, Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang, Clara Vania, Katharina Kann, and Samuel R. Bowman.  Intermediate-task transfer learning with pretrained language models: When and why does it work?  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pp. 5231–5247, Online, July 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.acl-main.467.  URL https://www.aclweb.org/anthology/2020.acl-main.467.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67, 2020.  URL http://jmlr.org/papers/v21/20-074.html.
  * Ruder (2017) Sebastian Ruder.  An overview of multi-task learning in deep neural networks.  _arXiv preprint arXiv:1706.05098_ , 2017.
  * Russin et al. (2019) Jake Russin, Jason Jo, Randall C O’Reilly, and Yoshua Bengio.  Compositional generalization in a deep seq2seq model by separating syntax and semantics.  _arXiv preprint arXiv:1904.09708_ , 2019.
  * Shaw et al. (2021) Peter Shaw, Ming-Wei Chang, Panupong Pasupat, and Kristina Toutanova.  Compositional generalization and natural language variation: Can a semantic parsing approach handle both?  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pp. 922–938, Online, August 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.acl-long.75.  URL https://aclanthology.org/2021.acl-long.75.
  * Sogaard & Goldberg (2016) Anders Sogaard and Yoav Goldberg.  Deep multi-task learning with low level tasks supervised at lower layers.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pp. 231–235, Berlin, Germany, August 2016. Association for Computational Linguistics.
  * Swayamdipta et al. (2018) Swabha Swayamdipta, Sam Thomson, Kenton Lee, Luke Zettlemoyer, Chris Dyer, and Noah A. Smith.  Syntactic scaffolds for semantic structures.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pp. 3772–3782, Brussels, Belgium, October-November 2018. Association for Computational Linguistics.  doi: 10.18653/v1/D18-1412.  URL https://www.aclweb.org/anthology/D18-1412.
  * Tang & Mooney (2001) Lappoon R Tang and Raymond J Mooney.  Using multiple clause constructors in inductive logic programming for semantic parsing.  In _European Conference on Machine Learning_ , pp. 466–477. Springer, 2001.
  * Thrun & Pratt (1998) Sebastian Thrun and Lorien Pratt.  _Learning to learn_.  Kluwer Academic Publishers, 1998.
  * Turc et al. (2019) Iulia Turc, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Well-read students learn better: The impact of student initialization on knowledge distillation.  _CoRR_ , abs/1908.08962, 2019.  URL http://arxiv.org/abs/1908.08962.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), _Advances in Neural Information Processing Systems 30_ , pp. 5998–6008. Curran Associates, Inc., 2017.
  * Vu et al. (2020) Tu Vu, Tong Wang, Tsendsuren Munkhdalai, Alessandro Sordoni, Adam Trischler, Andrew Mattarella-Micke, Subhransu Maji, and Mohit Iyyer.  Exploring and predicting transferability across NLP tasks.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pp. 7882–7926, Online, November 2020\. Association for Computational Linguistics.  doi: 10.18653/v1/2020.emnlp-main.635.  URL https://www.aclweb.org/anthology/2020.emnlp-main.635.
  * Yin et al. (2021) Pengcheng Yin, Hao Fang, Graham Neubig, Adam Pauls, Emmanouil Antonios Platanios, Yu Su, Sam Thomson, and Jacob Andreas.  Compositional generalization for neural semantic parsing via span-level supervised attention.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pp. 2810–2823, Online, June 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.naacl-main.225.  URL https://aclanthology.org/2021.naacl-main.225.
  * Zelle (1995) John M. Zelle.  _Using Inductive Logic Programming to Automate the Construction of Natural Language Parsers_.  PhD thesis, Department of Computer Sciences, The University of Texas at Austin, Austin, TX, 1995.  URL http://www.cs.utexas.edu/users/ai-lab?zelle:phd95.
  * Zheng & Lapata (2020) Hao Zheng and Mirella Lapata.  Compositional generalization via semantic tagging.  _arXiv preprint arXiv:2010.11818_ , 2020.

Appendix

In this supplementary material, we provide details omitted in the main text.
The content is organized as what follows:

  * •

Section A. Implementation details and optimization hyper-parameters.

  * •

Section B. Examples of T5 prompt for each of the task in our experiments.

  * •

Section C. Examples of cogs_var _vs_. cogs.

##  Appendix A Implementation details

###  A.1 Hyper-parameters in fine-tuning

First, we list the hyper-parameters in fine-tuning for each model and task.

For cogs, we fine-tune for a maximum of 30k steps with a learning rate of
$1e^{-4}$. Following Kim & Linzen (2020), we evaluate every 500 iterations on
the dev set of cogs and early-stop if the accuracy on the dev set has not
improved for 5 consecutive step. The batch size is 128 on T5-base and 512 on
bert2seq.

For geoquery, it has no dev set. Therefore, we tune hyper-parameters on the
standard split of the geoquery dataset. We then use the same hyper-parameters
for the geo ${}_{\text{cd}}$ and geo ${}_{\text{len}}$ splits. On T5-base,
we fine-tune for 5k steps with a batch size of 256 and a learning rate of
$1e^{-4}$. On bert2seq, we fine-tune for 3k steps with a batch size of 512 and
a learning rate of $2e^{-5}$.

For scan, likewise, we tune hyper-parameters on the standard split of it. We
then use the same hyper-parameters for the scan ${}_{\text{cd}}$ and scan
${}_{\text{len}}$ splits. On T5-base, we fine-tune on for 3.5k steps with a
batch size of 128 and a learning rate of $1e^{-4}$. On bert2seq, we fine-tune
for 2.5k steps with a batch size of 512 and a learning rate of $1e^{-4}$.

We use the Adam optimizer with weight decay. We take the label smoothing
(Müller et al., 2019) with weight factor 0.1 on bert2seq but not T5-base.

Next, we provide the computation time for fine-tuning on 1 TITAN Xp GPU. For
cogs, the average fine-tuning time is about 24 hours on T5-base and 4 hours on
bert2seq. For geoquery, about 2 hours on T5-base and 0.4 hour on bert2seq. For
scan, about 3 hours on T5-base and 0.5 hours on bert2seq.

###  A.2 Validation dataset for hyper-parameter tuning

As we train on the whole dataset with duel during pre-finetuning, we are not
able to directly tune hyper-parameters for duel. In order to validate the
learned representation is good at OOD generalization from  $s$ to
${\tilde{s}}$ without accessing the test data in  ${\tilde{s}}$, we hold out
a validation dataset for each task from  ${\tilde{s}}$. We then take the pre-
finetuned model, re-initialize the task-head, fine-tune it on  $s$ and test
on the hold-out set of  ${\tilde{s}}$. We select the hyper-parameters of duel
based on the test accuracy on the hold-out set.

For cogs, we create a validation dataset by holding a subset of 3000 examples
from cogs ${}_{\textsc{cg}}$. Likewise, the validation datasets of geoquery
and scan hold out 20% of the data from geo ${}_{\textsc{tmcd2}}$ and scan
${}_{\textsc{mcd2}}$, respectively.

###  A.3 Hyper-parameters in pre-finetuning

In pre-finetuning, we choose the maximum outer loop iterations
$T_{\textup{outer}}$ as 10, the maximum inner loop iterations
$T_{\textup{inner}}$ as 30k. For efficiency, we evaluate every 500 iterations
and set the early-stopping patience for the inner loop as 1. We select a
learning rate $\alpha$ from [$1e^{-5},2e^{-5},1e^{-4}$] for all tasks and all
models.

The batch size is different from each model. For T5-base, we select a batch
size of 32 from [8, 16, 32] for all tasks. For bert2seq, we select a batch
size of 512 from [256, 512, 1024].

The early-stopping minimum iterations $T_{\textup{min}}$ for the outer loop is
different for each model and task. For cogs, $T_{\textup{min}}=3000$ on
T5-base and $T_{\textup{min}}=2000$ on bert2seq. For geoquery,
$T_{\textup{min}}=1000$ on both T5-base and bert2seq. For scan,
$T_{\textup{min}}=2000$ on T5-base and $T_{\textup{min}}=1000$ on bert2seq. We
use the same optimizer and label smoothing method in fine-tuning as in pre-
finetuning.

Next, we provide the computation time for duel on 1 TITAN Xp GPU. When pre-
finetuning on cogs, the average running time is about 60 hours on T5-base and
24 hours on bert2seq. For geoquery, about 24 hours on T5-base and 2 hours on
bert2seq. And for scan, about 30 hours on T5-base and 3 hours on bert2seq.

##  Appendix B T5 prompt details

In this section, we provide examples of T5 prompt for each task in our
experiment. Our design of task prompt follows the standard way from Raffel et
al. (2020)

###  B.1 cogs

#### Original input:

Emma ate the ring beside a bed .

#### Prompt input:

cogs: Emma ate the ring beside a bed .

###  B.2 cogs_var

#### Original input:

The tiddler appreciated that Kim measure that a cooky was slid .

#### Prompt input:

cogs: The tiddler appreciated that Kim measure that a cooky was slid .

###  B.3 geoquery

#### Original input:

what is the largest state capital in population

#### Prompt input:

geo: what is the largest state capital in population

###  B.4 scan

#### Original input:

run around right after look opposite left

#### Prompt input:

scan: run around right after look opposite left

##  Appendix C More examples on cogs_var vs cogs

For each cogs input sentence listed following, we give three examples of the
corresponding cogs_var input sentences.

####  cogs input sentence 1:

The sailor dusted a boy .

####  cogs_var input sentence 1:

The crewman scattered a female_child .

The boater dotted a male_child .

The leghorn dotted a son .

####  cogs input sentence 2:

The cookie was passed to Emma .

####  cogs_var input sentence 2:

The cooky was faded to Joan .

The cooky was overstepped to Trudy .

The cooky was eliminated to Kim .

####  cogs input sentence 3:

Zoe thought that a hippo cleaned .

####  cogs_var input sentence 3:

Larry cerebrated that a Hippo housecleaned .

Dana guessed that a hippopotamus picked.

Ronnie retrieved that a hippopotamus make_cleaned .

####  cogs input sentence 4:

The zebra sold Ethan a cake in a cupboard in a castle on the road in the hole
beside the trainee on the table in a house in a room in the box beside a
sailor in a microwave .

####  cogs_var input sentence 4:

The zebra traded Lori a bar in a cupboard in a castling on the route in the
hollow beside the trainee on the mesa in a theater in a way in the boxful
beside a leghorn in a microwave_oven .

The zebra dealt Raul a patty in a cupboard in a castling on the route in the
hollow beside the trainee on the mesa in a menage in a elbow_room in the loge
beside a skimmer in a microwave_oven .

The zebra traded Ethan a patty in a cupboard in a castling on the route in the
golf_hole beside the trainee on the tabular_array in a star_sign in a
elbow_room in the boxful beside a skimmer in a microwave_oven .


# Conducting lines - drawable conductors as a source of active learning
activities

Petar Damjanović  Užice Grammar School, Trg Svetog Save 6, 31000 Užice, Serbia
Vladimir Veljić  Scientific Computing Laboratory, Center for the Study of
Complex Systems,
Institute of Physics Belgrade, University of Belgrade,
Pregrevica 118, 11080 Belgrade, Serbia  Aleksandra Alorić  Scientific
Computing Laboratory, Center for the Study of Complex Systems,
Institute of Physics Belgrade, University of Belgrade,
Pregrevica 118, 11080 Belgrade, Serbia

###### Abstract

Drawable electronics attract the attention of both educators and innovative
circuit engineers due to their affordability and simplicity. This paper
focuses on active learning activities related to conducting lines and presents
extensible laboratory projects suitable for students of all levels - from late
primary school pupils to early undergraduate students. We build on ideas of
Pouillet’s law classroom demonstrations with pencil drawings on paper and
continue with a line, circle and grid drawings and appropriate resistance
measurements. These are used as methodological examples of drawable circuits
suitable for learning about resistors in series, parallel and mixed
combinations. Particularly, grid drawings, e.g. mixed circuits, offer a
variety of exercises that can connect analytical and experimental approaches
and instigate discussions about assessments of uncertainty. Along the way, we
propose ideas for extracurricular projects students can undertake to further
their experimental knowledge and practise the scientific method.

††preprint: AIP/123-QED

##  I Introduction

The affordability of paper and the simplicity with which customizable electric
circuits can be drawn using graphite pencils and other conducting inks, make
it appealing for drawable paper electronics Tobjörk and Österbacka (2011);
Kurra, Dutta, and Kulkarni (2013); Kurra and Kulkarni (2013); Costa  _et al._
(2018), foldable electronics Liu  _et al._ (2017), pencil–paper-based on-skin
electronics Xu  _et al._ (2020). For the same reasons, pencil and paper can be
utilized in any classroom to effectively demonstrate key messages of basic and
more advanced electronics curriculum Inyeo and Wattanakasiwich (2018); Derman
and Goykadosh (1999); Marshall (2003).

This paper focuses on the educational aspect of conducting lines and the ways
in which this simple experimental set-up can be used to illustrate physics
phenomena typically taught in theoretical and mathematically technical terms.
The topic of electrical conductivity is commonly present in the early high
school physics (or STEM) curriculum111In Serbian curriculum, the topic is
already introduced in the eight (final) grade of the primary school (students’
age of 15) and then discussed more deeply in secondary school, usually in
second or third grade depending on the school’s profile., and as such shows
great potential for a variety of experimental approaches and hypothesis
testing of interest for students of different age and knowledge level.
Building on ideas for classroom demonstrations Kamata and Abe (2012);
Chiaverina (2014), we show that a similarly simple experimental set-up can be
used to introduce students to systematic experimentation and the scientific
method. Furthermore, these topics can motivate a plethora of ideas for
independent research regardless of the availability of extraordinary research
facilities.

The research topic and the student-led research approach have been greatly
inspired by the International Young Physicist’s Tournament (IYPT)Martchenko
(2013); Haskell, Britton, and Edge (2000); Plesch, Plesník, and Ružičková
(2020). The IYPT is a competition that fosters scientific investigation and
debate among high school students for more than thirty years. Every year 17
open-ended problems are chosen by the International Organizing Committee and
students start their individual and team investigations that culminate a year
later with a series of debates, e.g. ’physics fights’ when students challenge
each others’ scientific results. One problem in the 2020 cycle of the IYPT
initiated the study presented here: “A line drawn with a pencil on paper can
be electrically conducting. Investigate the characteristics of the conducting
line.” IYP (2019) As with many other IYPT problems, this simple formulation of
the phenomena invites students to do in-depth literature research, come up
with various hypotheses and devise an adequate experimental setup to test
them.

This makes the IYPT problem archive IYP  an excellent database for student-led
project work Planinsic (2009). However, devising classroom interventions
inspired by these problems that engage a wider group of students (not only
those already interested in physics) is not an easy task. That is why in this
paper we provide detailed ideas and instructions for classroom activities
ranging from a simple class demonstration (II), over ideas for physics
laboratory (or physics practicum, III) to project ideas (IV) that can be used
to foster Investigative Science Learning Environment (ISLE) Etkina (2015) in
classrooms.

##  II Classroom demonstration

Despite the popularity of conducting lines in physics education literature
Chiaverina (2014); Küçüközer (2015); Kamata and Abe (2012), science
exhibitions mus  and many commercial kits cir , the topic of electronic
circuits and resistance is often thought of in mathematically technical terms
through Ohm’s law or Pouillet’s law. And although many students acquire these
equations and use them readily to solve exam problems, this does not imply
comprehension of concepts used Ibrahim and Rebello (2012). This is why we
suggest that every lecture on electrical resistance starts with a simple
demonstration Kamata and Abe (2012) shown in Figure 1. This demonstration
needs only a single battery (9 V battery was used), a LED, paper and pencil
(8B pencil was used), but it can help sparkle discussion among students and
generate questions, e.g. given fixed input voltage (battery) is the light
brighter for a shorter or longer line. What about line thickness? What if the
line is closed?

Figure 1: A simple and safe home experiment showing that in a circuit with a
shorter line the diode lights brighter. This is an illustration from a
student’s submission during IYPT 2019 competition in Serbia, published with
the permission of the author Una Jaćimović.

Alternatively, pencil leads (instead of drawn lines) can be used for
demonstration and students can compare LED brightness for the conductors of
the same length Inyeo and Wattanakasiwich (2018); Küçüközer (2015). As
discussed previously Küçüközer (2015); Woolf and Streckert (1996), pencil
leads can be joined together (in series or in parallel), or lines of different
lengths and width can be drawn to introduce key conductor’s shape properties
that regulate its resistance. Thus, students can grasp Pouillet’s law
inductively, observing the demonstrations

$R\propto\frac{l}{A}\,,$ (1)

where $R$ denotes electrical resistance, $l$ the resistor’s length, $A$ its
cross-sectional area. Furthermore, using pencils with different graphite
content (e.g. HB scale, higher B numbers correspond to higher graphite to clay
ratios Sousa and Buchanan (2000)), students can deduce that the constant of
proportionality (resistivity $\rho$) is material dependent while using
dimensional analysis, they can deduce the constant’s units. Only then, would
we suggest that the educators take the path of introducing the full
theoretical framework as described here Grisales, Herrera, and Fajardo (2016).

Building on these demonstrations and the discussions it has inspired among
students, educators can make interdisciplinary connections with chemistry
diving deeper into explanations of graphite structure Adorno, Bellomonte, and
Pizzolato (2017).

##  III Experiments suitable for individual or group based activities

Expanding on students’ discussions initiated with simple demonstrations
presented in the previous section, the next step is to devise experiments to
test students’ generated hypotheses (e.g. dependencies formulated by
Pouillet’s law) in a systematic manner.

In this section, we will overview three groups of experiments - serial,
parallel and mixed circuits made of conducting lines. Materials needed include
pencils of different graphite content (we used pencils with high graphite
content: 5B to 8B), matte paper (most of the printing papers without a glossy
coat are suitable, we used standard $80\>\rm g/m^{2}$ printing paper) and a
digital ohmmeter. Due to the sensitivity of the drawn conductors to the
pressure applied in drawing, all the experimental results presented here are
drawn by the same person, following the same protocol - line of the desired
length and fixed width is traced six times, aiming to apply the same pressure
while drawing. Unless otherwise stated the line length was $12\>\rm cm$, while
the width was $1\>\rm mm$. The resistance of a given conductor was always
measured five times placing probes at a fixed distance in a way that maximises
the probe and line contact. In the plots, we show mean resistance with error
bars denoting maximal measurement deviation from the mean. To exclude possible
effects of the paper’s roughness anisotropy, all the lines were drawn along
the same paper’s direction.

###  III.1 Resistor lines - a series circuit

The investigation of graphite lead resistance as a function of its length
Inyeo and Wattanakasiwich (2018); Küçüközer (2015), and similarly for the
graphite drawn lines Chiaverina (2014); Grisales, Herrera, and Fajardo (2016);
Derman and Goykadosh (1999); Kamata and Abe (2012) is a simple introductory
laboratory work students can do to test the previously generated hypothesis
that resistance is proportional to resistor’s length. Here we suggest that
besides reinforcing Poulette’s law in students’ minds, the same exercise can
be used to think about the equivalent resistance of resistors connected in
series.

Figure 2: Illustration of the series circuit. Figure 3: Left: Resistance of
centimetre-long line segments within the longer graphite line (5B pencil was
used). Middle: Line resistance as a function of its length: measurement and
serial resistance calculation comparison (5B pencil). Right: Line resistance
as a function of its length for different pencil types (5-8B pencils).

The students start by drawing a line resistor of fixed width and length. In
the experiments shown in Fig 3 the lines were $12\>\rm mm$ long and $1\>\rm
mm$ wide. The first measurements students can perform are resistance
measurements of the short line segments (in Fig. 3 (Left) we show results for
$1\>\rm cm$). Through this measurement, students effectively investigate the
uniformity of the drawn line. As demonstrated in Fig. 3 (Left), the segments’
resistance variation is within the margins of error, confirming our assumption
that the cross-section of graphite line, as well as its resistivity, can be
treated as constant along the drawn line. With the measurements of individual
resistances of line segments $l_{i}=1\>\rm cm$, students can proceed to
calculate resistance for any length $l$ realising that the same way the total
length can be broken into segments

$l=l_{1}+l_{2}+\dots+l_{k}+\dots+l_{n-1}+l_{n}\,.$ (2)

As illustrated in Fig. 2, students can compare measured resistance of any
given line length, but also calculate it using individual measurements of
segments’ resistances

$R_{e}=R_{1}+R_{2}+\dots+R_{k}+\dots+R_{n-1}+R_{n}\,.$ (3)

Results of one such comparison is shown in Fig. 3 (Middle). This simple lab
exercise where a single line’s resistance is directly measured and calculated
based on multiple measurements can lead to a fruitful discussion about
measurement errors - the errors always increase when multiple measurements are
combined. This point is clearly demonstrated by increasing margins of error in
the Middle panel of Fig. 3.

Finally, we suggest that students proceed to measure how resistance varies
with the length for pencils with different graphite content (in Fig. 3 (Right)
we use pencils 5B to 8B). The key message is an experimental confirmation of
the demonstration induced hypothesis that resistance increases with the
resistor’s length. Also, students can realize that resistance increases faster
for pencils with lower graphite content (5B pencil’s slope is higher than 8B
pencil’s slope), which can help students deduce that the non-graphite content
in pencils is a non-conducting material, usually clay.

The results of this exercise students can compare with results presented
previously Inyeo and Wattanakasiwich (2018); Grisales, Herrera, and Fajardo
(2016); Chiaverina (2014), but also calculate resistivity per line’s thickness
Inyeo and Wattanakasiwich (2018) from experimental line slopes. Additionally,
comparing the resistivity of a drawn line with the one of the pencil lead
where all the dimensions can be measured, students can try to estimate the
thickness of the drawn line Woolf and Streckert (1996).

###  III.2 Resistor circles - a parallel circuit

Figure 4: Illustration of the parallel circuit. Figure 5: Left: The equivalent
resistance of a circular conductor as a function of angular separation of the
probes. Middle: The equivalent resistances of circles with different radius.
Right: The equivalent resistances of circles drawn with different pencils.

Contrary to the graphite series circuit demonstrations and particularly,
resistance as a function of length, the graphite parallel circuits classroom
explorations are less common. Kamata and Abe Kamata and Abe (2012) use
parallel conducting lines circuits with LED to show that the same resistor
lengths connected in parallel result in brighter LED. Other authors Chiaverina
(2014); Woolf and Streckert (1996); Marshall (2003); Küçüközer (2015) address
parallel circuits with proposed activities such as drawing parallel lines and
connecting them in a circuit, for detailed instruction see Derman and
Goykadosh Derman and Goykadosh (1999).Here instead, we propose a series of
systematic measurements that can be done on a resistor drawn in form of a
circle.

A circle-shaped resistor of radius $r$, cross-section area $A$ and resistivity
$\rho$ divided in two segments by probes placed at angular distance $\theta$
(see Fig. 4) exemplify a parallel circuit of two resistors. The reciprocal of
the equivalent resistance of resistors connected in parallel equals the sum of
reciprocals of each resistor in the circuit

$\frac{1}{R_{e}}=\frac{1}{R_{1}}+\frac{1}{R_{2}}\,.$ (4)

Both resistors’ resistances can be calculated (as discussed in previous
section) by using the appropriate arcs’ lengths, e.g. $l_{1}=r\theta$ and
$l_{2}=r(2\pi-\theta)$. Thus the equivalent resistance of a circular resistor
measured with the probes separated by angle $\theta$ reads

$R_{e}=\frac{\rho r}{A}\frac{\theta(2\pi-\theta)}{2\pi}\,.$ (5)

Getting this $R_{e}(\theta)$ dependence represents an interesting mathematical
exercise, but at the same time excited experimental challenge since measured
dependency between resistance and angle is not linear as students often expect
Van Dooren  _et al._ (2004).

We propose that students do this experiment in pairs or groups focusing on
different possible parameters they might want to explore, e.g. circles of
different radius, different number of line tracings over the circle, different
pencil grade, etc. To compare different results, we propose that the
equivalent resistance is normalized with $R_{e}(\theta=\pi)$, which yields to

$\frac{R_{e}(\theta)}{R_{e}(\theta=\pi)}=\frac{\theta(2\pi-\theta)}{\pi^{2}}\,.$
(6)

As the normalized equation is just $\theta$ dependent it represents a
universal result. In other words, this function is independent of other line’s
parameters and thus offers an excellent opportunity for multiple student
groups’ results to be compared together against the same theoretical line.
Since this $\theta$-dependence has the shape of an inverted parabola, it is
worth emphasizing that it reaches a maximum value of 1 when $\theta=\pi$,
which corresponds to the maximum possible equivalent resistance of this
configuration. Analogously, the minimum value of this function is 0 when
$\theta=0$ or $\theta=2\pi$, which corresponds to the minimum theoretical
equivalent resistance of this configuration.

In Figure 5 we present measurements from several experiments against the
theoretically obtained function (Eq. (6)) with overall excellent agreement.
The circles of stated radius were drawn by outlining different cylinders. One
of the probes was fixed, while the other was moved along the circular arc in
steps of $\pi/12$.

In Fig. 5 (Left) we show full range of angles for the circle with radius
$r=9\>\rm{cm}$ traced with 6B pencil and compare it with the theoretical
expectation. For this combination of parameters, we demonstrate the full shape
of the parabola to highlight symmetry and demonstrate uniformity of the drawn
circle (as results for $\theta>\pi$ correspond to the analogous circuits to
those $\theta<\pi$ if the angle is measured in the negative mathematical
direction). On the other hand, the middle and right panels of the same figure
depict the universality of the equivalent resistance for different choices of
line parameters (radius and pencil type). Namely, normalization given by Eq.
(6) allows us to obtain magnificent quantitative agreement with experimental
results as it collapses data onto a single curve.

###  III.3 Resistor grids - a mixed circuit

In the previous two sections, we have seen that resistors in series and
parallel circuits, can be experimentally addressed with drawable resistors.
Here we continue with mixed circuits. Particularly, we analyze mixed circuits
that consist of a variable number of identical cells connected in parallel.
The cells are constructed of two resistors $R_{1}$ and one resistor $R_{2}$
connected in series, as depicted in Fig. 6. This growing mixed circuit allows
students to experimentally and theoretically investigate how the equivalent
resistance changes with the number of cells in the grid.

When the circuit contains only one cell ($n=1$) the equivalent resistance
$R_{e}^{(1)}$ is an equivalent resistance of resistors connected in series,
given with Eq. (7). For the case when $n=2$ the new cell can be replaced with
a single resistor of resistance calculated in $n=1$ step - $R_{e}^{(1)}$, thus
new equivalent resistance for $n=2$ can be rewritten as in Eq. (8). Similarly,
when the circuit consists of just $n=3$ cells the last two cells can be
replaced with a single resistor of resistance $R_{e}^{(2)}$, so that
equivalent resistance can be rewritten as in Eq. (9). Namely, a specific
relationship can be observed between the equivalent resistance of a network
consisting of $n$ cells with the equivalent resistance of a network consisting
of $n-1$ cells, and it is given by a recurrent relation in Eq. (10). Having
this relation we can calculate the equivalent resistance of such a mixed
circuit for an arbitrary number of cells $n$.

$\displaystyle R_{e}^{(1)}$ $\displaystyle=2R_{1}+R_{2}$ (7) $\displaystyle
R_{e}^{(2)}$
$\displaystyle=2R_{1}+\frac{R_{2}(2R_{1}+R_{2})}{R_{2}+(2R_{1}+R_{2})}$
$\displaystyle=2R_{1}+\frac{R_{2}R_{e}^{(1)}}{R_{2}+R_{e}^{(1)}}$ (8)
$\displaystyle R_{e}^{(3)}$
$\displaystyle=2R_{1}+\frac{R_{2}\left(2R_{1}+\frac{R_{2}(2R_{1}+R_{2})}{R_{2}+(2R_{1}+R_{2})}\right)}{R_{2}+\left(2R_{1}+\frac{R_{2}(2R_{1}+R_{2})}{R_{2}+(2R_{1}+R_{2})}\right)}$
$\displaystyle=2R_{1}+\frac{R_{2}R_{e}^{(2)}}{R_{2}+R_{e}^{(2)}}$ (9)
$\displaystyle\vdots$ $\displaystyle R_{e}^{(n)}$
$\displaystyle=2R_{1}+\frac{R_{2}R_{e}^{(n-1)}}{R_{2}+R_{e}^{(n-1)}}$ (10)

Figure 6: Illustration of a mixed circuit resistor grid.

As the electrical circuits as the ones shown in Fig. 6 can be drawn, we can
experimentally confirm results given by the Eqs. (7)-(10). In place of
resistors denoted with $R_{1}$ using 6B pencil we draw lines of length $1\>\rm
cm$, while for $R_{2}$ we draw lines of length $7\>\rm cm$, with all other
parameters kept as previous (e.g. width, number of traces). Before connecting
them in the circuits, we measure resistances of the lines $R_{1}=(2.6\pm
0.2)\>\rm{k\Omega}$ and $R_{2}=(29.3\pm 0.8)\>\rm{k\Omega}$. We propose that
initially only the first cell is drawn, while the sequence of resistors
$R_{2}$, or the long parallel lines, is added equidistantly (one centimetre
apart) so that only with tracing resistors $R_{1}$ the cells can be joined and
added gradually as per Fig. 6. In Fig. 7 we show the equivalent resistance of
the mixed grid for each number of cells $n$, both measured and calculated
using Eqs. (7)-(10).

We see a good agreement between the calculated and measured equivalent
resistances across different cell numbers $n$. Additionally, we note that the
equivalent resistance decreases with the number of cells. This is to be
expected, as each new cell is added in parallel to the rest of the circuit,
but this offers further space for discussion with students. Since the
equivalent resistance is decreasing with the number of cells, but it is also
positive by the definition, it is implied that it must have a lower bound.
Theoretically, this bound can be calculated in the limit of infinite number of
cells (see the last circuit in Fig. 6). The large $n$ limit allows us to
exchange both $R_{e}^{(n)}$ and $R_{e}^{(n-1)}$ with $R_{e}^{(\infty)}$ in the
Eq. (10) leading to the following relation

$R_{e}^{(\infty)}=2R_{1}+\frac{R_{2}R_{e}^{(\infty)}}{R_{2}+R_{e}^{(\infty)}}\,.$
(11)

The last equation is a quadratic equation which can be solved analytically and
its solution is

$R_{e}^{(\infty)}=R_{1}+\sqrt{R_{1}^{2}+2R_{1}R_{2}}\,.$ (12)

Note that the second solution of the quadratic equation is rejected, since
equivalent resistance has to be positive. In Fig. 7 horizontal line represents
this theoretical lower bound, i.e. infinite $n$ circuit resistance.
Calculating the large $n$ theoretical resistance value and comparing it with
experimentally accessible number of circuit cells can help student grasp
limiting values. In this particular example, we see in Fig. 7 that already
with $n=4$ cells, the limiting $n$ resistance value is reached (within error
margins). So drawing circuits with $n\geq 5$ cells is practically
indistinguishable from the point of effective resistance.

Both finite and $n\rightarrow\infty$ calculated resistances shown in Fig. 7
have error margins and calculating them can be another theoretical exercise
for students interested in error propagation. We provide these in the
Appendix.

Figure 7: Comparison of measured and calculated equivalent resistance as a
function of number of cells of mixed circuits depicted in Fig. 6.

##  IV Independent research projects

Figure 8:  Left: Line resistance as a function of the number of line traces
for two types of paper denoted matte and glossy. Right: Line resistance as a
function of time passed from the drawing. In both panels, 5B pencil was used
for drawing $10\>\rm cm$ long lines.

As it is evident from previous examples, this area of research offers a wide
variety of ideas for independent research projects. Therefore, in this
section, we propose several additional ideas that we have encountered.

Previously, we focused on the effects of line length as the main variable
affecting the resistance and we aimed to keep the line thickness constant,
e.g. by tracing the line the same number of times. In the exploratory
analysis, we observed that a single traced line had too high resistance and
was often out of our measurement reach, so we settled for retracing several
times, in our case six, to ensure all required measurements will be within
reach. To investigate systematically how the resistance depends on line
thickness and with a simple experimental set-up described so far, we varied
the number of tracings of a line with a fixed length. In Fig. 8 (Left) we show
the measured resistances obtained by the same pencil lead (5B) and two
different paper types. The line length is $10\>\rm cm$, while its other
properties are kept as previously. As expected resistance decreases when the
line is traced multiple times. This was briefly explored previously Kamata and
Abe (2012); Inyeo and Wattanakasiwich (2018), on the smaller range of line
tracing, reaching the same general conclusion. However, within our measured
range, we see that the resistance has a lower bound and we hypothesize that
there is a pencil content saturation and consequently limit in the line
resistance, for the given length and width constraints of the line. Exploring
how this limiting resistance depends on relevant paper and pencil parameters
could become an engaging student’s independent project.

Within the same experiment, we investigated the effects of different paper
types as it is clear that the conductivity of the drawn line depends also on
the paper as that affects the amount of pencil trace left on the surface. In
Fig. 8 (Left) we use two papers; one matte ($80\>\rm g/m^{2}$ printing paper
used in experiments previously) and one glossy paper. As expected, the
resistance of the line of the same length and the same number of line tracings
(drawn by a pencil of the same lead type) was higher on the glossy paper. We
hypothesize this is due to the lower amount of graphite content that is
transferred to the paper. Thus, we suggest further investigations of surface
property influence on resistance by systematic variation of paper roughness,
but also measuring line resistance when drawing on different surfaces (see for
example resistance of lines drawn on MagicTM tape Derman and Goykadosh (1999),
or a wider range of different surfaces Shmavonyan and Mailian (2015)).

Differently from the previous experiments, in Fig. 8 (Left) we show resistance
measurements for three different lines made following the same protocol,
denoted by mean and maximal deviations from it. This was done to estimate the
differences that might occur among the lines when the lines are retraced as
sometimes the graphite content might be removed with a new trace, or
variations in pressure might become more visible. Due to this reason, the
error bars are greater than elsewhere in the paper, where the key source of
error is in the probe-line contact, but we note that the difference between
line resistances drawn on different papers are statistically significant.

Finally, we report on a serendipitous discovery that could lead towards more
interesting independent research activities. Within our reproducibility
analysis, we observed that all resistance measurements a day later were offset
by a constant value, while the paper with the drawn lines was kept under
typical room conditions. This prompted a preliminary study whose results are
shown in Fig. 8 (Right). Typically prepared resistor line of length $10\>\rm
cm$ was measured every hour for the total duration of $36$ hours. We observe
that the resistance steadily increases within the first couple of hours, while
after a day a change is much slower. Investigating whether the resistance
reaches a plateau or keeps increasing at an even slower rate after the time
window presented here could be a great practice in perseverance. To exclude
the hypothesis that the increased resistance is a consequence of small
graphite removal with every measurement, for this experiment we connected
ohmmeter probes to the line and only read of resistance value hourly without
further impact to the drawn resistor. We hypothesize that resistance change
might be due to room humidity and absorption of water from the air. Confining
the experimental set-up in an aquarium with a controlled humidity level
(coupled with humidity measured by another excellent IYPT project - a hair
hygrometer IYP ) could lead to a stimulating research project.

Until further investigations that will detail the reasons behind line
resistance changes in time, in this work we ensured that all the measurements
are done within the first hour after the lines were drawn and we suggest
similar practice, especially when aiming to investigate the effects of another
parameter such as line length or width on the resistance.

##  V Discussion and Conclusions

In this paper, we have gathered a few simple, yet instructive and engaging
experiments that rely on the typical students’ supplies - pencil and paper -
but instead of using them only for theoretical physics calculation, we suggest
more active experimental application. A strong case for using pencil leads and
traces at least for demonstrations of resistance was made by Kuccukozer
Küçüközer (2015) claiming that many teachers in Turkey omitted demonstration
that resistance decays when resistor’s length increases as it was not always
evident to students when copper was used and students would even reach wrong
conclusions. On the other hand with pencil trace and the simplest set-up as in
Fig. 1 this result becomes evident. Other authors Kamata and Abe (2012) even
transform this simple demonstration to a form of black-box investigative
exercise where students’ goal is to deduce the line length or type or circuit
drawn, given the LED brightness.

Besides classroom demonstration that can be effective and engaging, in this
paper we covered a variety of ideas for extensive measurements that still do
not require advanced equipment but allow practising experimental methods and
scientific discussion between students. Due to the simplicity of the
experimental set-up, many of the proposed exercises can be done individually
or in groups. For example, in the activities presented in Sec. III.1, all
students can individually draw lines with different pencils or papers and
perform uniformity measurements (e.g. produce plots like the one in Fig. 3
(Left)). Students can then collectively discuss these results and decide which
lines are the most suitable (uniform resistance) for further $R$ versus $l$
measurements which they can perform in pairs or groups and then aggregate them
in a joint plot for discussion and conclusions. Students who worked using the
same paper & pencil combination can discuss differences in results introduced
by different drawers and these discussions could lead to assessments of error
margins and meaningfulness of different conductors (different paper/pencil)
result comparison.

Derman and Goykadosh Derman and Goykadosh (1999) suggest measurements of
circle arc resistances where linear dependence on the angle ($R\sim
l\sim\theta$), similar to the one on the line length can be observed.
Additionally, the authors report on measuring the value of $\pi$ by comparing
the resistance of the arch and diameter. These measurements can be further
discussed with parallel resistance measurements (discussed in III.2) as
students can observe that drawn circle can be used to show both: a linear
relationship between resistance and angle (when only the fragment of circle
line is used) and quadratic relation (when the full circle is drawn). The
resistance measurements discussed in Sec. III can inspire students and
teachers to further connect the physics and mathematics curriculum by drawing
different geometric shapes and using the methods discussed as ways to measure
and calculate circumference or other geometric relations. This can be further
complicated by encouraging students to create their extensions of resistor
grids (e.g. triangular) for which they can measure and calculate equivalent
resistance as described previously.

Besides ideas for further research discussed in Sec. IV students can base
their research on other types of conductive inks Grisales, Herrera, and
Fajardo (2016); cir  and repeat any of the reported experiments or some of the
proposed variations. Another interesting research direction next to exchanging
paper for another type of surface could be a step towards the foldable
electronics by either stretching, bending or folding the surface after drawing
on it Inyeo and Wattanakasiwich (2018) and investigating how that affects
measured resistances. Apart from resistors, drawings on the paper could be
used to create capacitors Grisales, Herrera, and Fajardo (2016); Salami  _et
al._ (2007) and more complicated circuits Kurra and Kulkarni (2013); Kurra,
Dutta, and Kulkarni (2013), thus similar exercises could serve for further
electronics classroom activities.

In line with previous work that advocates for measurement uncertainty as one
of the key concepts students should grasp through laboratory experiments
Trumper (2002), various exercises we reviewed offer multiple learning
opportunities in this direction. Starting with a discussion about device
measurement errors (e.g. ohmmeter, ruler, protractor), continuing with
measures of the agreement among the repetitive resistance measurement (in this
case error mostly comes from probe-line contact) to uncertainty that is due to
small variations in sample preparations (e.g. one centimetre segment
resistances). Furthermore, students can discuss uncertainties from error
propagation arising from multiple measurements (e.g. Fig. 3 (Middle) and Fig.
7).

The simplicity and availability of the experimental set-up used in this paper
allow students to familiarize themselves deeper with the topic of electric
resistance, actively participate in designing apparatus and measurements and
pose relevant research questions. Particularly, through different conducting
lines and circuit types, we reviewed in the paper, students can grasp that
straight line resistance linearly depends on its length and that circle
resistance depends quadratically on angular probe separation. Finally, these
exercises help students familiarize themselves with the scientific method of
inquiry, processes of systematic measurements and the analysis of experimental
results that are practical and applicable in a variety of different contexts
besides the electric resistance of conducting lines.

##  VI Acknowledgements

A. A. acknowledges the funding provided by the Institute of Physics Belgrade,
through the grant by the Ministry of Education, Science, and Technological
Development of the Republic of Serbia. P. D. acknowledges valuable feedback
and guidance from his physics teacher Cmiljka Vasović. All authors are
grateful to the International Young Physicist Tournament, its Serbian
community and participants for the inspiring atmosphere and scientific
discussions which lead to this work, particularly, authors are grateful to
Jelena Pajović. All authors thank Una Jaćimović, author of the Fig. 1.

## References

  * Tobjörk and Österbacka (2011) D. Tobjörk and R. Österbacka, Advanced materials 23, 1935 (2011).
  * Kurra, Dutta, and Kulkarni (2013) N. Kurra, D. Dutta, and G. U. Kulkarni, Physical Chemistry Chemical Physics 15, 8367 (2013).
  * Kurra and Kulkarni (2013) N. Kurra and G. U. Kulkarni, Lab on a Chip 13, 2866 (2013).
  * Costa  _et al._ (2018) J. C. Costa, A. Wishahi, A. Pouryazdan, M. Nock, and N. Münzenrieder, Advanced Electronic Materials 4, 1700600 (2018).
  * Liu  _et al._ (2017) S. Liu, J. Li, X. Shi, E. Gao, Z. Xu, H. Tang, K. Tong, Q. Pei, J. Liang, and Y. Chen, Advanced Electronic Materials 3, 1700098 (2017).
  * Xu  _et al._ (2020) Y. Xu, G. Zhao, L. Zhu, Q. Fei, Z. Zhang, Z. Chen, F. An, Y. Chen, Y. Ling, P. Guo, _et al._ , Proceedings of the National Academy of Sciences 117, 18292 (2020).
  * Inyeo and Wattanakasiwich (2018) C. Inyeo and P. Wattanakasiwich, Journal of Physics: Conference Series 1144, 012165 (2018).
  * Derman and Goykadosh (1999) S. Derman and A. Goykadosh, The Physics Teacher 37, 400 (1999).
  * Marshall (2003) R. Marshall, Physics Education 38, 94 (2003).
  * Note (1) In Serbian curriculum, the topic is already introduced in the eight (final) grade of the primary school (students’ age of 15) and then discussed more deeply in secondary school, usually in second or third grade depending on the school’s profile.
  * Kamata and Abe (2012) M. Kamata and M. Abe, Physics Education 47, 741 (2012).
  * Chiaverina (2014) C. Chiaverina, The Physics Teacher 52, 570 (2014).
  * Martchenko (2013) I. Martchenko, IYPT-Magazine 1, 20 (2013).
  * Haskell, Britton, and Edge (2000) H. Haskell, C. Britton, and R. Edge, The Physics Teacher 38, 22 (2000).
  * Plesch, Plesník, and Ružičková (2020) M. Plesch, S. Plesník, and N. Ružičková, European Journal of Physics 41, 034001 (2020).
  * IYP (2019) “Problems for the 33rd IYPT,” https://www.iypt.org/problems/iypt-2020-problems/ (2019), [Online; accessed 27-July-2021].
  * (17) “The IYPT Archive,” https://archive.iypt.org/, [Online; accessed 27-July-2021].
  * Planinsic (2009) G. Planinsic, European journal of physics 30, S133 (2009).
  * Etkina (2015) E. Etkina, American Journal of Physics 83, 669 (2015).
  * Küçüközer (2015) A. Küçüközer, The Physics Teacher 53, 38 (2015).
  * (21) “SMG Learning Activities: Graphite Circuits,” https://learning.sciencemuseumgroup.org.uk/resources/graphite-circuits/, [Online; accessed 27-July-2021].
  * (22) “Circuit scribe,” https://circuitscribe.com/, [Online; accessed 27-July-2021].
  * Ibrahim and Rebello (2012) B. Ibrahim and N. S. Rebello, Physical Review Special Topics-Physics Education Research 8, 010126 (2012).
  * Woolf and Streckert (1996) L. D. Woolf and H. H. Streckert, PHYSICS TEACHER 34, 440 (1996).
  * Sousa and Buchanan (2000) M. C. Sousa and J. W. Buchanan, in  _Computer Graphics Forum_ , Vol. 19 (Wiley Online Library, 2000) pp. 27–49.
  * Grisales, Herrera, and Fajardo (2016) C. Grisales, N. Herrera, and F. Fajardo, Physics Education 51, 055011 (2016).
  * Adorno, Bellomonte, and Pizzolato (2017) D. P. Adorno, L. Bellomonte, and N. Pizzolato, European Journal of Physics 39, 013001 (2017).
  * Van Dooren  _et al._ (2004) W. Van Dooren, D. De Bock, A. Hessels, D. Janssens, and L. Verschaffel, Learning and Instruction 14, 485 (2004).
  * Shmavonyan and Mailian (2015) G. S. Shmavonyan and A. Mailian, in  _2nd International Conference on Green Materials and Environmental Engineering_ (Atlantis Press, 2015).
  * Salami  _et al._ (2007) M. A. Salami, J. B. d. R. Filho, C. Galli, and R. d. C. da Costa, The Physics Teacher 45, 392 (2007).
  * Trumper (2002) R. Trumper, Journal of Science Education and Technology 11, 221 (2002).
  * Ku  _et al._ (1966) H. H. Ku _et al._ , Journal of Research of the National Bureau of Standards 70, 263 (1966).
  * Narumi  _et al._ (2015) K. Narumi, X. Shi, S. Hodges, Y. Kawahara, S. Shimizu, and T. Asami, in  _Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems_ (2015) pp. 2307–2312.

##  Appendix A Uncertainty analysis

In line with previous work Trumper (2002) that singles out an estimate of
uncertainty as one of the key features students should learn while doing
laboratory experiments in order to grasp the scientific method, here we
provide additional information on the error analysis for the calculated values
presented in the paper.

Within the paper, we focus most of the attention on the measurement errors and
denote those appropriately within the plots. However, as discussed in the
context of Fig. 3, some of the measurements provide learning opportunities to
discuss propagation of errors where students can practically grasp the fact
that an error of a summed variable is obtained by summing the errors

$\Delta R_{e}=\Delta R_{1}+\Delta R_{2}+\dots+\Delta R_{n}\,.$ (13)

The lecture on error propagation can be even further deepened in the Sec III.3
where students can assess errors not only for the directly measured
resistances, but also for the equivalent resistances calculated using Eqs.
(7), (10), and (12). Here we provide absolute error relations that we used to
calculate errors denoted in Fig. 7 (for more information on error propagation
see for example Ku  _et al._ (1966)).

$\displaystyle\Delta R_{e}^{(1)}=$ $\displaystyle 2\Delta R_{1}+\Delta
R_{2}\,,$ (14) $\displaystyle\Delta R_{e}^{(n)}=$ $\displaystyle 2\Delta
R_{1}+\left(\frac{R_{e}^{(n-1)}}{R_{2}+R_{e}^{(n-1)}}\right)^{2}\Delta R_{2}$
$\displaystyle+\left(\frac{R_{2}}{R_{2}+R_{e}^{(n-1)}}\right)^{2}\Delta
R_{e}^{(n-1)}\,,\>n\geq 2\,,$ (15) $\displaystyle\Delta R_{e}^{(\infty)}=$
$\displaystyle\left(1+\frac{R_{1}+R_{2}}{\sqrt{R_{1}^{2}+2R_{1}R_{2}}}\right)\Delta
R_{1}$ $\displaystyle+\frac{R_{1}}{\sqrt{R_{1}^{2}+2R_{1}R_{2}}}\Delta
R_{2}\,.$ (16)