# Local Law 144: A Critical Analysis of Regression Metrics

Giulio Filippi, Sara Zannone , Airlie Hilliard, Adriano Koshiyama
Holistic AI
London
UK
contact<EMAIL_ADDRESS>
###### Abstract

The use of automated decision tools in recruitment has received an increasing
amount of attention. In November 2021, the New York City Council passed a
legislation (Local Law 144) that mandates bias audits of Automated Employment
Decision Tools. From 15th April 2023, companies that use automated tools for
hiring or promoting employees are required to have these systems audited by an
independent entity. Auditors are asked to compute bias metrics that compare
outcomes for different groups, based on sex/gender and race/ethnicity
categories at a minimum. Local Law 144 proposes novel bias metrics for
regression tasks (scenarios where the automated system scores candidates with
a continuous range of values). A previous version of the legislation proposed
a bias metric that compared the mean scores of different groups. The new
revised bias metric compares the proportion of candidates in each group that
falls above the median. In this paper, we argue that both metrics fail to
capture distributional differences over the whole domain, and therefore cannot
reliably detect bias. We first introduce two metrics, as possible alternatives
to the legislation metrics. We then compare these metrics over a range of
theoretical examples, for which the legislation proposed metrics seem to
underestimate bias. Finally, we study real data and show that the legislation
metrics can similarly fail in a real-world recruitment application.

_K_ eywords AEDT $\cdot$ disparate impact $\cdot$ fair AI $\cdot$ AI law

##  1 Introduction

The fairness of Artificial Intelligence systems has received increased
attention in recent years. As more of these systems become ubiquitous in our
society, it becomes imperative to ensure that their use is fair towards all
members of our society. This is particularly true for sensitive applications
that are involved in critical decision-making processes. For example,
automated tools are used in healthcare to aid medical diagnosis [1], in
finance to predict credit scores [2], and in recruitment to evaluate
candidates and predict future job performance [3]. In these cases, the use of
automated tools has the potential of affecting the lives and life chances of
the individuals involved. For all of these reasons, there has been an
increasing demand for the regulation of AI systems and their applications. For
example, the European Commission proposed the Harmonised Rules on Artificial
Intelligence, colloquially known as the EU AI Act, which aims to create an
ecosystem of trust and regulate the AI tools that are available on the EU
market.

Taking a more targeted approach, the New York City Council passed Local Law
144 in November 2021, requiring bias audits of automated employment decision
tools (AEDTs), thereby narrowing their focus to automation used in talent
management. Under the law, Employers that use AEDTS to screen candidates for
employment or employees for promotion in the City must have these systems
audited by an independent entity beginning April 15, 2023 (postponed from
January 1, 2023). However, one of the key problems in formulating adequate
legislation, and in the AI fairness field in general, is to find effective
ways to quantify bias. Accordingly, the Department of Consumer and Worker
Protection (DCWP) has proposed bias metrics that require bias to be determined
using impact ratios that compare outcomes for subgroups based on sex/gender
and race/ethnicity categories at a minimum. Whereas the use of disparate
impact metrics for binary outputs is quite established in the field of pre-
employment testing ([4, 5, 6]), the legislation proposes novel rules to
measure bias for continuous outputs.

A first draft of the DCWP’s proposed rules required auditors to compare the
mean scores of different groups. In contrast, an updated version of the rules
proposes that auditors should measure and compare the proportion of candidates
in each subgroup that fall above the median for the sample. In this paper, we
will present a critical analysis of both metrics and argue that they fail to
capture distributional differences over the whole domain, and therefore cannot
reliably detect bias. We will show that these metrics underestimate bias over
a range of theoretical examples. We suggest two alternative metrics that could
be better at capturing the evolution over the whole distribution. Finally, we
assess and compare all the metrics on real-world data, obtained from a machine
learning system used in recruitment.

##  2 Background

Artificial intelligence is commonly used in talent management to screen
candidates and decide who should advance to the next stage of the recruitment
funnel. This type of task is called binary classification, because the aim is
to classify the data (candidates, in our example) into unsuccessful (0) and
successful (1) classes. Let $X$ denote the random variable associated with the
feature vector, and let $f$ be the binary prediction function satisfying
$f(X)\in\\{0,1\\}$ (which may or may not be random). Also suppose we have a
protected attribute $A$ with $K$ possible values encoded as $i=1,\ldots,K$. We
assume we have a distribution on $(f(X),A)$. We do not assume we are in
possession of a distribution involving the instance $X$ because we usually do
not have access to the instances or the model $f$, only its outputs.

In binary classification, the most common metric used to measure bias is known
as disparate impact [7], which examines whether hiring rates are different
across groups. To compute disparate impact, we first compute the selection
rate for each group $i$ as
$SR_{i}=\mathbb{P}(f(X)=1|A=i)=\mathbb{E}(f(X)|A=i)$. Note that in binary
classification, the selection rate can also be written as an expectation.

The Binary Disparate Impact ($DI$) is then computed as the ratio of selection
rates between groups, taking the largest as the denominator (so that the
computed metric is below 1).

$DI_{i}=\frac{\mathbb{E}(f(X)|A=i)}{\max_{j}\mathbb{E}(f(X)|A=j)}$ (1)

This metric has been extensively used and studied within the computational
fairness literature [8, 9, 10, 11]. In the case that the selection rates are
equal across groups, we can say our classifier satisfies Statistical Parity
(or Demographic Parity), which is represented by the following equation:

$SR_{i}=SR_{j}\ \forall\ i,j\in\\{1,\ldots,K\\}$

Statistical Parity is an Equality of Outcome fairness notion, meaning that it
seeks to equalize the outcomes of our model for different subgroups of the
population [12, 13, 14]. There are also Equality of Opportunity fairness
notions that seek to equalize the performance of our model for different
subgroups [15]. In this paper, we focus on Equality of Outcome, because we do
not assume we have the true labels that are necessary to compute performance
metrics.

However, not all applications of Artificial Intelligence in recruitment result
in binary outcomes; many systems use regression. In regression, the outputs
$f(X)$ of our model are no longer restricted to 0,1 – they can now take a
continuous range of values (e.g., 0 to 100). While it is not immediately clear
how the fairness notions from binary classification might generalise to the
regression setting, following the definition of [14], a regressor satisfies
Statistical Parity if the outcome $f(X)$ is independent of the protected
attribute:

$\mathbb{P}(f(X)\geq z|A=i)=\mathbb{P}(f(X)\geq z|A=j),\forall i,j\ \forall z$

While binary classification allowed us to replace the probability with an
expectation, this is not possible in the regression setting. Also note that
what was once a single condition now becomes a family of equations that have
to be satisfied, indexed by the threshold $z$. This means that the condition
for regression data is considerably stronger than the binary one, where
fairness is imposed at all possible thresholds. Originally, the metric
proposed in the legislation for regression data is taken in analogy to the
metric in the binary setting, where impact ratios compared the mean score for
different groups. In equations, it is computed as

$MeanDI_{i}=\frac{\mathbb{E}(f(X)|A=i)}{\max_{j}\mathbb{E}(f(X)|A=j)}$ (2)

This metric was then withdrawn in the updated rules published by the DCWP and
an alternative was offered. For a proportion $p$, we denote by $Q(p)$ the
corresponding quantile value (function satisfying $P(f(x)\leq Q(p))=p$). The
newly proposed metric uses the median value as threshold ($median=Q(0.5)$) to
compute a binary disparate impact.

$MedDI_{i}=\frac{P(f(X)\geq median|A=i)}{\max_{j}P(f(X)\geq median|A=j)}$ (3)

Note that this metric is equivalent to setting the success rate of the overall
sample to $50\%$, and then comparing the success rates of the different
subgroups.

##  3 Proposed Metrics

The metrics we propose in this section are based on reducing the regression
case to the binary case. There are different paradigms for thresholding
(binarizing) data that we cover in the Discussion section. Here, we will use
the paradigm based on quantile values. The reason for this choice is that it
reduces the sensitivity of the metric to outliers and makes it so that we can
work in the range $[0,1]$ regardless of the original data domain. We first
introduce useful notation for the binary disparate impact obtained at a given
proportion. For a proportion $p$, we denote by $Q(p)$ the corresponding
quantile value (function satisfying $P(f(X)\leq Q(p))=p$).

$BinDI_{i}(p)=\frac{P(f(X)\geq Q(p)|A=i)}{\max_{j}P(f(X)\geq Q(p)|A=j)}$ (4)

In this section, we introduce two new metrics: $AucDI$ (area under binary
disparate impact curve) and $PfDI$ (probability of fair binary disparate
impact). Suppose we start with a prior distribution $\pi(p),p\in[0,1]$ for the
proportion of unsuccessful candidates. If we know for certain which proportion
$p^{*}$ will be unsuccessful, we can use a Kronecker delta prior
$\pi(p)=\delta_{p^{*}}(p)$. If we have no prior knowledge on the proportion of
rejected candidates, we can use a flat (uniform) prior $\pi(p)=1,p\in[0,1]$.
We will only use flat priors throughout the paper for simplicity, but note
that using a good prior for the proportion $p$ can drastically improve the
quality of the metrics. The equations for these metrics are as follows:

$AucDI_{i}=\int_{p=0}^{1}BinDI_{i}(p)\pi(p)dp$

and

$PfDI_{i}=\int_{p=0}^{1}\mathbbm{1}[BinDI_{i}(p)\geq 0.8]\pi(p)dp$

In the above equation, $\mathbbm{1}[\textrm{condition}]$ denotes the indicator
function on the given condition (function that is 1 if condition holds, 0 if
not). The above metrics offer two different ways of aggregating the data
obtained from the binarization procedure into a metric for regression data.
Note that we used disparate impact based metrics, to match the conventions of
the US legislation. Nothing prevents us from aggregating other binary
classification bias metrics (e.g., statistical parity difference) in the same
ways. In the next sections, we compare the metrics proposed by the DCWP with
the metrics proposed in this paper and look for differences in their
properties.

##  4 Fairness Bounds

Local Law 144 does not explicitly indicate a threshold for fairness, neither
for binary classification nor regression metrics. It only mandates that the
metrics be computed for each protected attribute, as well as intersectional
subgroups. In recruitment, when it comes to binary classification, it is
common to refer to the four-fifth rule, which indicates that the impact ratio
should be larger than 4/5 (0.8) for the selection rates to be considered fair
(US Equal Employment Opportunity Commission, EEOC, 1979). While the bound
isn’t stated to be the same for regression disparate impact metrics, one can
extrapolate that we would use the same bound.

What about bounds for $AucDI$ and $PfDI$? A perfectly fair regressor
satisfying Statistical Parity will have both $AucDI$ and $PfDI$ of 1, whereas
the minimum value for these metrics is 0. Even though there is no perfect way
of choosing a fairness threshold, we will consider here a fairness threshold
of 0.8 for both metrics. In the case of $AucDI$, this means the average Binary
Disparate Impact over the whole range of possible proportions should be no
less than 0.8. In the case of $PfDI$, this means that by taking a random
proportion in $[0,1]$, the probability of the binary data produced being fair
is above 0.8. However, it is worth noting that alternative thresholds are
possible.

##  5 Experiments

###  5.1 Theoretical Examples

In this section, we give some examples to motivate the argument that $MeanDI$
and $MedDI$ are not reliable measurements of bias. For the ratio of averages
($MeanDI$), the reason lies in the way this metric uses the average as a
descriptor of a distribution. By reducing distributions to their average
value, we throw away a lot of valuable distributional information. For the
median-based thresholding ($MedDI$), the reason is that considering only one
threshold is not enough to get a full distributional picture. Distributions
are infinite dimensional objects, so they are difficult to describe with a
scalar value. Of course, all metrics involve reduction from a high information
object to a low information object (usually to a scalar). That is why metrics
must be carefully designed to measure a specific notion. Even then, a single
notion will usually have different metrics that measure it, each of which uses
a slightly different lens.

To get a better idea of the way bias changes across the whole distribution, we
use binary disparate impact plots. In a binary disparate impact plot, we
threshold the regression scores at different quantile values, and then plot
the binary disparate impact as a function of the various proportion quantiles.
Essentially, we can look at how the binary disparate impact varies when we
select all the candidates, $90\%$ of the candidates, $80\%$ of the candidates
and so forth until none of the candidates are selected. If the regression
scores for the two groups considered have exactly the same distribution, then
we get perfect fairness at every threshold (a flat disparate impact curve
constant at 1). In practice, however, the distributions will often differ, and
the binary disparate impact curve will not be fully flat. Usually, the curve
will vary in and out of the fairness range. The only properties that all
binary disparate impact curves share is that they have a similar behaviour at
the extremes, that is, they always contain the points $(0,1)$ and $(1,0)$
because at quantile 0 all candidates succeed and at quantile 1 none of the
candidates succeed.

Notably, the middle point of the binary disparate impact plot corresponds to
selecting half of the candidates and is thus equivalent to $MedDI$, the metric
proposed for Local Law 144. The binary disparate impact plot provides us with
an outlook over the whole distribution, rather than just one point. We will
see that in many cases, even if $MedDI$ yields a fair result, the binary
disparate impact curve might speak a different story. We will also argue that
our metrics, $AucDI$ and $PfDI$, are better suited to capture bias over the
whole distribution.

####  5.1.1 Example 1

As a first example, take the distributions of groups $a$ and $b$ to be normal
with mean $\mu=50$. These distributions only differ in their standard
deviations $\sigma_{a}$ and $\sigma_{b}$. We create a parameter range using an
extra parameter $\delta$ to regulate the standard deviation difference. We set
$\sigma_{a}=10-\delta$, and $\sigma_{b}=10+\delta$. The distribution of group
$a$ gets increasingly narrow around 50 as $\delta$ increases while the
distribution of group $b$ spreads out (Fig. 1(a)). Clearly, for $\delta=0$ the
distributions will be identical and thus we expect to find perfect fairness.
As $\delta$ increases, the distributions diverge and the fairness properties
change. For large values of $\delta$, we will have candidates from group $a$
score consistently around 50, while group $b$ scores will be more evenly
distributed over the whole range. Whether this is fair or not will strongly
depend on the choice of binarization threshold. If we choose a small
threshold, then more of the selected candidates will belong to group $a$, and
the binary disparate impact will detect bias against group $b$ (Fig. 1(b)). On
the other hand, higher thresholds result in bias against group $a$ (Fig.
1(b)). Since the two groups have the same median and mean, even for varying
parameter $\delta$, the value of the metrics $MeanDI$ and $MedDI$ will be $1$
for any $\delta$. $MeanDI$ compares the means of the two groups and so it
cannot capture the change in standard deviation, while $MedDI$ looks only at
one data point on the binary disparate impact curve. On the other hand,
$AucDI$ and $PfDI$ find increasing bias as $\delta$ increases (Fig. 1(c)).
This is because these metrics take into account the whole distribution of
scores, and thus are able to capture the dependency of the fairness properties
on the parameter $\delta$. Furthermore, $PfDI$ flags unfair outcomes for both
group $a$ and group $b$, for large values of $\delta$. This is because for
lower thresholds group $a$ is advantaged, whereas larger thresholds advantage
group $b$.

(a) Distributions

(b) Binary Disparate Impact Curves

(c) Metrics

Figure 1: Example 1. The first example consists of a $N(50,10-\delta)$
distribution for group $a$ and a $N(50,10+\delta)$ for group $b$, where we let
the parameter $\delta$ have range $\\{0,\ldots,9\\}$. (a) Distributions. We
show the distributions of group $a$ and group $b$ for parameter
$\delta\in\\{0,4,8\\}$. Observe how the distribution of group $a$ narrows
while that of group $b$ spreads with increasing $\delta$. (b) Binary Disparate
Impact Curves. We show the evolution of the binary disparate impact as we
change the proportion threshold from 0 to 1, for groups $a$ and $b$. (c)
Metrics. We plot the metrics $MeanDI,MedDI,AucDI,PfDI$ as a function of
parameter $\delta$, for groups $a$ and $b$.

####  5.1.2 Example 2

As a second example, we take one normal distribution for group $a$ centered at
50 with standard deviation $\sigma$, and a bimodal mixture of two normals
centered at 20 and 80 for group $b$ with standard deviation equal to 10 in
both cases (see Fig. 2(a)). This example is notable for three reasons.
Firstly, candidates from group $b$ score at either the lower or higher end of
the value range, while group $a$ scores consistently around the middle point.
This means that for a sufficiently small threshold, only candidates from group
$b$ will be rejected. Conversely, for a sufficiently large threshold, only
candidates from group $b$ will be selected. Fairness is therefore highly
dependent on the binarization threshold: lower thresholds result in bias
against group $b$, while higher thresholds result in bias against group $a$
(Fig. 2(b)). Secondly, fairness depends on the standard deviation parameter
$\sigma$. For smaller values of $\sigma$, the disparity between groups is more
pronounced. When $\sigma$ increases, the overlap between the two distributions
increases, resulting in a fairer outcome (Fig. 2(b)). Thirdly, we can see that
group $a$ and group $b$ have the same mean and median, independently of the
choice of the parameter $\sigma$. The regression metrics proposed by the
legislation, $MeanDI$ and $MedDI$, will therefore be consistently equal to 1
(Fig. 2(c)). $MeanDI$ fails at detecting any bias because the group mean
cannot capture the spread of the distribution. $MedDI$ is lacking in that it
only considers the middle point on the binary disparate impact curve, which
happens to be fair in this example. On the other hand, $AucDI$ and $PfDI$ are
able to capture the trend over the whole distribution. Under our settings,
$AucDI$ and $PfDI$ detect bias mostly against group $a$, and both metrics
yield fairer values as the standard deviation $\sigma$ increases.
Interestingly, for lower values of $\sigma$, $PfDI$ falls under the fairness
threshold for both group $a$ and group $b$. $PfDI$ is, therefore, able to
detect bias against both groups at the same time.

(a) Distributions

(b) Binary Disparate Impact Curves

(c) Metrics

Figure 2: Example 2. The second example consists of a $N(50,\sigma)$
distribution for group $a$ and a bimodal mixture of $N(20,10)$ and $N(80,10)$
for group $b$. We let the parameter $\sigma$ have range $\\{10,\ldots,1\\}$.
(a) Distributions. We show the distributions of group $a$ and group $b$ for
parameter $\sigma\in\\{1,4,8\\}$. Observe how the distribution of group $a$
narrows while that of group $b$ stays fixed with decreasing $\sigma$. (b)
Binary Disparate Impact Curves. We show the evolution of the binary disparate
impact as we change the proportion threshold from 0 to 1, for groups $a$ and
$b$. (c) Metrics. We plot the metrics $MeanDI,MedDI,AucDI,PfDI$ as a function
of parameter $\delta$, for groups $a$ and $b$.

####  5.1.3 Example 3

As a third example, we take bimodal distributions for both groups $a$ and $b$.
The bimodal distribution for group $a$ has modes at 30 and 70 with standard
deviations 10 and $\sigma$, the bimodal distribution for group $b$ has modes
at 30 and 80 with standard deviations 10 and $\sigma$ (Fig. 3(a)). In this
example, the lower half of each group have the same distribution. However, the
upper halves of the two groups are centered at different values (70 for group
$a$, 80 for group $b$). This means that candidates in the top half of the
distribution score quite differently depending on the group they belong to,
and this difference gets larger for lower values of $\sigma$. The overall data
is set up so as to have mean and median values close to 50 and all
distributional bias occurs after the value 50. Hence, the metrics $MeanDI$ and
$MedDI$, cannot detect bias and are close to 1 for any value of the parameter
$\sigma$. On the other hand, we begin to see distributional bias against group
$a$ as soon as our threshold exceeds 50 (Fig. 3(b)). This shows as a sharp
decrease in the binary disparate impact curve after proportion 0.5. We can see
that fairness is strictly related to the choice of binarization. Since $AucDI$
and $PfDI$ take into account the whole distributional range, they consistently
flag these pairs of distributions as biased against group $a$, for any
$\sigma$, with bias increasing as $\sigma$ decreases (Fig.3(c)).

(a) Distributions

(b) Binary Disparate Impact Curves

(c) Metrics

Figure 3: Example 3. The third example consists of a bimodal mixture of
$N(30,10)$ and $N(70,\sigma)$ for group $a$ and a bimodal mixture of
$N(30,10)$ and $N(80,\sigma)$ for group $b$. We let the parameter $\sigma$
have range $\\{10,\ldots,1\\}$. (a) Distributions. We show the distributions
of group $a$ and group $b$ for parameter $\sigma\in\\{1,4,8\\}$. (b) Binary
Disparate Impact Curves. We show the evolution of the binary disparate impact
as we change the proportion threshold from 0 to 1, for groups $a$ and $b$. (c)
Metrics. We plot the metrics $MeanDI,MedDI,AucDI,PfDI$ as a function of
parameter $\delta$, for groups $a$ and $b$.

###  5.2 Real Data

The previous examples served as a proof of principle; they showed that it is
important that regression metrics consider the trend over the whole output
distribution, rather than only one data point. We will next show how similar
issues can present in real-world data as well. We analysed data from a
recruitment company that used machine learning systems to score candidates on
a scale from 0 to 100. This was an interesting example for us, since the
company ended up binarizing these scores by thresholding at score 50 (in the
middle of the 0 to 100 range).

They trained 10 different models, which we named predictor0, predictor1, …,
predictor9. In addition to the scores, the company provided us with ethnicity
and gender information for most of the candidates (77460 after dropping NaN
values). The gender attribute contains the Male and Female subgroups, while
the ethnicity contains the Asian, Black, Hispanic/Latino, White and Two or
more ethnicities (Two+) subgroups. We name the disparate impact obtained by
binarizing at 50, $ThreshDI$, to respect our naming convention. We decided to
analyse this case in order to compare the bias metrics calculated on the
regression scores to the final actual binarization used.

Given that we have 10 predictors and 2 protected attributes, there are 20
settings in total. We calculated both the binary disparate impact obtained at
the threshold of 50 ($ThreshDI$) and that obtained at the median ($MedDI$). We
considered the metrics to detect bias if they fell under the 0.8 fairness
threshold. We scanned the settings looking for cases where $ThreshDI$ and
$MedDI$ disagree (one violates the four-fifths rule and the other does not)
for at least one of the relevant subgroups. Note that we are comparing a
binarization at the median and at score 50 (which is right in the middle of
the range). If the data distribution was symmetric and centered in the middle
range (like a Gaussian, for example), then these two metrics would yield the
same result. We, therefore, did not expect the metrics to differ
substantially. Nonetheless, we found 4 cases where bias was masked, with
$ThreshDI$ detecting bias and $MedDI$ not for 3 of these cases. Interestingly,
$MeanDI$ undervalued bias on all of these cases (Tables in Appendix A). One of
these examples, predictor3 with the ethnicity protected attribute, is
displayed in Fig. 4. As observed in the metrics Table 1, the subgroups Black
and Two+ are flagged as biased by some metrics. The Black group is flagged by
$MedDI$, $ThreshDI$, $AucDI$ and $PfDI$, while the Two+ group is flagged by
$ThreshDI$ and $PfDI$. We can see from the Disparate Impact curves (Fig. 4)
that binarizing at the median (red line) or threshold (green line) fall
respectively in and out of the fairness threshold for the subgroup Two+.

(a) Data distributions

(b) Binary Disparate Impact curves

Figure 4: Real Data Example. This example consists of real company regression data (predictor3), ranging from 0 to 100. We study this example with the ethnicity protected attribute. (a) Data Distribution. We show the (kde estimated) distributions of all groups (Asian, White, Black, Hispanic/Latino, Two+). (b) Binary Disparate Impact curves. We plot the binary disparate impact for each group as a function of the proportion threshold. | Asian | Black | Hispanic/Latino | Two+ | White
---|---|---|---|---|---
$MeanDI$ | 1.00 | 0.840863 | 0.908923 | 0.892794 | 0.918503
$MedDI$ | 1.00 | 0.718925 | 0.853346 | 0.851637 | 0.871630
$ThreshDI$ | 1.00 | 0.696958 | 0.832264 | 0.745903 | 0.846828
$AucDI$ | 0.99 | 0.744136 | 0.860472 | 0.828036 | 0.866973
$PfDI$ | 0.99 | 0.330000 | 0.830000 | 0.650000 | 0.830000
Table 1: Metrics Table. We display all computed metrics for each subgroup of
the ethnicity attribute, for the scores of predictor3. Metrics with values
below 0.8 (flag) and we denote that by marking their value in red.

We should note here that we are not claiming that our bias metrics can always
detect bias. Indeed, the concepts of bias and fairness are multi-faceted and
complex. The fairness of a regression model will depend on how the score is
taken into account in the final decision. For example, if we know that a
specific threshold will be used, like in the data analysed, it would be
preferable to treat the problem as a binary classification task. However, in
cases where there is no final thresholding or the threshold is undefined, we
argue that it is indeed preferable to use metrics that take into account the
whole distribution. The metrics we proposed are just instances of many
possible metrics, but we suggest that they are better indicators of the
fairness of the data as a whole (as opposed to the legislation metrics). We
list the observations obtained from our experiments below:

  * •

The DCWP’s original regression disparate impact $MeanDI$ based on the ratio of
means seems to consistently under-detect bias.

  * •

$MedDI$ is not sufficient by itself to get a picture of the fairness
properties of the data. We find real-world cases where bias is revealed only
at certain thresholds.

  * •

$PfDI$ seems to flag more often than other metrics and take on more extreme
values, both for fair and unfair ranges.

  * •

$AucDI$ seems to be able to detect distributional bias. Adding an appropriate
prior will make the metric even more meaningful.

  * •

It would be interesting to study the behaviour of both $PfDI$ and $AucDI$
further and deduce the most meaningful choice of threshold for each one.

  * •

In our opinion, computing multiple metrics is the best option to get a fuller
picture of the fairness properties of the data.

##  6 Discussion

In this paper, we have argued that the proposed metrics for determining
whether regression systems are biased under NYC Local Law 144 are not reliable
in detecting bias. Since $MeanDI$ uses the mean as a descriptor of each
distribution, this metric will usually disguise the intricacies of
distributional information, and consistently underestimate bias. Indeed, we
have shown that this metric almost never triggers, even in cases that are
clearly biased. The $MedDI$ metric offers an improvement upon the original
proposal and is equivalent to fixing the success rate of the whole population
to $50\%$ and then comparing the success rates of each group. Yet, we argue
that looking at one threshold ($50\%$) is not enough to capture distributional
bias. For instance, bias may only reveal itself at higher thresholds. To solve
this issue, we propose two alternative metrics that function by aggregating
bias information at every possible threshold: $AucDI$ and $PfDI$. $AucDI$
functions by aggregating the information over all proportion thresholds with
an area under the curve approach, while $PfDI$ can be thought of as a
probability of obtaining fair binary data ($DI\geq 0.8$) for a randomly chosen
threshold. As mentioned above, we include the option to add a prior in both
proposed metrics, encoding any information we have about the proportion of
rejected/accepted candidates. It would also be interesting to explore
different priors and fairness bounds to see how it would change the results
for these metrics.

It is important to note that we chose to look at quantiled domains in both of
these metrics (domain that encodes proportion of rejected applicants). The
reason behind the choice is that a quantiled range makes the metrics more
robust to outliers. We discuss two alternatives to this choice. We could, for
example, binarize based on score, similar to the $ThreshDI$ metric from the
real data example. In this case, we would choose some threshold $z$ from the
original data domain, and consider all scores above $z$ successful and all
scores below $z$ unsuccessful. Otherwise, we could binarize by selecting the
top $K$ candidates and set everyone else’s labels to 0. Both of these
alternative paradigms would lead to differences in the way we define and
compute the metrics. Which paradigm is most fit to describe a situation will
crucially depend on context, and there is no a-priori way to decide which to
use.

Furthermore, all of the metrics discussed thus far are based on thresholding
the data. However, there are two other types of metrics that can be suited for
regression data and are worth discussing. The first is ranking-based bias
metrics. These are useful in the setting where the scores are used to create a
ranking of the candidates. The paradigm in ranking is that the utility
(benefit) will go down as a function of rank, so that ranking first is better
than second and so forth. We can think of candidates coming up on a webpage in
order. In this case, being candidate number 100 means the chances of being
selected are almost null. A number of such metrics have been introduced and
discussed in the literature [16, 17, 18, 19]. Finally, we would like to
mention that another approach to assessing the fairness of regression data can
be found in statistical testing. For example, the Kolmogorov-Smirnov (KS) test
[20] can be used to compare $F_{a}$ and $F_{b}$, the empirical cumulative
distributions for the two groups, by computing the supremum of their
difference $D=\sup_{x}|F_{a}(x)-F_{b}(x)|$. If this difference is large enough
for the null hypothesis to be rejected, the model can be assumed to be unfair.
If the difference is not large enough, the test is inconclusive. In practice,
this test will require a large enough sample size to achieve appropriate
confidence levels, but it is a good custom to use a statistical test, even
after a dataset has been flagged as biased by a bias metric [21].

## References

  * [1] Scott Mayer McKinney, Marcin Sieniek, Varun Godbole, Jonathan Godwin, Natasha Antropova, Hutan Ashrafian, Trevor Back, Mary Chesus, Greg C. Corrado, Ara Darzi, Mozziyar Etemadi, Florencia Garcia-Vicente, Fiona J. Gilbert, Mark Halling-Brown, Demis Hassabis, Sunny Jansen, Alan Karthikesalingam, Christopher J. Kelly, Dominic King, Joseph R. Ledsam, David Melnick, Hormuz Mostofi, Lily Peng, Joshua Jay Reicher, Bernardino Romera-Paredes, Richard Sidebottom, Mustafa Suleyman, Daniel Tse, Kenneth C. Young, Jeffrey De Fauw, and Shravya Shetty.  International evaluation of an ai system for breast cancer screening.  Nature 2020 577:7788, 577:89–94, 1 2020.
  * [2] Chih Fong Tsai and Jhen Wei Wu.  Using neural network ensembles for bankruptcy prediction and credit scoring.  Expert Systems with Applications, 34:2639–2649, 5 2008.
  * [3] Jeffrey Dastin.  Amazon scraps secret ai recruiting tool that showed bias against women.  In Ethics of data and analytics, pages 296–299. Auerbach Publications, 2018.
  * [4] Robert E Ployhart and Brian C Holtz.  The diversity–validity dilemma: Strategies for reducing racioethnic and sex subgroup differences and adverse impact in selection.  Personnel Psychology, 61(1):153–172, 2008.
  * [5] Leatta M Hough, Frederick L Oswald, and Robert E Ployhart.  Determinants, detection and amelioration of adverse impact in personnel selection procedures: Issues, evidence and lessons learned.  International Journal of Selection and Assessment, 9(1-2):152–194, 2001.
  * [6] Frederick L Oswald, Eric M Dunleavy, and Amy Shaw.  Measuring practical significance in adverse impact analysis.  In Adverse Impact Analysis, pages 112–132. Routledge, 2016.
  * [7] Michael Feldman, Sorelle A. Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasubramanian.  Certifying and removing disparate impact.  Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2015-August:259–268, 12 2014.
  * [8] Solon Barocas, Moritz Hardt, and Arvind Narayanan.  Fairness in machine learning.  Nips tutorial, 1:2017, 2017.
  * [9] Alexandra Chouldechova.  Fair prediction with disparate impact: A study of bias in recidivism prediction instruments.  Big data, 5(2):153–163, 2017.
  * [10] Michael Feldman, Sorelle A Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasubramanian.  Certifying and removing disparate impact.  In proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, pages 259–268, 2015.
  * [11] Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rodriguez, and Krishna P Gummadi.  Fairness beyond disparate treatment & disparate impact: Learning classification without disparate mistreatment.  In Proceedings of the 26th international conference on world wide web, pages 1171–1180, 2017.
  * [12] Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel.  Fairness through awareness.  In Proceedings of the 3rd innovations in theoretical computer science conference, pages 214–226, 2012.
  * [13] Alekh Agarwal, Alina Beygelzimer, Miroslav Dudík, John Langford, and Hanna Wallach.  A reductions approach to fair classification.  In International Conference on Machine Learning, pages 60–69. PMLR, 2018.
  * [14] Alekh Agarwal, Miroslav Dudík, and Zhiwei Steven Wu.  Fair regression: Quantitative definitions and reduction-based algorithms.  36th International Conference on Machine Learning, ICML 2019, 2019-June:166–183, 5 2019.
  * [15] M Hardt, E Price, N Srebro, et al.  Equality of opportunity in supervised learning, in ‘advances in neural information processing systems’.  2016\.
  * [16] Tim Draws, Nava Tintarev, Ujwal Gadiraju, Alessandro Bozzon, and Benjamin Timmermans.  Assessing viewpoint diversity in search results using ranking fairness metrics.  09 2020.
  * [17] Sahin Cem Geyik, Stuart Ambler, and Krishnaram Kenthapadi.  Fairness-aware ranking in search & recommendation systems with application to linkedin talent search.  Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 2221–2231, 7 2019.
  * [18] Alex Beutel, Jilin Chen, Tulsee Doshi, Hai Qian, Li Wei, Yi Wu, Lukasz Heldt, Zhe Zhao, Lichan Hong, Ed H Chi, et al.  Fairness in recommendation ranking through pairwise comparisons.  In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pages 2212–2220, 2019.
  * [19] Amifa Raj, Connor Wood, Ananda Montoly, and Michael D Ekstrand.  Comparing fair ranking metrics.  arXiv preprint arXiv:2009.01311, 2020.
  * [20] Andrey Kolmogorov.  Sulla determinazione empirica di una lgge di distribuzione.  Inst. Ital. Attuari, Giorn., 4:83–91, 1933.
  * [21] Bahar Taskesen, Jose Blanchet, Daniel Kuhn, and Viet Anh Nguyen.  A statistical test for probabilistic fairness.  In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 648–665, 2021.

##  Appendix A Real Data Experiments (further Plots and Metrics Tables)

Figure 5: Binary disparate impact curves. We show the binary disparate impact curves for all 4 examples where $MedDI$ and $ThreshDI$ disagreed (one flags and other doesn’t) for at least one group. | Asian | Black | Hispanic/Latino | Two+ | White
---|---|---|---|---|---
$MeanDI$ | 0.980857 | 0.861800 | 1.00000 | 0.951457 | 0.966043
$MedDI$ | 0.938114 | 0.800478 | 1.00000 | 0.901684 | 0.925310
$ThreshDI$ | 0.938070 | 0.786071 | 1.00000 | 0.910472 | 0.922407
$AucDI$ | 0.975231 | 0.760605 | 0.97396 | 0.886844 | 0.943328
$PfDI$ | 1.000000 | 0.500000 | 0.99000 | 0.880000 | 0.970000
Table 2: Metrics Table (predictor 1 / ethnicity) | Asian | Black | Hispanic/Latino | Two+ | White
---|---|---|---|---|---
$MeanDI$ | 0.931422 | 1.000000 | 0.977229 | 0.969312 | 0.962149
$MedDI$ | 0.798804 | 1.000000 | 0.906356 | 0.912210 | 0.883028
$ThreshDI$ | 0.911784 | 1.000000 | 0.988763 | 0.984916 | 0.967474
$AucDI$ | 0.828851 | 0.999598 | 0.924852 | 0.905651 | 0.878264
$PfDI$ | 0.500000 | 1.000000 | 0.960000 | 0.960000 | 0.790000
Table 3: Metrics Table (predictor 2 / ethnicity) | Asian | Black | Hispanic/Latino | Two+ | White
---|---|---|---|---|---
$MeanDI$ | 1.00 | 0.840863 | 0.908923 | 0.892794 | 0.918503
$MedDI$ | 1.00 | 0.718925 | 0.853346 | 0.851637 | 0.871630
$ThreshDI$ | 1.00 | 0.696958 | 0.832264 | 0.745903 | 0.846828
$AucDI$ | 0.99 | 0.744136 | 0.860472 | 0.828036 | 0.866973
$PfDI$ | 0.99 | 0.330000 | 0.830000 | 0.650000 | 0.830000
Table 4: Metrics Table (predictor 3 / ethnicity) | Asian | Black | Hispanic/Latino | Two+ | White
---|---|---|---|---|---
$MeanDI$ | 0.949931 | 0.884560 | 0.928716 | 0.902873 | 1.000000
$MedDI$ | 0.939701 | 0.843386 | 0.908458 | 0.878517 | 1.000000
$ThreshDI$ | 0.904694 | 0.825959 | 0.882696 | 0.775178 | 1.000000
$AucDI$ | 0.916716 | 0.816245 | 0.869071 | 0.838914 | 0.990665
$PfDI$ | 0.970000 | 0.720000 | 0.820000 | 0.660000 | 0.990000
Table 5: Metrics Table (predictor 5 / ethnicity)


# Distilling Causal Effect from Miscellaneous Other-Class
for Continual Named Entity Recognition

Junhao Zheng, Zhanxian Liang, Haibin Chen, Qianli Ma*
School of Computer Science and Engineering,
South China University of Technology, Guangzhou, China
<EMAIL_ADDRESS><EMAIL_ADDRESS>*Corresponding author

###### Abstract

Continual Learning for Named Entity Recognition (CL-NER) aims to learn a
growing number of entity types over time from a stream of data. However,
simply learning Other-Class in the same way as new entity types amplifies the
catastrophic forgetting and leads to a substantial performance drop. The main
cause behind this is that Other-Class samples usually contain old entity
types, and the old knowledge in these Other-Class samples is not preserved
properly. Thanks to the causal inference, we identify that the forgetting is
caused by the missing causal effect from the old data. To this end, we propose
a unified causal framework to retrieve the causality from both new entity
types and Other-Class. Furthermore, we apply curriculum learning to mitigate
the impact of label noise and introduce a self-adaptive weight for balancing
the causal effects between new entity types and Other-Class. Experimental
results on three benchmark datasets show that our method outperforms the
state-of-the-art method by a large margin. Moreover, our method can be
combined with the existing state-of-the-art methods to improve the performance
in CL-NER. 111Our codes are publicly available at
https://github.com/zzz47zzz/CFNER

##  1 Introduction

Named Entity Recognition (NER) is a vital task in various NLP applications (Ma
and Hovy, 2016). Traditional NER aims at extracting entities from unstructured
text and classifying them into a fixed set of entity types (e.g., Person,
Location, Organization, etc). However, in many real-world scenarios, the
training data are streamed, and the NER systems are required to recognize new
entity types to support new functionalities, which can be formulated into the
paradigm of continual learning (CL, a.k.a. incremental learning or lifelong
learning) (Thrun, 1998; Parisi et al., 2019). For instance, voice assistants
such as Siri or Alexa are often required to extract new entity types (e.g.
Song, Band) for grasping new intents (e.g. GetMusic) (Monaikul et al., 2021).

However, as is well known, continual learning faces a serious challenge called
catastrophic forgetting in learning new knowledge (McCloskey and Cohen, 1989;
Robins, 1995; Goodfellow et al., 2013; Kirkpatrick et al., 2017). More
specifically, simply fine-tuning a NER system on new data usually leads to a
substantial performance drop on previous data. In contrast, a child can
naturally learn new concepts (e.g., Song and Band) without forgetting the
learned concepts (e.g., Person and Location). Therefore, continual learning
for NER (CL-NER) is a ubiquitous issue and a big challenge in achieving human-
level intelligence.

Figure 1: An illustration of Other-class in CL-NER. Suppose that a model
learns four entity types in CoNLL2003 sequentially. “LOC”: Location; “MISC”:
Miscellaneous; “ORG”: Organisation; “PER”: Person.

In the standard setting of continual learning, only new entity types are
recognized by the model in each CL step. For CL-NER, the new dataset contains
not only new entity types but also Other-class tokens which do not belong to
any new entity types. For instance, about 89% tokens belongs to Other-class in
OntoNotes5 (Hovy et al., 2006). Unlike accuracy-oriented tasks such as the
image/text classification, NER inevitably introduces a vast number of Other-
class samples in training data. As a result, the model strongly biases towards
Other-class (Li et al., 2020). Even worse, the meaning of Other-class varies
along with the continual learning process. For example, “Europe” is tagged as
Location if and only if the entity type Location is learned in the current CL
step. Otherwise, the token “Europe” will be tagged as Other-class. An
illustration is given in Figure 1 to demonstrate Other-class in CL-NER. In a
nutshell, the continually changing meaning of Other-class as well as the
imbalance between the entity and Other-class tokens amplify the forgetting
problem in CL-NER.

Figure 2: An illustration of the impact of Other-class samples on OntoNotes5.
We consider two scenarios with different extra annotation levels on Other-
class samples: (1) annotate all recognized entity types on the data in the
current CL step (Current); (2) no extra annotations on Other-class samples
(None).

Figure 2 is an illustration of the impact of Other-class samples. We divide
the training set into 18 disjoint splits, and each split corresponds to one
entity type to learn. Then, we only retain the labels of the corresponding
entity type in each split while the other tokens are tagged as Other-class.
Next, the NER model learns 18 entity types one after another, as in CL. To
eliminate the impact of forgetting, we assume that all recognized training
data can be stored. Figure 2 shows two scenarios where Other-class samples are
additionally annotated with ground-truth labels or not. Results show that
ignoring the different meanings of Other-classes affects the performance
dramatically. The main cause is that Other-class contains old entities. From
another perspective, the old entities in Other-class are similar to the
reserved samples of old classes in the data replay strategy (Rebuffi et al.,
2017). Therefore, we raise a question: how can we learn from Other-class
samples for anti-forgetting in CL-NER?

In this study, we address this question with a Causal Framework for CL-NER
(CFNER) based on causal inference (Glymour et al., 2016; Schölkopf, 2022).
Through causal lenses, we determine that the crux of CL-NER lies in
establishing causal links from the old data to new entity types and Other-
class. To achieve this, we utilize the old model (i.e., the NER model trained
on old entity types) to recognize old entities in Other-class samples and
distillate causal effects (Glymour et al., 2016) from both new entity types
and Other-class simultaneously. In this way, the causality of Other-class can
be learned to preserve old knowledge, while the different meanings of Other-
classes can be captured dynamically. In addition, we design a curriculum
learning (Bengio et al., 2009) strategy to enhance the causal effect from
Other-class by mitigating the label noise generated by the old model.
Moreover, we introduce a self-adaptive weight to dynamically balance the
causal effects from Other-class and new entity types. Extensive experiments on
three benchmark NER datasets, i.e., OntoNotes5, i2b2 (Murphy et al., 2010) and
CoNLL2003 (Sang and De Meulder, 2003), validate the effectiveness of the
proposed method. The experimental results show that our method outperforms the
previous state-of-the-art method in CL-NER significantly. The main
contributions are summarized as follows:

  * •

We frame CL-NER into a causal graph (Pearl, 2009) and propose a unified causal
framework to retrieve the causalities from both Other-class and new entity
types.

  * •

We are the first to distillate causal effects from Other-class for anti-
forgetting in CL, and we propose a curriculum learning strategy and a self-
adaptive weight to enhance the causal effect in Other-class.

  * •

Through extensive experiments, we show that our method achieves the state-of-
the-art performance in CL-NER and can be implemented as a plug-and-play module
to further improve the performances of other CL methods.

##  2 Related Work

###  2.1 Continual Learning for NER

Despite the fast development of CL in computer vision, most of these methods
(Douillard et al., 2020; Rebuffi et al., 2017; Hou et al., 2019) are devised
for accuracy-oriented tasks such as image classification and fail to preserve
the old knowledge in Other-class samples. In our experiment, we find that
simply applying these methods to CL-NER does not lead to satisfactory
performances.

In CL-NER, a straightforward solution for learning old knowledge from Other-
class samples is self-training (Rosenberg et al., 2005; De Lange et al.,
2019). In each CL step, the old model is used to annotate the Other-class
samples in the new dataset. Next, a new NER model is trained to recognize both
old and new entity types in the dataset. The main disadvantage of self-
training is that the errors caused by wrong predictions of the old model are
propagated to the new model (Monaikul et al., 2021). Monaikul et al. (2021)
proposed a method based on knowledge distillation (Hinton et al., 2015) called
ExtendNER where the old model acts as a teacher and the new model acts as a
student. Compared with self-training, this distillation-based method takes the
uncertainty of the old model’s predictions into consideration and reaches the
state-of-the-art performance in CL-NER.

Recently, Das et al. (2022) alleviates the problem of Other-tokens in few-shot
NER by contrastive learning and pretraining techniques. Unlike them, our
method explicitly alleviates the problem brought by Other-Class tokens through
a causal framework in CL-NER.

(a) forgetting

(b) anti-forgetting

Figure 3: The causal graph for CL-NER: (a) forgetting happens when there are
no causal paths from old data to new predictions; (b) anti-forgetting is to
build causal paths from old data to new predictions through new entities
($D^{E}$) and Other-class samples ($D^{O}$). We call the causal effects in
these two links $\textit{Effect}_{E}$ and $\textit{Effect}_{O}$, respectively.

###  2.2 Causal Inference

Causal inference (Glymour et al., 2016; Schölkopf, 2022) has been recently
introduced to various computer vision and NLP tasks, such as semantic
segmentation (Zhang et al., 2020), long-tailed classification (Tang et al.,
2020; Nan et al., 2021), distantly supervised NER (Zhang et al., 2021) and
neural dialogue generation (Zhu et al., 2020). Hu et al. (2021) first applied
causal inference in CL and pointed out that the vanishing old data effect
leads to forgetting. Inspired by the causal view in (Hu et al., 2021), we
mitigate the forgetting problem in CL-NER by mining the old knowledge in
Other-class samples.

##  3 Causal Views on (Anti-) Forgetting

In this section, we explain the (anti-) forgetting in CL from a causal
perspective. First, we model the causalities among data, feature, and
prediction at any consecutive CL step with a causal graph (Pearl, 2009) to
identify the forgetting problem. The causal graph is a directed acyclic graph
whose nodes are variables, and directed edges are causalities between nodes.
Next, we introduce how causal effects are utilized for anti-forgetting.

###  3.1 Causal Graph

Figure 3a shows the causal graph of CL-NER when no anti-forgetting techniques
are used. Specifically, we denote the old data as $S$; the new data as $D$;
the feature of new data extracted from the old and new model as $X_{0}$ and
$X$; the prediction of new data as $\hat{Y}$ (i.e., the probability
distribution (scores)). The causality between notes is as follows: (1)
$D\rightarrow X\rightarrow\hat{Y}$: $D\rightarrow X$ represents that the
feature $X$ is extracted by the backbone model ($e.g.,$ BERT (Devlin et al.,
2019)), and $X\rightarrow\hat{Y}$ indicates that the prediction $\hat{Y}$ is
obtained by using the feature $X$ with the classifier ($e.g.,$ a fully-
connected layer); (2) $S\rightarrow X_{0}\leftarrow D$: these links represent
that the old feature representation of new data $X_{0}$ is determined by the
new data $D$ and the old model trained on old data $S$. Figure 3a shows that
the forgetting happens because there are no causal links between $S$ and
$\hat{Y}$. More explanations about the forgetting in CL-NER are demonstrated
in Appendix A.

###  3.2 Colliding Effects

In order to build cause paths from $S$ to $\hat{Y}$, a naive solution is to
store (a fraction of) old data, resulting in a causal link $S\rightarrow D$ is
built. However, storing old data contradicts the scenario of CL to some
extent. To deal with this dilemma, Hu et al. (2021) proposed to add a causal
path $S\leftrightarrow D$ between old and new data by using Colliding Effect
(Glymour et al., 2016). Consequently, $S$ and $D$ will be correlated to each
other when we control the collider $X_{0}$. Here is an intuitive example: a
causal graph $sprinkler\rightarrow pavement\leftarrow weather$ represents the
pavement’s condition (wet/dry) is determined by both the weather (rainy/sunny)
and the sprinkler (on/off). Typically, the weather and the sprinkler are
independent of each other. However, if we observe that the pavement is wet and
know that the sprinkler is off, we can infer that the weather is likely to be
rainy, and vice versa.

Figure 4: A demonstration of the colliding effect. The anchor token’s feature
(anchor features) collides with matched tokens’ features (matched features) on
the colliding feature in the old feature space. (a) initial: the class
boundary is retained since the new model is initialized by the old model
condition. (b) w/o Colliding Effect: the class boundary is forgot in the new
feature space since there are no causal effects from old data to new
predictions. (c) w/ Colliding Effect: the class boundary is preserved after an
CL step since the anchor and matched tokens collide in the old feature space.

##  4 A Causal Framework for CL-NER

In this section, we frame CL-NER into a causal graph and identify that
learning the causality in Other-class is crucial for CL-NER. Based on the
characteristic of CL-NER, we propose a unified causal framework to retrieve
the causalities from both Other-class and new entity types. We are the first
to distillate causal effects from Other-class for anti-forgetting in CL.
Furthermore, we introduce a curriculum-learning-based strategy and a self-
adaptive weight to allow the model to better learn the causalities from Other-
class samples.

###  4.1 Problem Formulation

In the $i$-th CL step, given an NER model $M_{i}$ which is trained on a set of
entities $E_{i}=\\{e_{1},e_{2}\cdots,e_{n_{i}}\\}$, the target of CL-NER is to
learn the best NER model $M_{i+1}$ to identify the extended entity set
$E_{i+1}=\\{e_{1},e_{2}\cdots,e_{n_{i+1}}\\}$ with a training set annotated
only with $E_{i}^{New}=\\{e_{n_{i}+1},e_{n_{i}+2},\cdots,e_{n_{i+1}}\\}$.

Suppose the model consists of a backbone network for feature extraction and a
classifier for classification. As a common practice, $M_{i+1}$ is first
initialized by the parameter of $M_{i}$, and then the dimensions of the
classifier are extended to adapt to new entity types. Then, $M_{i}$ will guide
the learning process of $M_{i+1}$ through knowledge distillation (Monaikul et
al., 2021) or regularization terms (Douillard et al., 2020) to preserve old
knowledge. Our method is based on knowledge distillation where the old model
$M_{i}$ acts as a teacher and the new model $M_{i+1}$ acts as a student. Our
method further distillates causal effects in the process of knowledge
distillation.

Figure 5: A demonstration of the proposed causal framework for CL-NER.

###  4.2 Distilling Colliding Effects in CL-NER

Based on the causal graph in Figure 3a, we figure out that the crux of CL lies
in building causal paths between old data and prediction on the new model. If
we utilize colliding effects, the causal path between old and new data can be
built without storing old data.

To distillate the colliding effects, we first need to find tokens in new data
which have the same feature representation $X_{0}$ in the old feature space,
i.e., condition on $X_{0}$. However, it is almost impossible to find such
matched tokens since features are sparse in high dimensional space (Altman and
Krzywinski, 2018). Following Hu et al. (2021), we approximate the colliding
effect using K-Nearest Neighbor (KNN) strategy. Specifically, we select a
token as anchor token and search the k-nearest neighbor tokens whose features
bear a resemblance to the anchor token’s feature in the old feature space.
Next, when calculating the prediction of the anchor token, we use matched
tokens for joint prediction. Note that in backpropagation, only the gradient
of the anchor token is computed. Figure 4 shows a demonstration for distilling
colliding effects.

Although Other-class tokens usually do not directly guide the model to
recognize new entity types, they contain tokens from old entity types, which
allow models to recall what they have learned. Naturally, we use the old model
to recognize the Other-class tokens which actually belong to old entity types.
Since these Other-class tokens belong to the predefined entity types, we call
them as Defined-Other-Class tokens, and we call the rest tokens in Other-class
as Undefined-Other-Class tokens.

Based on the characteristics of NER, we extend the causal graph in Figure 3a
to Figure 3b. The key adjustment is that the node of new data is split into
two nodes, including new entity tokens $D^{E}$ and Defined-Other-Class tokens
$D^{O}$. Then, we apply the colliding effects on $D^{E}$ and $D^{O}$
respectively, resulting in that $D^{E}$ and $D^{O}$ collide with $S$ on nodes
$X_{0}^{E}$ and $X_{0}^{O}$ in the old feature space. In this way, we build
two causal paths from old data $S$ to new predictions $\hat{Y}$. In the causal
graph, we ignore the token from Undefined-Other-Class since they do not help
models learn new entity types or review old knowledge. Moreover, we expect the
model to update instead of preserving the knowledge about Other-class in each
CL step. Here, we consider two paths separately because the colliding effects
are distilled from different kinds of data and calculated in different ways.

###  4.3 A causal framework for CL-NER

Formally, we define the total causal effects Effect as follow:

Effect $\displaystyle=\textit{Effect}_{E}+\textit{Effect}_{O}$ (1)
$\displaystyle=-\Sigma_{i}\textit{CE}(\overline{Y}_{i},Y_{i})-\Sigma_{j}\textit{KL}(\overline{Y}_{j},\widetilde{Y}_{j}),$
(2) $\displaystyle\textit{s.t.}\quad D_{i}\in D^{E},\quad D_{j}\in D^{O}.$

In Eq.(1), $\textit{Effect}_{E}$ and $\textit{Effect}_{O}$ denote the
colliding effect of new entity types and Defined-Other-Class. In Eq.(2),
$\textit{CE}(\cdot,\cdot)$ and $\textit{KL}(\cdot,\cdot)$ represent the cross-
entropy and KL divergence loss, and $D_{i}$,$D_{j}$ are the $i$-th,$j$-th
token in new data. In cross-entropy loss, $Y_{i}$ is the ground-truth entity
type of the $i$-th token in new data. In KL divergence loss,
$\widetilde{Y}_{j}$ is the soft label of the $j$-th token given by the old
model over old entity types. In both losses, $\overline{Y}$ represents the
weighted average of prediction scores over anchor and matched tokens. When
calculating $\textit{KL}(\cdot,\cdot)$, we follow the common practice in
knowledge distillation to introduce temperature parameters $T_{t}$, $T_{s}$
for teacher (old model) and student model (new model) respectively. Here, we
omit $T_{t}$, $T_{s}$ in $\textit{KL}(\cdot,\cdot)$ for notation simplicity.
The weighted average scores of the $i$-th token is calculated as follow:

$\displaystyle\overline{Y}_{i}$
$\displaystyle=W_{i}\hat{Y}_{i}+\Sigma_{k=1}^{K}W_{ik}\hat{Y}_{ik}$ (3) s.t.
$\displaystyle\quad W_{i}\geq W_{i1}\geq W_{i2}\geq\cdots\geq W_{iK}$
$\displaystyle\quad W_{i}+\Sigma_{k=1}^{K}W_{ik}=1,$

where the $i$-th token is the anchor token and $K$ matched tokens are selected
according to the KNN strategy. We sort the $K$ matched tokens in ascending
order according to the distance to the anchor token in the old feature space.
$\hat{Y}_{i}$, $\hat{Y}_{ik}$ are the prediction scores of the $i$-th token
and its $k$-th matched token, and $W_{i}$, $W_{ik}$ are the weight for
$\hat{Y}_{i}$, $\hat{Y}_{ik}$, respectively. The weight constraints ensure
that the token closer to the colliding feature has a more significant effect.

Until now, we calculate the effects in $D^{E}$ and $D^{O}$ and ignore the
Undefined-Other-Class. Following Monaikul et al. (2021), we apply the standard
knowledge distillation to allow new models to learn from old models. To this
end, we just need to re-write $\textit{Effect}_{O}$ in Eq.(1) as follow:

$\displaystyle\textit{Effect}_{O}$
$\displaystyle=-\Sigma_{j}\textit{KL}(\overline{Y}_{j},\widetilde{Y}_{j})-\Sigma_{n}\textit{KL}(\hat{Y}_{n},\widetilde{Y}_{n})$
(4) s.t. $\displaystyle\quad D_{j}\in D^{O},\quad D_{n}\in D^{UO},$

where $D^{UO}$ is the data belong to the Undefined-Other-Class. The Eq.(4) can
be seen as calculating samples from $D^{O}$ and $D^{UO}$ in the same way,
except that samples from $D^{UO}$ have no matched tokens. We summarize the
proposed causal framework in Figure 5.

###  4.4 Mitigating Label Noise in $\textit{Effect}_{O}$

In our method, we use the old model to predict the labels of Defined-Other-
Class tokens. However, it inevitably generates label noise when calculating
$\textit{Effect}_{O}$. To address this problem, we adopt curriculum learning
to mitigate the label noise in the proposed method.

Curriculum learning has been widely used for handling label noises (Guo et
al., 2018) in computer vision. Arazo et al. (2019) empirically find that
networks tend to fit correct samples before noisy samples. Motivated by this,
we introduce a confidence threshold $\delta$ ($\delta\in[0,1]$) to encourage
the model to learn first from clean Other-class samples and then noisier ones.
Specifically, when calculating $\textit{Effect}_{O}$, we only select Defined-
Other-Class tokens whose predictive confidences are larger than $\delta$ for
distilling colliding effects while others are for knowledge distillation. The
value of $\delta$ changes along with the training process and the value of
$\delta$ in the $i$-th epoch is calculated as follow:

$\delta_{i}=\begin{cases}\delta_{1}+\frac{i-1}{m-1}(\delta_{m}-\delta_{1}),\quad
1\leq i\leq m\\\ \delta_{m},\quad i>m,\end{cases}$ (5)

where $m$, $\delta_{1}$ and $\delta_{m}$ are the predefined hyper-parameters
and $\delta_{m}$ should be smaller than $\delta_{1}$.

###  4.5 Balancing $\textit{Effect}_{E}$ and $\textit{Effect}_{O}$

Figure 5 shows that the total causal effect Effect consists of
$\textit{Effect}_{E}$ and $\textit{Effect}_{O}$, where $\textit{Effect}_{E}$
is for learning new entities while $\textit{Effect}_{O}$ is for reviewing old
knowledge. With the learning process of CL, the need to preserve old knowledge
varies (Hou et al., 2019). For example, more efforts should be made to
preserve old knowledge when there are 15 old classes and 1 new class v.s. 5
old classes and 1 new class. In response to this, we introduce a self-adaptive
weight for balancing $\textit{Effect}_{E}$ and $\textit{Effect}_{O}$:

$\lambda=\lambda_{base}\sqrt{C_{O}/C_{N}}$ (6)

where $\lambda_{base}$ is the initial weight and $C_{O}$,$C_{N}$ are the
numbers of old and new entity types respectively. In this way, the causal
effects from new entity types and Other-class are dynamically balanced when
the ratio of old classes to new classes changes. Finally, the objective of the
proposed method is given as follow:

$\max\textit{Effect}=\textit{Effect}_{E}+\lambda*\textit{Effect}_{O}$ (7)

Table 1: Comparisons with state-of-the-art methods on I2B2 and OntoNotes5. The
average results as well as standard derivations are provided. Mi-F1: micro-F1;
Ma-F1: macro-F1; Forget: Forgetting; : higher is better; : lower is better.
The best F1 results are bold.

|  | FG-1-PG-1 | FG-2-PG-2 | FG-8-PG-1 | FG-8-PG-2
---|---|---|---|---|---
Dataset | Method | Mi-F1 | Ma-F1 | Mi-F1 | Ma-F1 | Mi-F1 | Ma-F1 | Mi-F1 | Ma-F1
|  | 17.43 | 13.81 | 28.57 | 21.43 | 20.83 | 18.11 | 23.60 | 23.54
| Finetune Only | ±0.54 | ±1.14 | ±0.26 | ±0.41 | ±1.78 | ±1.66 | ±0.15 | ±0.38
|  | 12.31 | 17.14 | 34.67 | 24.62 | 39.26 | 27.23 | 36.22 | 26.08
| PODNet | ±0.35 | ±1.03 | ±2.65 | ±1.76 | ±1.38 | ±0.93 | ±12.9 | ±7.42
|  | 43.86 | 31.31 | 64.32 | 43.53 | 57.86 | 33.04 | 68.54 | 46.94
| LUCIR | ±2.43 | ±1.62 | ±0.76 | ±0.59 | ±0.87 | ±0.39 | ±0.27 | ±0.63
|  | 31.98 | 14.76 | 55.44 | 33.38 | 49.51 | 23.77 | 48.94 | 29.00
| ST | ±2.12 | ±1.31 | ±4.78 | ±3.13 | ±1.35 | ±1.01 | ±6.78 | ±3.04
|  | 42.85 | 24.05 | 57.01 | 35.29 | 43.95 | 23.12 | 52.25 | 30.93
| ExtendNER | ±2.86 | ±1.35 | ±4.14 | ±3.38 | ±2.01 | ±1.79 | ±5.36 | ±2.77
|  | 62.73 | 36.26 | 71.98 | 49.09 | 59.79 | 37.3 | 69.07 | 51.09
I2B2 | CFNER(Ours) | ±3.62 | ±2.24 | ±0.50 | ±1.38 | ±1.70 | ±1.15 | ±0.89 | ±1.05
|  | 15.27 | 10.85 | 25.85 | 20.55 | 17.63 | 12.23 | 29.81 | 20.05
| Finetune Only | ±0.26 | ±1.11 | ±0.11 | ±0.24 | ±0.57 | ±1.08 | ±0.12 | ±0.16
|  | 9.06 | 8.36 | 34.67 | 24.62 | 29.00 | 20.54 | 37.38 | 25.85
| PODNet | ±0.56 | ±0.57 | ±1.08 | ±0.85 | ±0.86 | ±0.91 | ±0.26 | ±0.29
|  | 28.18 | 21.11 | 64.32 | 43.53 | 66.46 | 46.29 | 76.17 | 55.58
| LUCIR | ±1.15 | ±0.84 | ±1.79 | ±1.11 | ±0.46 | ±0.38 | ±0.09 | ±0.55
|  | 50.71 | 33.24 | 68.93 | 50.63 | 73.59 | 49.41 | 77.07 | 53.32
| ST | ±0.79 | ±1.06 | ±1.67 | ±1.66 | ±0.66 | ±0.77 | ±0.62 | ±0.63
|  | 50.53 | 32.84 | 67.61 | 49.26 | 73.12 | 49.55 | 76.85 | 54.37
| ExtendNER | ±0.86 | ±0.84 | ±1.53 | ±1.49 | ±0.93 | ±0.90 | ±0.77 | ±0.57
|  | 58.94 | 42.22 | 72.59 | 55.96 | 78.92 | 57.51 | 80.68 | 60.52
OntoNotes5 | CFNER(Ours) | ±0.57 | ±1.10 | ±0.48 | ±0.69 | ±0.58 | ±1.32 | ±0.25 | ±0.84

(a) I2B2 (FG-1-PG-1)

(b) I2B2 (FG-2-PG-2)

(c) I2B2 (FG-8-PG-1)

(d) I2B2 (FG-8-PG-2)

(e) OntoNotes5 (FG-1-PG-1)

(f) OntoNotes5 (FG-2-PG-2)

(g) OntoNotes5 (FG-8-PG-1)

(h) OntoNotes5 (FG-8-PG-2)

Figure 6: Comparison of the step-wise micro-f1 score on I2B2 (16 entity
types), OntoNotes5 (18 entity types).

##  5 Experiments

###  5.1 Settings

Datasets. We conduct experiments on three widely used datasets, i.e.,
OntoNotes5 (Hovy et al., 2006), i2b2 (Murphy et al., 2010) and CoNLL2003 (Sang
and De Meulder, 2003). To ensure that each entity type has enough samples for
training, we filter out the entity types which contain less than 50 training
samples. We summarize the statistics of the datasets in Table 5 in Appendix F.

Following Monaikul et al. (2021), we split the training set into disjoint
slices, and in each slice, we only retain the labels which belong to the
entity types to learn while setting other labels to Other-class. Different
from Monaikul et al. (2021), we adopt a greedy sampling strategy to partition
the training set to better simulate the real-world scenario. Specifically, the
sampling algorithm encourages that the samples of each entity type are mainly
distributed in the slice to learn. We provide more explanations and the
detailed algorithm in Appendix B.

Training. We use bert-base-cased (Devlin et al., 2019) as the backbone model
and a fully-connected layer for classification. Following previous work in CL
(Hu et al., 2021), we predefine a fixed order of classes (alphabetically in
this study) and train models with the corresponding data slice sequentially.
Specifically, $FG$ entity types are used to train a model as the initial
model, and every $PG$ entity types are used for training in each CL step
(denoted as $FG$-$a$-$PG$-$b$). For evaluation, we only retain the new entity
types’ labels while setting other labels to Other-class in the validation set.
In each CL step, we select the model with the best validation performance for
testing and the next step’s learning. For testing, we retain the labels of all
recognized entity types while setting others to Other-class in the test set.

Metrics. Considering the class imbalance problem in NER, we adopt Micro F1 and
Macro F1 for measuring the model performance. We report the average result on
all CL steps (including the first step) as the final result.

Baselines. We consider four baselines: ExtendNER (Monaikul et al., 2021),
Self-Training (ST) (Rosenberg et al., 2005; De Lange et al., 2019), LUCIR (Hou
et al., 2019) and PODNet (Douillard et al., 2020). ExtendNER is the previous
state-of-the-art method in CL-NER. LUCIR and PODNet are state-of-the-art CL
methods in computer vision. Detailed descriptions of the baselines and their
training settings are demonstrated in Appendix C.

Hyper-Parameters. We set the number of matched tokens $K=3$, the weights
$W_{i}=1/2$ and $W_{ik}=\frac{1}{2K}$. For parameters in the curriculum
learning strategy, we set $\delta_{1}=1,\delta_{m}=0$ and $m=10$. We set the
initial value of balancing weight $\lambda_{base}=2$. More training details
are shown in Appendix D.

###  5.2 Results and Analysis

Comparisons with State-Of-The-Art. We consider two scenarios for each dataset:
(1) training the first model the same as the following CL steps; (2) training
the first model with half of all entity types. The former scenario is more
challenging, whereas the latter is closer to the real-world scenario since it
allows models to learn enough knowledge before incremental learning. Apart
from that, we consider fine-tuning without any anti-forgetting techniques
(Finetune Only) as a lower bound for comparison.

The results on I2B2 and OntoNotes5 are summarized in Table 1 and Figure 6. Due
to the space limitation, we provide the results on CoNLL2003 in Table 6 and
Figure 9 in Appendix F. In most cases, our method achieves the best
performance. Especially, our method outperforms the previous state-of-the-art
method in CL-NER (i,e., ExtendNER) by a large margin. Besides, we visualize
the features of our method and ExtendNER for comparison in Appendix E. The
performances of PODNet and LUCIR are much worse than our methods when more CL
steps are performed. The reason could be that neither of them differentiates
Other-class from entity types, and the old knowledge in Other-class is not
preserved. Our method encourages the model to review old knowledge from both
new entity types and Other-class in the form of distilling causal effects.

Table 2: The ablation study of our method on three datasets in the setting
FG-1-PG-1. AW: adaptive weight; CuL: curriculum learning strategy; Mi-F1:
micro-F1; Ma-F1: macro-F1.

Methods | I2B2 | OntoNotes5 | CoNLL2003
---|---|---|---
Mi-F1 | Ma-F1 | Mi-F1 | Ma-F1 | Mi-F1 | Ma-F1
CFNER(Ours) | 62.73 | 36.26 | 58.94 | 42.22 | 80.91 | 79.11
w/o AW | 61.65 | 35.86 | 57.63 | 40.28 | 80.75 | 78.43
w/o CuL | 61.21 | 34.79 | 57.95 | 39.54 | 80.32 | 78.71
w/o AW & CuL | 60.78 | 33.15 | 56.83 | 38.95 | 79.89 | 77.54
w/o $\textit{Effect}_{O}$ | 59.68 | 30.56 | 53.09 | 34.99 | 78.68 | 76.15
w/o $\textit{Effect}_{E}$ | 53.62 | 28.75 | 54.88 | 37.29 | 79.83 | 77.45
w/o $\textit{Effect}_{O}$ & $\textit{Effect}_{E}$ | 42.85 | 24.05 | 50.53 | 32.84 | 76.36 | 73.04

Ablation Study. We ablate our method, and the results are summarized in Table
2. To validate the effectiveness of the proposed causal framework, we only
remove the colliding effects in Other-class and new entity types for the
settings w/o $\textit{Effect}_{O}$ and w/o $\textit{Effect}_{E}$,
respectively. Specifically, in the w/o $\textit{Effect}_{O}$ setting, we apply
knowledge distillation for all Other-class samples, while in w/o
$\textit{Effect}_{E}$ setting, we calculate the cross-entropy loss for
classification. Note that our model is the same as ExtendNER when no causal
effects are used (i.e., w/o $\textit{Effect}_{O}$ & $\textit{Effect}_{E}$).
The results show that both $\textit{Effect}_{O}$ and $\textit{Effect}_{E}$
play essential roles in our framework. Furthermore, the adaptive weight and
the curriculum-learning strategy help model better learn causal effects in new
data.

Table 3: Hyper-parameter analysis on I2B2 (FG-8-PG-2). Mi-F1: micro-F1; Ma-F1:
macro-F1.

$K$ | Mi-F1 | Ma-F1 | $\delta_{1}$ | Mi-F1 | Ma-F1 | $\lambda_{base}$ | Mi-F1 | Ma-F1
---|---|---|---|---|---|---|---|---
1 | 65.48 | 46.82 | 0 | 65.25 | 46.33 | 0.5 | 69.27 | 48.71
2 | 67.12 | 49.37 | 0.5 | 66.09 | 48.51 | 1 | 69.46 | 52.45
3 | 69.07 | 51.09 | 0.9 | 68.23 | 49.1 | 2 | 69.07 | 51.09
5 | 70.25 | 52.51 | 0.95 | 68.64 | 49.66 | 5 | 64.4 | 46.65
10 | 70.69 | 52.26 | 1 | 69.07 | 51.09 | 10 | 54.12 | 40.66

Hyper-Parameter Analysis. We provide hyper-parameter analysis on I2B2 with the
setting FG-8-PG-2. We consider three hyper-parameters: the number of matched
tokens $K$, the initial value of balancing weight $\lambda_{base}$ and the
initial value of confidence threshold $\delta_{1}$. The results in Table 3
shows that a larger $K$ is beneficial. However, as $K$ becomes larger, the run
time increases correspondingly. The reason is that more forward passes are
required during training. Therefore, We select $K=3$ by default to balance
effectiveness and efficiency. Results also show that $\delta_{1}=1$ reaches
the best result, which indicates that it is more effective to learn
$\textit{Effect}_{E}$ first and then gradually introduce $\textit{Effect}_{O}$
during training. Otherwise, the old model’s wrong predictions will
significantly affect the model’s performance. Additionally, we find that the
performance drops substantially when $\lambda_{base}$ is too large. Note that
we did not carefully search for the best hyper-parameters, and the default
ones are used throughout the experiments. Therefore, elaborately adjusting the
hyper-parameters may lead to superior performances on specific datasets and
scenarios.

Table 4: Combining our methods with other baselines on three datasets in the
setting FG-1-PG-1. Mi-F1: micro-F1; Ma-F1: macro-F1. CF represents applying
causal effects.

Methods | I2B2 | OntoNotes5 | CoNLL2003
---|---|---|---
Mi-F1 | Ma-F1 | Mi-F1 | Ma-F1 | Mi-F1 | Ma-F1
CFNER(Ours) | 62.73 | 36.26 | 58.94 | 42.22 | 80.91 | 79.11
LUCIR | 43.86 | 31.31 | 28.18 | 21.11 | 74.15 | 70.48
LUCIR+CF | 66.27 | 38.52 | 62.03 | 44.34 | 81.28 | 79.56
ST | 31.98 | 14.76 | 50.71 | 33.24 | 76.17 | 72.88
ST+CF | 61.41 | 33.43 | 57.06 | 41.28 | 80.59 | 79.11

Combining with Other Baselines. Furthermore, the proposed causal framework can
be implemented as a plug-and-play module (denoted as CF). As shown in Figure
5, $\textit{Effect}_{E}$ is based on the cross-entropy loss for classifying
new entities, while $\textit{Effect}_{O}$ is based on the KL divergence loss
for the prediction-level distillation. We use LUCIR and ST as the baselines
for a demonstration. To combine LUCIR with our method, we substitute the
classification loss in LUCIR with $\textit{Effect}_{E}$ and substitute the
feature-level distillation with $\textit{Effect}_{O}$. When combining with ST,
we only replace the soft label with the hard label when calculating
$\textit{Effect}_{O}$. The results in Table 4 indicate that our method
improves LUCIR and ST substantially. It is worth noticing that LUCIR+CF
outperforms our method consistently, indicating our method has the potential
to combine with other CL methods in computer vision and reach a superior
performance in CL-NER. In addition, CFNER outperforms ST+CF due to the fact
that soft labels convey more knowledge than hard labels.

##  6 Conclusion

How can we learn from Other-class for anti-forgetting in CL-NER? This question
is answered by the proposed causal framework in this study. Although Other-
class is “useless” in traditional NER, we point out that in CL-NER, Other-
class samples are naturally reserved samples from old classes, and we can
distillate the causal effects from them to preserve old knowledge. Moreover,
we apply curriculum learning to alleviate the noisy label problem to enhance
the distillation of $\textit{Effect}_{O}$. We further introduce a self-
adaptive weight for dynamically balancing causal effects between new entity
types and Other-class. Experimental results show that the proposed causal
framework not only outperforms state-of-the-art methods but also can be
combined with other methods to further improve performances.

## Ethics Statement

For the consideration of ethical concerns, we would make description as
follows: (1) We conduct all experiments on existing datasets, which are
derived from public scientific researches. (2) We describe the statistics of
the datasets and the hyper-parameter settings of our method. Our analysis is
consistent with the experimental results. (3) Our work does not involve
sensitive data or sensitive tasks. (4) We will open source our code for
reproducibility.

## Limitations

Although the proposed method alleviates the catastrophic forgetting problem to
some extent, its performances are still unsatisfactory when more CL steps are
performed. Additionally, calculating causal effects from Other-class depends
on the old model predictions, resulting in errors propagating to the following
CL steps. Moreover, the proposed method requires more computation and a longer
training time since the predictions of matched samples are calculated.

## References

  * Altman and Krzywinski (2018) Naomi Altman and Martin Krzywinski. 2018.  The curse (s) of dimensionality.  _Nat Methods_ , 15(6):399–400.
  * Arazo et al. (2019) Eric Arazo, Diego Ortego, Paul Albert, Noel O’Connor, and Kevin McGuinness. 2019\.  Unsupervised label noise modeling and loss correction.  In _International conference on machine learning_ , pages 312–321. PMLR.
  * Bengio et al. (2009) Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009\.  Curriculum learning.  In _Proceedings of the 26th annual international conference on machine learning_ , pages 41–48.
  * Das et al. (2022) Sarkar Snigdha Sarathi Das, Arzoo Katiyar, Rebecca Passonneau, and Rui Zhang. 2022\.  CONTaiNER: Few-shot named entity recognition via contrastive learning.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 6338–6353, Dublin, Ireland. Association for Computational Linguistics.
  * De Lange et al. (2019) Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. 2019.  Continual learning: A comparative study on how to defy forgetting in classification tasks.  _arXiv preprint arXiv:1909.08383_ , 2(6).
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Douillard et al. (2020) Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle. 2020.  Podnet: Pooled outputs distillation for small-tasks incremental learning.  In _ECCV 2020-16th European Conference on Computer Vision_ , volume 12365, pages 86–102. Springer.
  * Glymour et al. (2016) Madelyn Glymour, Judea Pearl, and Nicholas P Jewell. 2016.  _Causal inference in statistics: A primer_.  John Wiley & Sons.
  * Goodfellow et al. (2013) Ian J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. 2013\.  An empirical investigation of catastrophic forgetting in gradient-based neural networks.  _arXiv preprint arXiv:1312.6211_.
  * Guo et al. (2018) Sheng Guo, Weilin Huang, Haozhi Zhang, Chenfan Zhuang, Dengke Dong, Matthew R Scott, and Dinglong Huang. 2018.  Curriculumnet: Weakly supervised learning from large-scale web images.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pages 135–150.
  * Hinton et al. (2015) Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. 2015.  Distilling the knowledge in a neural network.  _arXiv preprint arXiv:1503.02531_ , 2(7).
  * Hou et al. (2019) Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. 2019.  Learning a unified classifier incrementally via rebalancing.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 831–839.
  * Hovy et al. (2006) Eduard Hovy, Mitch Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006\.  Ontonotes: the 90% solution.  In _Proceedings of the human language technology conference of the NAACL, Companion Volume: Short Papers_ , pages 57–60.
  * Hu et al. (2021) Xinting Hu, Kaihua Tang, Chunyan Miao, Xian-Sheng Hua, and Hanwang Zhang. 2021.  Distilling causal effect of data in class-incremental learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3957–3966.
  * Kirkpatrick et al. (2017) James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. 2017.  Overcoming catastrophic forgetting in neural networks.  _Proceedings of the national academy of sciences_ , 114(13):3521–3526.
  * Li et al. (2020) Xiaoya Li, Xiaofei Sun, Yuxian Meng, Junjun Liang, Fei Wu, and Jiwei Li. 2020.  Dice loss for data-imbalanced nlp tasks.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 465–476.
  * Ma and Hovy (2016) Xuezhe Ma and Eduard Hovy. 2016.  End-to-end sequence labeling via bi-directional lstm-cnns-crf.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1064–1074.
  * McCloskey and Cohen (1989) Michael McCloskey and Neal J Cohen. 1989.  Catastrophic interference in connectionist networks: The sequential learning problem.  In _Psychology of learning and motivation_ , volume 24, pages 109–165. Elsevier.
  * Monaikul et al. (2021) Natawut Monaikul, Giuseppe Castellucci, Simone Filice, and Oleg Rokhlenko. 2021\.  Continual learning for named entity recognition.  In _Proceedings of the Thirty-Fifth AAAI Conference on Artificial Intelligence_ , pages 13570–13577.
  * Murphy et al. (2010) Shawn N Murphy, Griffin Weber, Michael Mendis, Vivian Gainer, Henry C Chueh, Susanne Churchill, and Isaac Kohane. 2010.  Serving the enterprise and beyond with informatics for integrating biology and the bedside (i2b2).  _Journal of the American Medical Informatics Association_ , 17(2):124–130.
  * Nan et al. (2021) Guoshun Nan, Jiaqi Zeng, Rui Qiao, Zhijiang Guo, and Wei Lu. 2021.  Uncovering main causalities for long-tailed information extraction.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 9683–9695.
  * Parisi et al. (2019) German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. 2019.  Continual lifelong learning with neural networks: A review.  _Neural Networks_ , 113:54–71.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019\.  Pytorch: An imperative style, high-performance deep learning library.  _Advances in neural information processing systems_ , 32.
  * Pearl (2009) Judea Pearl. 2009.  _Causality_.  Cambridge university press.
  * Pearl (2014) Judea Pearl. 2014.  Interpretation and identification of causal mediation.  _Psychological methods_ , 19(4):459.
  * Rebuffi et al. (2017) Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. 2017.  icarl: Incremental classifier and representation learning.  In _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_ , pages 2001–2010.
  * Robins (1995) Anthony Robins. 1995.  Catastrophic forgetting, rehearsal and pseudorehearsal.  _Connection Science_ , 7(2):123–146.
  * Rosenberg et al. (2005) Chuck Rosenberg, Martial Hebert, and Henry Schneiderman. 2005.  Semi-supervised self-training of object detection models.  In _Proceedings of the Seventh IEEE Workshops on Application of Computer Vision (WACV/MOTION’05)-Volume 1-Volume 01_ , pages 29–36.
  * Sang and De Meulder (2003) Erik Tjong Kim Sang and Fien De Meulder. 2003.  Introduction to the conll-2003 shared task: Language-independent named entity recognition.  In _Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003_ , pages 142–147.
  * Schölkopf (2022) Bernhard Schölkopf. 2022.  Causality for machine learning.  In _Probabilistic and Causal Inference: The Works of Judea Pearl_ , pages 765–804.
  * Tang et al. (2020) Kaihua Tang, Jianqiang Huang, and Hanwang Zhang. 2020.  Long-tailed classification by keeping the good and removing the bad momentum causal effect.  _Advances in Neural Information Processing Systems_ , 33:1513–1524.
  * Thrun (1998) Sebastian Thrun. 1998.  Lifelong learning algorithms.  In _Learning to learn_ , pages 181–209. Springer.
  * Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019.  Huggingface’s transformers: State-of-the-art natural language processing.  _arXiv preprint arXiv:1910.03771_.
  * Zhang et al. (2020) Dong Zhang, Hanwang Zhang, Jinhui Tang, Xian-Sheng Hua, and Qianru Sun. 2020.  Causal intervention for weakly-supervised semantic segmentation.  _Advances in Neural Information Processing Systems_ , 33:655–666.
  * Zhang et al. (2021) Wenkai Zhang, Hongyu Lin, Xianpei Han, and Le Sun. 2021.  De-biasing distantly supervised named entity recognition via causal intervention.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 4803–4813.
  * Zhu et al. (2020) Qingfu Zhu, Weinan Zhang, Ting Liu, and William Yang Wang. 2020.  Counterfactual off-policy training for neural dialogue generation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 3438–3448.

##  Appendix A Forgetting in CL-NER

To identify the cause of forgetting, we consider the differences in prediction
$Y$ when the old data $S$ exists or not. For each CL step, the effect of old
data $S$ can be calculate as:

$\displaystyle\textit{Effect}_{S}$
$\displaystyle=P(\hat{Y}=\hat{y}|\textit{do}(S=s))$ (8)
$\displaystyle-P(\hat{Y}=\hat{y}|\textit{do}(S=0))$
$\displaystyle=P(\hat{Y}=\hat{y}|S=s)-P(\hat{Y}=\hat{y}|S=0)$ (9)
$\displaystyle=P(\hat{Y}=\hat{y})-P(\hat{Y}=\hat{y})$ (10) $\displaystyle=0,$
(11)

where $\textit{do}(\cdot)$ is the causal intervention (Pearl, 2014, 2009)
representing that assigning a certain value to a variable without considering
all parent nodes (causes). In the first equation, $\textit{do}(S=0)$
represents null intervention, i.e., setting old data to null. In the second
equation, $P(\hat{Y}=\hat{y}|\textit{do}(S)=P(\hat{Y}=\hat{y}|S)$ due to the
fact that $S$ has no parent nodes. In the third equation,
$P(\hat{Y}=\hat{y}|S)=P(\hat{Y}=\hat{y})$ since all causal paths from $S$ to
$Y$ are blocked by the collider $X_{0}$. From Eq.(11), we find that the
missing old data effect causes the forgetting. We neglect the effect of
initial parameters adopted from the old model since it will be exponentially
decayed towards 0 during learning (Kirkpatrick et al., 2017).

(a) Greedy Sampling (b) Random Sampling

Figure 7: Comparison of the greedy sampling and random sampling on OntoNotes5.
Each slice contains one entity types to lean.

Input: $\mathcal{D}=\\{(s_{i},t_{i})\\}_{i=1}^{n}$: a training set contains
$n$ sentences $s$ and their label sequences $t$;

Require : $G$: the number of slices; $\mathcal{E}$: the entity set;
$\mathcal{E}_{j}$: the entity set in the $j$-th slice;

Output: $G$ slices of datasets:
$\mathcal{D}^{(1)},\mathcal{D}^{(2)},\cdots,\mathcal{D}^{(G)}$

1 Initialize
$\mathcal{D}^{(1)},\cdots,\mathcal{D}^{(G)}=\\{\\},\cdots,\\{\\}$;

2 Initialize $cnt_{1},\cdots,cnt_{G}=0,\cdots,0$;

// Calculate the number of sentences to allocate in each slice

3 for  _$j$ in range(1,G+1)_ do

4 $n_{j}=n*|\mathcal{E}_{j}|/|\mathcal{E}|$

5 end for

6Sort $\mathcal{E}$ in ascending order based on the frequency in
$\mathcal{D}$;

7 for  _( $s_{i},t_{i}$) in $\mathcal{D}$_ do

8 is_select = False;

// Allocate low frequency entity types in priority

9 for  _$e$ in $\mathcal{E}$_ do

10 if  _not $e$ in $t_{i}$_ then

11 continue;

12

13 end if

14 $j$ is index of $\mathcal{E}_{j}$ s.t. $e$ belongs to $\mathcal{E}_{j}$;

15 if  _$cnt_{j} <n_{j}$_ then

16 $\mathcal{D}^{(j)}=\mathcal{D}^{(j)}\cup{(s_{i},t_{i})}$;

17 $cnt_{j}$ +=1;

18 is_select = True;

19 break;

20

21 end if

22

23 end for

// Otherwise, the sentence is randomly assigned to an incomplete slice

24 if  _not is_select_ then

25 randomly choice $j$ s.t. $cnt_{j}<n_{j}$;

26 $\mathcal{D}^{(j)}=\mathcal{D}^{(j)}\cup{(s_{i},t_{i})}$;

27 $cnt_{j}$ +=1;

28

29 end if

30

31 end for

32return $\mathcal{D}^{(1)},\mathcal{D}^{(2)},\cdots,\mathcal{D}^{(G)}$;

Algorithm 1 Greedy Sampling Algorithm for CL-NER

##  Appendix B Greedy Sampling Algorithm for CL-NER

In real-world scenarios, the new data should focus on the new entity types,
i.e.,  most sentences contain the tokens from new entity types. Suppose we
randomly partition a dataset for CL-NER as in Monaikul et al. (2021). In that
case, each slice contains a large number of sentences whose tokens all belong
to Other-class, resulting in that models tend to bias to Other-class when
inference. A straightforward solution is to filter out all sentences with only
Other-class tokens. However, it brings a new problem: the slices’ sizes are
imbalanced.

To address this problem, we propose a sampling algorithm for partitioning a
dataset in CL-NER (Algorithm 1). Simply put, we allocate the sentence
containing low-frequency entity types to the corresponding slice in priority
until the slice contains the required number of sentences. If a sentence
contains no entity types or the corresponding slices are full, we randomly
allocate the sentence to an incomplete slice. In this way, we partition the
dataset into slices with balanced sizes, and each slice mainly contains the
entity types to learn.

For comparing Algorithm 1 and the random sampling as in Monaikul et al.
(2021), we provide the label distributions in each slices of training data in
Figure 7. Figure 7 shows that the greedy sampling generates more realistic
datasets for CL-NER. When we use the randomly partitioned dataset for training
in the setting FG-1-PG-1, the micro-f1 score of our method is 16.12, 17.43,
and 12.75 (%) on OntoNotes5, I2B2, and CoNLL2003, respectively, indicating
that the number of entities in each slice is inadequate for learning a NER
model. Therefore, the greedy sampling alleviates the need for data in CL-NER.

##  Appendix C Baselines Introductions and Settings

The introductions about the baselines in experiments and their experimental
settings are as follows. Note that we do not apply any reserved samples from
old classes in LUCIR and PODNet for a fair comparison since our method
requires no old data.

  * •

Self-Training (ST) (Rosenberg et al., 2005; De Lange et al., 2019): ST first
utilizes the old model to annotate the Other-class tokens with old entity
types. Then, the new model is trained on new data with annotations of all
entity types. Finally, the cross-entropy loss on all entity types is
minimized.

  * •

ExtendNER (Monaikul et al., 2021): ExtendNER has a similar idea to ST, except
that the old model provides the soft label (i.e., probability distribution) of
Other-class tokens. Specifically, the cross-entropy loss is computed for
entity types’ tokens, and KL divergence loss is computed for Other-class
tokens. During training, the sum of cross-entropy loss and KL divergence loss
is minimized. Following Monaikul et al. (2021), we set the temperature of the
teacher (old model) and student model (new model) to 1 and 2, respectively.

  * •

LUCIR (Hou et al., 2019): LUCIR develops a framework for incrementally
learning a unified classifier for the continual image classification tasks.
The total loss consists of three terms: (1) the cross-entropy loss on the new
classes samples; (2) the distillation loss on the features extracted by the
old model and those by the new one; (3) the margin-ranking loss on the
reserved samples for old classes. In our experiments, we compute the cross-
entropy loss for new entity types, the distillation loss for all entity types,
and the margin-ranking loss for Other-class samples instead of the reserved
samples. Following (Hou et al., 2019), $\lambda_{base}$ (i.e., loss weight for
the distillation loss) is set to 50, K (i.e., the top-K new class embeddings
are chosen for the margin-ranking loss) is set to 1 and m (i.e., the threshold
of margin ranking loss) is set to 0.5 for all the tasks.

  * •

PODNet (Douillard et al., 2020): PODNet has a similar idea to LUCIR to combat
the catastrophic forgetting in continual learning for image classification.
The total loss consists of the classification loss and distillation loss. To
compute the distillation loss, PodNet constrains the output of each
intermediate convolutional layer while LUCIR only considers the final feature
embeddings. For classification, PODNet used NCA loss instead of the cross-
entropy loss. In our experiments, we constrain the output of each intermediate
stage of BERT as PODNet constrains each stage of a ResNet. Following
(Douillard et al., 2020), we set the $\lambda_{c}$ (i.e., loss weight for the
POD-spatial loss) to 3 and $\lambda_{f}$ (i.e., loss weight for the POD-flat
loss) to 1.

(a) ExtendNER

(b) CFNER(Ours)

Figure 8: The T-SNE visualization of the feature representations from BERT.
The model is trained on OntoNotes5 in the setting FG-1-PG-1. We randomly
select six classes for a demonstration. Table 5: The statistics and the entity
sequence for each dataset

| # Class | # Sent | # Entity | Entity sequence (Alphabetical Order)
---|---|---|---|---
CoNLL2003 | 4 | 21k | 35k |  | LOCATION, MISC, ORGANISATION, PERSON
---
I2B2 | 16 | 141k | 29k |  | AGE, CITY, COUNTRY, DATE, DOCTOR, HOSPITAL,
---
IDNUM, MEDICALRECORD, ORGANIZATION,
PATIENT, PHONE, PROFESSION, STATE, STREET,
USERNAME, ZIP
OntoNotes5 | 18 | 77k | 104k |  | CARDINAL, DATE, EVENT, FAC, GPE, LANGUAGE,
---
LAW, LOC, MONEY, NORP, ORDINAL, ORG,
PERCENT, PERSON, PRODUCT, QUANTITY, TIME,
WORK_OF_ART

##  Appendix D Training Details

The models were implemented in Pytorch Paszke et al. (2019) on top of the BERT
Huggingface implementation Wolf et al. (2019). We use the default hyper-
parameters in BERT: hidden dimensions are 768, and the max sequence length is
512. Following Hu et al. (2021), we normalize the feature vector output by
BERT and compute the cosine similarities between the feature vector and class
representations for predictions.

We use the BIO tagging schema for all three datasets. For CoNLL2003, we train
the model for ten epochs in each CL step. For OntoNotes5 and I2B2, we train
the model for 20 epochs when PG=2, and 10 epochs when PG=1. The batch size is
set as 8, and the learning rate is set as 4e-4. The experiments are run on
GeForce RTX 2080 Ti GPU. Each experiment is repeated 5 times.

##  Appendix E T-SNE Visualizations

To deepen the understanding of the forgetting in CL-NER, we visualize the
feature representations from BERT in Figure 8. Results show that our method
preserves more old knowledge and learns better feature representations than
ExtendNER.

##  Appendix F Additional Experimental Results

(a) CoNLL2003 (FG-1-PG-1)

(b) CoNLL2003 (FG-2-PG-1)

Figure 9: Comparison of the step-wise micro-f1 score on CoNLL2003 (4 entity
types). Table 6: Comparisons with state-of-the-art methods on CoNLL2003. The
average results as well as standard derivations are provided. Mi-F1: micro-F1;
Ma-F1: macro-F1. The best F1 results are bold.

|  | FG-1-PG-1 | FG-2-PG-1
---|---|---|---
Dataset | Method | Mi-F1 | Ma-F1 | Mi-F1 | Ma-F1
|  | 50.84 | 40.64 | 57.45 | 43.58
| Finetune Only | ±0.10 | ±0.16 | ±0.05 | ±0.18
|  | 36.74 | 29.43 | 59.12 | 58.39
| PODNet | ±0.52 | ±0.28 | ±0.54 | ±0.99
|  | 74.15 | 70.48 | 80.53 | 77.33
| LUCIR | ±0.43 | ±0.66 | ±0.31 | ±0.31
|  | 76.17 | 72.88 | 76.65 | 66.72
| ST | ±0.91 | ±1.12 | ±0.24 | ±0.11
|  | 76.36 | 73.04 | 76.66 | 66.36
| ExtendNER | ±0.98 | ±1.8 | ±0.66 | ±0.64
|  | 80.91 | 79.11 | 80.83 | 75.20
CoNLL2003 | CFNER(Ours) | ±0.29 | ±0.50 | ±0.36 | ±0.32