# Learning Executable Semantic Parsers for
Natural Language Understanding

Percy Liang

Computer Science Department  Stanford University  Stanford, CA
<EMAIL_ADDRESS>
###### Abstract

For building question answering systems and natural language interfaces,
semantic parsing has emerged as an important and powerful paradigm. Semantic
parsers map natural language into logical forms, the classic representation
for many important linguistic phenomena. The modern twist is that we are
interested in learning semantic parsers from data, which introduces a new
layer of statistical and computational issues. This article lays out the
components of a statistical semantic parser, highlighting the key challenges.
We will see that semantic parsing is a rich fusion of the logical and the
statistical world, and that this fusion will play an integral role in the
future of natural language understanding systems.

###### category:

I.2.7 Artificial Intelligence Natural Language Processing

###### keywords:

Language parsing and understanding

##  1 Introduction

A long-standing goal of artificial intelligence (AI) is to build systems
capable of understanding natural language. To focus the notion of
“understanding” a bit, let us say that the system must produce an appropriate
action upon receiving an input utterance from a human. For example:

Context: | knowledge of mathematics
---|---
Utterance: | _What is the largest prime less than 10?_
Action: | 7

Context: | knowledge of geography
---|---
Utterance: | _What is the tallest mountain in Europe?_
Action: | Mt. Elbrus

Context: | user’s calendar
---|---
Utterance: | _Cancel all my meetings after 4pm tomorrow._
Action: | (removes meetings from calendar)

We are interested in utterances such as the ones above, which require deep
understanding and reasoning. This article focuses on _semantic parsing_ , an
area within the field of natural language processing (NLP), which has been
growing over the last decade. Semantic parsers map input utterances into
semantic representations called _logical forms_ that support this form of
reasoning. For example, the first utterance above would map onto the logical
form $\max(\texttt{primes}\cap(-\infty,10))$. We can think of the logical form
as a program that is _executed_ to yield the desired behavior (e.g., answering
$7$). The second utterance would map onto a database query; the third, onto an
invocation of a calendar API.

Semantic parsing is rooted in formal semantics, pioneered by logician Richard
Montague (montague73ptq, ), who famously argued that there is “no important
theoretical difference between natural languages and the artificial languages
of logicians.” Semantic parsing, by residing in the practical realm, is more
exposed to the differences between natural language and logic, but it inherits
two general insights from formal semantics: The first idea is _model theory_ ,
which states that expressions (e.g., primes) are mere symbols which only
obtain their meaning or denotation (e.g., $\\{2,3,5,\dots\\}$) by executing
the expression with respect to a model, or in our terminology, a context. This
property allows us to factor out the understanding of language (semantic
parsing) from world knowledge (execution). Indeed, one can understand the
utterance “ _What is the largest prime less than 10?_ ” without actually
computing the answer. The second idea is _compositionality_ , a principle
often attributed to Gottlob Frege, which states that the denotation of an
expression is defined recursively in terms of the denotations of its
subexpressions. For example, primes denotes the set of primes, $(-\infty,10)$
denotes the set of numbers smaller than $10$, and so
$\texttt{primes}\cap(-\infty,10)$ denotes the intersection of those two sets.
This compositionality is what allows us to have a succinct characterization of
meaning for a combinatorial range of possible utterances.

Early systems. Logical forms have played a foundational role in natural
language understanding systems since their genesis in the 1960s. Early
examples included LUNAR, a natural language interface into a database about
moon rocks (woods72lunar, ), and SHRDLU, a system that could both answer
questions and perform actions in a toy blocks world environment
(winograd1972language, ). For their time, these systems were significant
achievements. They were able to handle fairly complex linguistic phenomena and
integrate syntax, semantics, and reasoning in an end-to-end application. For
example, SHRDLU was able to process “ _Find a block which is taller than the
one you are holding and put it into the box._ ” However, as the systems were
based on hand-crafted rules, it became increasingly difficult to generalize
beyond the narrow domains and handle the intricacies of general language.

Rise of machine learning. In the early 1990s, influenced by the successes of
statistical techniques in the neighboring speech recognition community, the
field of NLP underwent a statistical revolution. Machine learning offered a
new paradigm: Collect _examples_ of the desired input-output behavior and then
fit a statistical model to these examples. The simplicity of this paradigm
coupled with the increase in data and computation allowed machine learning to
prevail.

What fell out of favor was not only rule-based methods, but also the natural
language understanding problems. In the statistical NLP era, much of the
community’s attention turned to tasks—documentation classification, part-of-
speech tagging, and syntactic parsing—which fell short of full end-to-end
understanding. Even question answering systems relied less on understanding
and more on a shallower analysis coupled with a large collection of
unstructured text documents (brill2002askmsr, ), typified by the TREC
competitions.

Statistical semantic parsing. The spirit of deep understanding was kept alive
by researchers in statistical semantic parsing (zelle96geoquery, ;
miller96statistical, ; wong07synchronous, ; zettlemoyer05ccg, ;
kwiatkowski10ccg, ). A variety of different semantic representations and
learning algorithms were employed, but all of these approaches relied on
having a labeled dataset of natural language utterances paired with annotated
logical forms, for example:

Utterance: | _What is the largest prime less than 10?_
---|---
Logical form: | $\max(\texttt{primes}\cap(-\infty,10))$

Weak supervision. Over the last few years, two exciting developments have
really spurred interest in semantic parsing. The first is reducing the amount
of supervision from annotated logical forms to answers (clarke10world, ;
liang11dcs, ):

Utterance: | _What is the largest prime less than 10?_
---|---
Action: | 7

This form of supervision is much easier to obtain via crowdsourcing. Although
the logical forms are not observed, they are still modeled as latent
variables, which must be inferred from the answer. This results in a more
difficult learning problem, but liang11dcs showed that it is possible to solve
it without degrading accuracy.

Scaling up. The second development is the scaling up of semantic parsers to
more complex domains. Previous semantic parsers had only been trained on
limited domains such as US geography, but the creation of broad-coverage
knowledge bases such as Freebase (bollacker2008freebase, ) set the stage for a
new generation of semantic parsers for question answering. Initial systems
required annotated logical forms (cai2013large, ), but soon, systems became
trainable from answers (berant2013freebase, ; kwiatkowski2013scaling, ;
berant2014paraphrasing, ). Semantic parsers have even been extended beyond
fixed knowledge bases to semi-structured tables (pasupat2015compositional, ).
With the ability to learn semantic parsers from question-answer pairs, it is
easy to collect datasets via crowdsourcing. As a result, semantic parsing
datasets have grown by an order of magnitude.

In addition, semantic parsers have been applied to a number of applications
outside question answering: robot navigation (tellex2011understanding, ;
artzi2013weakly, ), identifying objects in a scene (matuszek2012grounded, ;
krishnamurthy2013jointly, ), converting natural language to regular
expressions (kushman2013regex, ), and many others.

Outlook. Today, the semantic parsing community is a vibrant field, but it is
still young and grappling with the complexities of the natural language
understanding problem. Semantic parsing poses three types of challenges:

  * •

Linguistic: How should we represent the semantics of natural language and
construct it compositionally from the natural language?

  * •

Statistical: How can we learn semantic parsers from weak supervision and
_generalize_ well to new examples?

  * •

Computational: How do we efficiently search over the combinatorially large
space of possible logical forms?

The rest of the paper is structured as follows: We first present a general
framework for semantic parsing, introducing the key components (Section 2).
The framework is pleasantly modular, with different choices of the components
corresponding to existing semantic parsers in the literature (Section 3). We
describe the datasets (Section 4) and then conclude (Section 5).

##  2 Framework

Figure 1:  A natural language understanding problem where the goal is to map
an utterance $x$ in a context $c$ to an action $y$.

Natural language understanding problem. In this article, we focus on the
following natural language understanding problem: Given an _utterance_ $x$ in
a _context_ $c$, output the desired _action_ $y$. Figure 1 shows the setup for
a question answering application, in which case $x$ is a question, $c$ is a
knowledge base, and $y$ is the answer. In a robotics application, $x$ is a
command, $c$ represents the robot’s environment, and $y$ is the desired
sequence of actions to be carried by the robot (tellex2011understanding, ). To
build such a system, assume that we are given a set of $n$ examples
$\\{(x_{i},c_{i},y_{i})\\}_{i=1}^{n}$. We would like to use these examples to
train a model that can generalize to new unseen utterances and contexts.

Semantic parsing components. This article focuses on a statistical semantic
parsing approach to the above problem, where the key is to posit an
intermediate _logical form_ $z$ that connects $x$ and $y$. Specifically, $z$
captures the semantics of the utterance $x$, and it also executes to the
action $y$ (in the context of $c$). In our running example, $z$ would be
$\texttt{max}(\texttt{primes}\cap(-\infty,10))$. Our semantic parsing
framework consists of the following five components (see Figure 2):

  1. 1.

Executor: computes the denotation (action) $y=\llbracket z\rrbracket_{c}$
given a logical form $z$ and context $c$. This defines the semantic
representation (logical forms along with their denotations).

  2. 2.

Grammar: a set of rules $G$ that produces $D(x,c)$, a set of candidate
derivations of logical forms.

  3. 3.

Model: specifies a distribution $p_{\theta}(d\mid x,c)$ over derivations $d$
parameterized by $\theta$.

  4. 4.

Parser: searches for high probability derivations $d$ under the model
$p_{\theta}$.

  5. 5.

Learner: estimates the parameters $\theta$ (and possibly rules in $G$) given
training examples $\\{(x_{i},c_{i},y_{i})\\}_{i=1}^{n}$.

Figure 2:  Semantic parsing framework depicting the executor, grammar, and
model. The parser and learner are algorithmic components that are responsible
for generating the logical form $z$ and parameters $\theta$, respectively.

We now instantiate each of these components for our running example: “ _What
is the largest prime less than 10?_ ”

Executor. Let the semantic representation be the language of mathematics, and
the executor is the standard interpretation, where the interpretations of
predicates (e.g., primes) are given by $c$. With
$c(\texttt{primes})=\\{2,3,5,7,11,\dots,\\}$, the denotation is
$\llbracket\texttt{primes}\cap(-\infty,10)\rrbracket_{c}=\\{2,3,5,7\\}$.

Grammar. The grammar $G$ connects utterances to possible _derivations_ of
logical forms. Formally, the grammar is a set of rules of the form
$\alpha\Rightarrow\beta$.111 The standard way context-free grammar rules are
written is $\beta\to\alpha$. Because our rules build logical forms, reversing
the arrow is more natural.  Here is a simple grammar for our running example:

(R1) | _prime_ | $\Rightarrow$ | $\text{\sc{NP}}[\texttt{primes}]$
---|---|---|---
(R2) | _10_ | $\Rightarrow$ | $\text{\sc{NP}}[10]$
(R3) |  _less than_ $\text{\sc{NP}}[z]$ | $\Rightarrow$ | $\text{\sc{QP}}[(-\infty,z)]$
(R4) |  $\text{\sc{NP}}[z_{1}]$ $\text{\sc{QP}}[z_{2}]$ | $\Rightarrow$ | $\text{\sc{NP}}[z_{1}\cap z_{2}]$
(R5) |  _largest_ $\text{\sc{NP}}[z]$ | $\Rightarrow$ | $\text{\sc{NP}}[\max(z)]$
(R6) |  _largest_ $\text{\sc{NP}}[z]$ | $\Rightarrow$ | $\text{\sc{NP}}[\min(z)]$
(R7) |  _What is the_ $\text{\sc{NP}}[z]$? | $\Rightarrow$ | $\text{\sc{ROOT}}[z]$

We start with the input utterance and repeatedly apply rules in $G$. A rule
$\alpha\Rightarrow\beta$ can be applied if some span of the utterance matches
$\alpha$, in which case a derivation over the same span with a new syntactic
category and logical form according to $\beta$ is produced. Here is one
possible derivation (call it $d_{1}$) for our running example:

$\displaystyle\begin{array}[]{*{6}{c}}\emph{What is
the}&\emph{largest}&\emph{prime}&\emph{less than}&\emph{10}&\emph{?}\\\
&&\lx@intercol\hfil\hrulefill\text{\tiny(R1)}\hrulefill\hfil\lx@intercol&&\lx@intercol\hfil\hrulefill\text{\tiny(R2)}\hrulefill\hfil\lx@intercol&\\\
&&\text{\sc{NP}}[\texttt{primes}]&&\text{\sc{NP}}[10]&\\\
&&&\lx@intercol\hfil\hrulefill\text{\tiny(R3)}\hrulefill\hfil\lx@intercol&\\\
&&&\lx@intercol\hfil\text{\sc{QP}}[(-\infty,10)]\hfil\lx@intercol&\\\
&&\lx@intercol\hfil\hrulefill\text{\tiny(R4)}\hrulefill\hfil\lx@intercol&\\\
&&\lx@intercol\hfil\text{\sc{NP}}[\texttt{primes}\cap(-\infty,10)]\hfil\lx@intercol&\\\
&\lx@intercol\hfil\hrulefill\text{\tiny(R5)}\hrulefill\hfil\lx@intercol&\\\
&\lx@intercol\hfil\text{\sc{NP}}[\max(\texttt{primes}\cap(-\infty,10))]\hfil\lx@intercol&\\\
\lx@intercol\hfil\hrulefill\text{\tiny(R7)}\hrulefill\hfil\lx@intercol\\\
\lx@intercol\hfil\text{\sc{ROOT}}[\max(\texttt{primes}\cap(-\infty,10))]\hfil\lx@intercol\\\
\end{array}$ (12)

For example, applying (R3) produces category QP and logical form
$(-\infty,10)$ over span $[5\\!:\\!7]$ corresponding to “ _less than 10_ ”. We
stop when we produce the designated ROOT category over the entire utterance.
Note that we could have also applied (R6) instead of (R5) to generate the
(incorrect) logical form $\min(\texttt{primes}\cap(-\infty,10))$; let this
derivation be $d_{2}$. We have $D(x,c)=\\{d_{1},d_{2}\\}$ here, but in
general, there could be exponentially many derivations, and multiple
derivations can generate the same logical form. In general, the grammar might
contain nonsense rules (R6) that do not reflect ambiguity in language but are
rather due to model uncertainty prior to learning.

Model. The model scores the set of candidate derivations generated by the
grammar. A common choice used by virtually all existing semantic parsers are
log-linear models (generalizations of logistic regressions). In a log-linear
model, define a _feature vector_ $\phi(x,c,d)\in\mathbb{R}^{F}$ for each
possible derivation $d$. We can think of each feature as casting a vote for
various derivations $d$ based on some coarse property of the derivation. For
example, define $F=7$ features, each counting the number of times a given
grammar rule is invoked in $d$, so that $\phi(x,c,d_{1})=[1,1,1,1,1,0,1]$ and
$\phi(x,c,d_{2})=[1,1,1,1,0,1,1]$.

Next, let $\theta\in\mathbb{R}^{F}$ denote the _parameter vector_ , which
defines a weight for each feature representing how reliable that feature is.
Their weighted combination $\text{score}(x,c,d)=\phi(x,c,d)\cdot\theta$
represents how good the derivation is. We can exponentiate and normalize these
scores to obtain a distribution over derivations:

$\displaystyle p_{\theta}(d\mid
x,c)=\frac{\exp(\text{score}(x,c,d))}{\sum_{d^{\prime}\in
D(x,c)}\exp(\text{score}(x,c,d^{\prime}))}.$ (13)

If $\theta=[0,0,0,0,+1,-1,0]$, then $p_{\theta}$ would assign probability
$\frac{\exp(1)}{\exp(1)+\exp(-1)}\approx 0.88$ to $d_{1}$ and $\approx 0.12$
to $d_{2}$.

Parser. Given a trained model $p_{\theta}$, the parser (approximately)
computes the highest probability derivation(s) for an utterance $x$ under
$p_{\theta}$. Assume the utterance $x$ is represented as a sequence of tokens
(words). A standard approach is to use a _chart parser_ , which recursively
builds derivations for each span of the utterance. Specifically, for each
category $A$ and span $[i\\!:\\!j]$ (where $0\leq i<j\leq\text{length}(x)$),
we loop over the applicable rules in the grammar $G$ and apply each one to
build new derivations of category $A$ over $[i\\!:\\!j]$. For binary
rules—those of the form $B\,C\Rightarrow A$ such as (R4), we loop over split
points $k$ (where $i<k\leq j$), recursively compute derivations $B[z_{1}]$
over $[i\\!:\\!k]$ and $C[z_{2}]$ over $[k\\!:\\!j]$, and combine them into a
new derivation $A[z]$ over $[i\\!:\\!j]$, where $z$ is determined by the rule;
for example, $z=z_{1}\cap z_{2}$ for (R4). The final derivations for the
utterance are collected in the ROOT category over span
$[0\\!:\\!\text{length}(x)]$.

The above procedure would generate all derivations, which could be
exponentially large. Generally, we only wish to compute compute the
derivations with high probability under our model $p_{\theta}$. If the
features of $p_{\theta}$ were to _decompose_ as a sum over the rule
applications in $d$—that is, $\phi(x,c,d)=\sum_{(r,i,j)\in
d}\phi_{\text{rule}}(x,c,r,i,j)$, then we could use dynamic programming: For
each category $A$ over $[i\\!:\\!j]$, compute the highest probability
derivation. However, in executable semantic parsing, feature decomposition
isn’t sufficient, since during learning, we also need to incorporate the
constraint that the logical form executes to the true denotation
($\mathbb{I}[\llbracket d.z\rrbracket_{c}=y]$); see (17) below. To maintain
exact computation in this setting, the dynamic programming state would need to
include the entire logical form $d.z$, which is infeasible, since there are
exponentially many logical forms. Therefore, _beam search_ is generally
employed, where we keep only the $K$ sub-derivations with the highest model
score based on only features of the sub-derivations. Beam search is not
guaranteed to return the $K$ highest scoring derivations, but it is often an
effective heuristic.

Learner. While the parser turns parameters into derivations, the learner
solves the inverse problem. The dominant paradigm in machine learning is to
set up an objective function and optimize it. A standard principle is to
maximize the likelihood of the training data
$\\{(x_{i},c_{i},y_{i})\\}_{i=1}^{n}$. An important point is that we don’t
observe the correct derivation for each example, but only the action $y_{i}$,
so we must consider all derivations $d$ whose logical form $d.z$ satisfy
$\llbracket d.z\rrbracket_{c_{i}}=y_{i}$. This results in the log-likelihood
of the observed action $y_{i}$:

$\displaystyle\mathcal{O}_{i}(\theta)$
$\displaystyle\stackrel{{\scriptstyle\rm
def}}{{=}}\log\sum_{\stackrel{{\scriptstyle d\in D(x_{i},c_{i})}}{{\llbracket
d.z\rrbracket_{c_{i}}=y_{i}}}}p_{\theta}(d\mid x_{i},c_{i}).$ (14)

The final objective is then simply the sum across all $n$ training examples:

$\displaystyle\mathcal{O}(\theta)$ $\displaystyle\stackrel{{\scriptstyle\rm
def}}{{=}}\sum_{i=1}^{n}\mathcal{O}_{i}(\theta),$ (15)

The simplest approach to maximize $\mathcal{O}(\theta)$ is to use _stochastic
gradient descent_ (SGD), an iterative algorithm that takes multiple passes
(e.g., say 5) over the training data and makes the following update on example
$i$:

$\displaystyle\theta\leftarrow\theta+\eta\nabla\mathcal{O}_{i}(\theta),$ (16)

where $\eta$ is a step size that governs how aggressively we want to update
parameters (e.g., $\eta=0.1$). In the case of log-linear models, the gradient
has a nice interpretable form:

$\displaystyle\nabla\mathcal{O}_{i}(\theta)=\sum_{d\in
D(x_{i},c_{i})}(q(d)-p_{\theta}(d\mid x_{i},c_{i}))\phi(x_{i},c_{i},d),$ (17)

where $q(d)\propto p_{\theta}(d\mid x_{i},c_{i})\mathbb{I}[\llbracket
d.z\rrbracket_{c_{i}}=y_{i}]$ is the model distribution $p_{\theta}$ over
derivations $d$, but restricted to ones consistent with $y_{i}$. The gradient
pushes $\theta$ to put more probability mass on $q$ and less on $p_{\theta}$.
For example, if $p_{\theta}$ assigns probabilities $[0.2,0.4,0.1,0.3]$ to four
derivations and the middle two derivations are consistent, then $q$ assigns
probabilities $[0,0.8,0.2,0]$.

The objective function $\mathcal{O}(\theta)$ is not concave, so SGD is at best
guaranteed to converge to a local optimum, not a global one. Another problem
is that we cannot enumerate all derivations $D(x_{i},c_{i})$ generated by the
grammar, so we approximate this set with the result of beam search, which
yields $K$ candidates (typically $K=200$); $p_{\theta}$ is normalized over
this set. Note that this candidate set depends on the current parameters
$\theta$, resulting a heuristic approximation of the gradient
$\nabla\mathcal{O}_{i}$.

Summary. We have covered the components of a semantic parsing system. Observe
that the components are relatively loosely coupled: The executor is concerned
purely with what we want to express independent of how it would be expressed
in natural language. The grammar describes how candidate logical forms are
constructed from the utterance but does not provide algorithmic guidance nor
specify a way to score the candidates. The model focuses on a particular
derivation and defines features that could be helpful for predicting
accurately. The parser and the learner provide algorithms largely independent
of semantic representations. This modularity allows us to improve each
component in isolation.

##  3 Refining the Components

Having toured the components of a semantic parsing system, we now return to
each component and discuss the key design decisions and possibilities for
improvement.

###  3.1 Executor

By describing an executor, we are really describing the language of the
logical form. A basic textbook representation of language is _first-order
logic_ , which can be used to make quantified statements about relations
between objects. For example, “ _Every prime greater than two is odd._ ” would
be expressed in first-order logic as $\forall
x.\texttt{prime}(x)\wedge\texttt{more}(x,2)\to\texttt{odd}(x)$. Here, the
context $c$ is a model (in the model theory sense), which maps predicates to
sets of objects or object pairs. The execution of the above logical form with
respect to the standard mathematical context would be true.
blackburn05semantics gives a detailed account on how first-order logic is used
for natural language semantics.

First-order logic is reasonably powerful, but it fails to capture some common
phenomena in language. For example, “ _How many primes are less than 10?_ ”
requires constructing a set and manipulating it and thus goes beyond the power
of first-order logic. We can instead augment first-order logic with constructs
from _lambda calculus_. The logical form corresponding to the above question
would be $\texttt{count}(\lambda
x.\texttt{prime}(x)\wedge\texttt{less}(x,10))$, where the $\lambda$ operator
can be thought of as constructing a set of all $x$ that satisfy the condition;
in symbols, $\llbracket\lambda x.f(x)\rrbracket_{c}=\\{x:\llbracket
f(x)\rrbracket_{c}=\texttt{true}\\}$. Note that count is a higher-order
functions that takes a function as an argument.

Another logical language, which can be viewed as syntactic sugar for lambda
calculus, is _lambda dependency-based semantics (DCS)_ (liang2013lambdadcs, ).
In lambda DCS, the above logical form would be
$\texttt{count}(\texttt{prime}\sqcap(\texttt{less}.10))$, where the constant
$10$ represent $\lambda x.(x=10)$, the intersection operator $z_{1}\sqcap
z_{2}$ represents $\lambda x.z_{1}(x)\wedge z_{2}(x)$, and the join operator
$r.z$ represents $\lambda x.\exists y.r(x,y)\wedge z(y)$.

Lambda DCS is “lifted” in the sense that operations combine functions from
objects to truth values (think sets) rather than truth values. As a result,
lambda DCS logical forms partially eliminate the need for variables. Noun
phrases in natural language (e.g., “ _prime less than 10_ ”) also denote sets.
Thus, lambda DCS arguably provides a transparent interface with natural
language.

From a linguistic point of view, the logical language seeks primarily to model
natural language phenomena. From an application point of view, the logical
language dictates what actions we support. It is thus common to use
application-specific logical forms, for example, regular expressions
(kushman2013regex, ). Note that the other components of the framework are
largely independent of the exact logical language used.

###  3.2 Grammar

Recall that the goal of the grammar in this article is just to define a set of
candidate derivations for each utterance and context. Note that this is in
contrast to a conventional notion of a grammar in linguistics, where the goal
is to precisely characterize the set of valid sentences and interpretations.
This divergence is due to two factors: First, we will learn a statistical
model over the derivations generated by the grammar anyway, so the grammar can
be simple and coarse. Second, we might be interested in application-specific
logical forms. In a flight reservation domain, the logical form we wish to
extract from “ _I’m in Boston and would like to go to Portland_ ” is
$\texttt{flight}\sqcap\texttt{from}.\texttt{Boston}\sqcap\texttt{to}.\texttt{Portland}$,
which is certainly not the full linguistic meaning of the utterance, but
suffices for the task at hand. Note that the connection here between language
and logic is less direct compared to “ _prime less than 10_ ”
$\Rightarrow\texttt{prime}\sqcap(\texttt{less}.10)$.

CCG. One common approach to the grammar in semantic parsing is _combinatory
categorial grammar_ (CCG) (steedman00ccg, ), which had been developed
extensively in linguistics before it was first used for semantic parsing
(zettlemoyer05ccg, ). CCG is typically coupled with logical forms in lambda
calculus. Here is an example of a CCG derivation:

$\displaystyle\small\begin{array}[]{*{3}{c}}\emph{prime}&\emph{less
than}&\emph{10}\\\
\lx@intercol\hrulefill\hfil\hfil\lx@intercol&\lx@intercol\hrulefill\hfil\hfil\lx@intercol&\lx@intercol\hrulefill\hfil\hfil\lx@intercol\\\
\text{\sc{N}}[\lambda
x.\texttt{prime}(x)]&(\text{\sc{N}}\backslash\text{\sc{N}})/\text{\sc{NP}}[\lambda
y.\lambda f.\lambda x.f(x)\wedge\texttt{less}(x,y)&\text{\sc{NP}}[10]\\\
&\lx@intercol\hfil\hrulefill\text{\tiny(>)}\hrulefill\hfil\lx@intercol\\\
&\lx@intercol\hfil\text{\sc{N}}\backslash\text{\sc{N}}[\lambda f.\lambda
x.f(x)\wedge\texttt{less}(x,10)]\hfil\lx@intercol\\\
\lx@intercol\hfil\hrulefill\text{\tiny(<)}\hrulefill\hfil\lx@intercol\\\
\lx@intercol\hfil\text{\sc{N}}[\lambda
x.\texttt{prime}(x)\wedge\texttt{less}(x,10)]\hfil\lx@intercol\\\ \end{array}$
(25)

Syntactic categories in CCG include nouns (N) denoting sets and noun phrases
(NP) denoting objects. Composite categories
($(\text{\sc{N}}\backslash\text{\sc{N}})/\text{\sc{NP}}$) represent functions
of multiple arguments, but where the directionality of the slashes indicate
the location of the arguments. Most rules in CCG are lexical (e.g., [_prime_
$\Rightarrow\text{\sc{N}}[\lambda x.\texttt{prime}(x)$]), which mention
particular words. The rest of the rules are glue; for example, we have a
forward application ($>$) and backward application ($<$) rule:

(>) | $A/B[f]$ | $B[x]$ | $\Rightarrow$ | $A[f(x)]$
---|---|---|---|---
(<) | $B[x]$ | $A\backslash B[f]$ | $\Rightarrow$ | $A[f(x)]$

It is common to use other combinators which both handle more complex
linguistic phenomena such as NP coordination “ _integers that divide 2 or 3_ ”
as well as ungrammatical language (zettlemoyer07relaxed, ), although these
issues can also be handled by having a more expansive lexicon
(kwiatkowski10ccg, ).

Crude rules. In CCG, the lexical rules can become quite complicated. An
alternative approach that appears mostly in work on lambda DCS is to adopt a
much cruder grammar whose rules correspond directly to the lambda DCS
constructions, join and intersect. This results in the following derivation:

$\displaystyle\small\begin{array}[]{*{3}{c}}\emph{prime}&\emph{less than}&\emph{10}\\\ \lx@intercol\hrulefill\hfil\hfil\lx@intercol&\lx@intercol\hrulefill\hfil\hfil\lx@intercol&\lx@intercol\hrulefill\hfil\hfil\lx@intercol\\\ \text{\sc{N}}[\texttt{prime}]&\text{\sc{N|N}}[\texttt{less}]&\text{\sc{N}}[10]\\\ &\lx@intercol\hfil\hrulefill\text{\tiny(join)}\hrulefill\hfil\lx@intercol\\\ &\lx@intercol\hfil\text{\sc{N}}[\texttt{less}.10]\hfil\lx@intercol\\\ \lx@intercol\hfil\hrulefill\text{\tiny(intersect)}\hrulefill\hfil\lx@intercol\\\ \lx@intercol\hfil\text{\sc{N}}[\texttt{prime}\sqcap\texttt{less}.10]\hfil\lx@intercol\\\ \end{array}$ (33) (join) | $\text{\sc{N|N}}[r]$ | $\text{\sc{N}}[z]$ | $\Rightarrow$ | $\text{\sc{N}}[r.z]$
---|---|---|---|---
(intersect) | $\text{\sc{N}}[z_{1}]$ | $\text{\sc{N}}[z_{2}]$ | $\Rightarrow$ | $\text{\sc{N}}[z_{1}\sqcap z_{2}]$

The lack of constraints permits the derivation of other logical forms such as
$\texttt{10}\sqcap\texttt{less}.\texttt{prime}$, but these incorrect logical
forms can be ruled out statistically via the model. The principle is to keep
the lexicon simple and lean more heavily on features (whose weights are
learned from data) to derive drive the selection of logical forms.

Floating rules. While this new lexicon is simpler, the bulk of the complexity
comes from having it in the first place. Where does the rules [_prime_
$\Rightarrow\text{\sc{N|N}}:\texttt{prime}$] and [_less than_
$\Rightarrow\text{\sc{N|N}}:\texttt{less}$] come from? We can reduce the
lexicon even further by introducing _floating_ rules
[$\emptyset\Rightarrow\text{\sc{N|N}}:\texttt{prime}$] and
[$\emptyset\Rightarrow\text{\sc{N|N}}:\texttt{less}$], which can be applied in
the absence of any lexical trigger. This way, the utterance “ _primes smaller
than 10_ ” would also be able to generate
$\texttt{prime}\sqcap\texttt{less}.10$, where the predicates prime and less
are not anchored to any specific words.

While this relaxation may seem linguistically blasphemous, recall that the
purpose of the grammar is to merely deliver a set of logical forms, so
floating rules are quite sensible provided we can keep the set of logical
forms under control. Of course, since the grammar is so flexible, even more
nonsensical logical forms are generated, so we must lean heavily on the
features to score the derivations properly. When the logical forms are simple
and we have strong type constraints, this strategy can be quite effective
(berant2014paraphrasing, ; wang2015overnight, ; pasupat2015compositional, ).

The choice of grammar is arguably the most important component of a semantic
parser, as it most directly governs the tradeoff between expressivity and
statistical/computational complexity. We have explored three grammars, from a
tightly-constrained CCG to a very laissez faire floating grammar. CCG is
natural for capturing complex linguistic phenomena in well-structured
sentences, whereas for applications where the utterances are noisy but the
logical forms are simple, more flexible grammars are appropriate. In practice,
one would probably want to mix and match depending on the domain.

###  3.3 Model

At the core of a statistical model is a function $\text{score}(x,c,d)$ that
judges how good a derivation $d$ is with respect to the utterance $x$ and
context $c$. In Section 2, we described a simple log-linear model (13) in
which $\text{score}(x,c,d)=\phi(x,c,d)\cdot\theta$, and with simple rule
features. There are two ways to improve the model: use more expressive
features and and use a non-linear scoring function.

Most existing semantic parsers stick with a linear model and leverage more
targeted features. One simple class of features $\\{\phi_{a,b}\\}$ is equal to
the number of times word $a$ appears in the utterance and predicate $b$ occurs
in the logical form; if the predicate $b$ is generated by a floating rule,
then $a$ is allowed to range over the entire utterance; otherwise, it must
appear in the span of $b$.

The above features are quite numerous, which provides flexibility but require
a lot of data. Note that logical predicates (e.g., birthplace) and the natural
language (e.g., _born in_) often share a common vocabulary (English words).
One can leverage this structure by defining a few features based on statistics
from large corpora, external lexical resources, or simply string overlap. For
example, $\phi_{\text{match}}$ might be the number of word-predicate pairs
with that match almost exactly. This way, the lexical mapping need not be
learned from scratch using only the training data (berant2013freebase, ;
kwiatkowski2013scaling, ).

A second class of useful features are based on the denotation $\llbracket
d.z\rrbracket_{c}$ of the predicted logical form. For example, in a robot
control setting, a feature would encode whether the predicted actions are
valid in the environment $c$ (picking up a cup requires the cup to be
present). These features make it amply clear that semantic parsing is not
simply a natural language problem but one that involves jointly reasoning
about the non-linguistic world. In addition, we typically include type
constraints (features with weight $-\infty$) to prune out ill-formed logical
forms such as $\texttt{birthplace}.4$ that are licensed the grammar.

One could also adopt a non-linear scoring function, which does not require any
domain knowledge about semantic parsing. For example, one could use a simple
one-layer neural network, which takes a weighted combination of $m$ non-linear
basis functions:
$\text{score}(x,c,d)=\sum_{i=1}^{m}\alpha_{i}\tanh(\phi(x,c,d)\cdot w_{i})$.
The parameters of the model that we would learn are then the $m$ top-level
weights $\\{\alpha_{i}\\}$ and the $mF$ bottom-level weights $\\{w_{ij}\\}$;
recall that $F$ is the number of features ($\phi(x,c,d)\in\mathbb{R}^{F}$).
With more parameters, the model becomes more powerful, but also requires more
data to learn.

###  3.4 Parsing

Most semantic parsing algorithms are based on chart parsing, where we
recursively generate a set of candidate derivations for each syntactic
category (e.g., NP) and span (e.g., $[3\\!:\\!5]$). There are two
disadvantages of chart parsing: First, it does not easily support incremental
contextual interpretation: The features of a span $[i\\!:\\!j]$ can only
depend on the sub-derivations in that span, not on the derivations constructed
before $i$. This makes anaphora (resolution of “ _it_ ”) difficult to model. A
solution is to use shift-reduce parsing rather than chart parsing
(zhao2015type, ). Here, one parses the utterance from left to right, and new
sub-derivations can depend arbitrarily on the sub-derivations constructed thus
far.

A second problem is that the chart parsing generally builds derivations in
some fixed order—e.g., of increasing span size. This causes the parser to
waste equal resources on non-promising spans which are unlikely to participate
in the final derivation that is chosen. This motivates agenda-based parsing,
in which derivations that are deemed more promising by the model are built
first (berant2015agenda, ).

###  3.5 Learner

In learning, we are given examples of utterance-context-response triples
$(x,c,y)$. There are two aspects of learning: inducing the grammar rules and
estimating the model parameters. It is important to remember that practical
semantic parsers do not do everything from scratch, and often the hard-coded
grammar rules are as important as the training examples. First, some lexical
rules that map named entities (e.g., [_paris_
$\Rightarrow\texttt{ParisFrance}$]), dates, and numbers are generally assumed
to be given (zettlemoyer05ccg, ), though we need not assume that these rules
are perfect (liang11dcs, ). These rules are also often represented implicitly
(liang11dcs, ; berant2013freebase, ).

How the rest of the grammar is handled varies across approaches. In CCG-style
approach, inducing lexical rules is an important part of learning. In
zettlemoyer05ccg , a procedure called GENLEX is used to generate candidate
lexical rules from a utterance-logical form pair $(x,z)$. A more generic
induction algorithm based on higher-order unification does not require any
initial grammar (kwiatkowski10ccg, ). (wong07synchronous, ) use machine
translation ideas to induce a synchronous grammar (which can also be used to
generate utterances from logical forms). However, all these lexicon induction
methods require annotated logical forms $z$. In approaches that learn from
denotations $y$ (liang11dcs, ; berant2013freebase, ), an initial crude grammar
is used to generate candidate logical forms, and rest of the work is done by
the features.

As we discussed earlier, parameter estimation can be performed by stochastic
gradient descent on the log-likelihood; similar objectives based on max-margin
are also possible (liang2015semantics, ). It can be helpful to also add an
$L_{1}$ regularization term $\lambda\|\theta\|_{1}$, which encourages feature
weights to be zero, which produces a more compact model that generalizes
better (berant2014paraphrasing, ). In addition, one can use AdaGrad
(duchi10adagrad, ), which maintains a separate step size for each feature.
This can improve stability and convergence.

##  4 Datasets and Results

In a strong sense, datasets are the main driver of progress for statistical
approaches. We will now survey some of the existing datasets, describe their
properties, and discuss the state-of-the-art.

The Geo880 dataset (zelle96geoquery, ) drove nearly a decade of semantic
parsing research. This dataset consists of 880 questions and database queries
about US geography (e.g., “ _what is the highest point in the largest state?_
”). The utterances are compositional, but the language is clean and the domain
is limited. On this dataset, learning from logical forms
(kwiatkowski2013scaling, ) and answers (liang11dcs, ) both achieve around 90%

The ATIS-3 dataset (zettlemoyer07relaxed, ) consists of 5418 utterances paired
with logical forms (e.g., “ _show me information on american airlines from
fort worth texas to philadelphia_ ”). The utterances contain more disfluencies
and flexible word order compared with Geo880, but they are logically simpler.
As a result, slot filling methods have been a successful paradigm in the
spoken language understanding community for this domain since the 1990s. The
best reported result is based on semantic parsing and obtains 84.6%

The Regexp824 dataset (kushman2013regex, ) consists of 824 natural language
and regular expression pairs (e.g., “ _three letter word starting with ’X’_
”). The main challenge here is that there are many logically equivalent
regular expressions, some aligning better to the natural language than others.
(kushman2013regex, ) uses semantic unification to test for logical form
equivalence and obtains 65.6%

The Free917 dataset (cai2013large, ) consists of 917 examples of question-
logical form pairs that can be answered via Freebase, e.g. “ _how many works
did mozart dedicate to joseph haydn?_ ” The questions are logically less
complex than those in the semantic parsing datasets above, but introduces the
new challenge of scaling up to many more predicates (but is in practice
manageable by assuming perfect named entity resolution and leveraging the
strong type constraints in Freebase). The state-of-the-art accuracy is 68%

WebQuestions (berant2013freebase, ) is another dataset on Freebase consisting
of 5810 question-answer pairs (no logical forms) such as “ _what do australia
call their money?_ ” Like Free917, the questions are not very compositional,
but unlike Free917, they are real questions asked by people on the Web
independent from Freebase, so they are more realistic and more varied. Because
the answers are required to come from a single Freebase page, a noticeable
fraction of the answers are imperfect. The current state-of-the-art is 52.5%

The goal of WikiTableQuestions (pasupat2015compositional, ) is to extend
question answering beyond Freebase to HTML tables on Wikipedia, which are
semi-structured. The dataset consists of 22033 question-table-answer triples
(e.g., “ _how many runners took 2 minutes at the most to run 1500 meters?_ ”),
where each question can be answered by aggregating information across the
table. At test time, we are given new tables, so methods must learn how to
generalize to new predicates. The result on this new dataset is 37.1%

wang2015overnight proposed a new recipe for quickly using crowdsourcing to
generate new compositional semantic parsing datasets consisting of question-
logical form pairs. Using this recipe, they created eight new datasets in
small domains consisting of 12602 total question-answer pairs, and achieved an
average accuracy on across datasets of 58.8%

(chen11navigate, ) introduced a dataset of 706 navigation instructions (e.g.,
“ _facing the lamp go until you reach a chair_ ”) in a simple grid world. Each
instruction sequence contains multiple sentences with various imperative and
context-dependent constructions not found in previous datasets.
artzi2013weakly obtained 65.3%

##  5 Discussion

We have presented a semantic parsing framework for the problem of natural
language understanding. Going forward, the two big questions are (i) how to
represent the semantics of language and (ii) what supervision to use to learn
the semantics from.

Alternative semantic representations. One of the main difficulties with
semantic parsing is the divergence between the structure of the natural
language and the logical forms—purely compositional semantics will not work.
This has led to some efforts to introduce an intermediate layer between
utterances and logical forms. One idea is to use general paraphrasing models
to map input utterances to the “canonical utterances” of logical forms
(berant2014paraphrasing, ; wang2015overnight, ). This reduces semantic parsing
to a text-only problem for which there is much more data and resources.

One could also use domain-general logical forms that capture the basic
predicate-argument structures of sentences (kwiatkowski2013scaling, ).
Abstract meaning representation (AMR) (banarescu2013amr, ) is one popular
representation backed by an extension linguistic annotation effort. Multiple
AMR parsers have been developed, including one based on CCG (artzi2015broad,
). While these representations offer broad coverage, solving downstream
understanding tasks still require additional work (e.g., inferring that “
_population of X_ ” is synonymous with “ _number of people living in X_ ”). In
contrast, _executable_ semantic parsing operates on the full pipeline from
utterance to task output, but compromises on domain-generality. This is
partially inevitable, as any understanding task requires some domain-specific
knowledge or grounding. Designing the best general representation that
supports many downstream tasks remains an open challenge.

Alternative supervision. Much of the progress in semantic parsing has been due
to being able to learn from weaker supervision. In the framework we presented,
this supervision are the desired actions $y$ (e.g., answers to questions). One
can use a large corpus of text to exploit even weaker supervision
(krishnamurthy2012weakly, ; reddy2014large, ). More generally, one can think
about language interpretation in a reinforcement learning setting
(branavan09reinforcement, ), where an agent who presented with an utterance in
some context performs some action, and receives a corresponding reward signal.
This framework highlights the importance of context-dependence in language
interpretation (zettlemoyer09context, ; artzi2013weakly, ).

Due to their empirical success, there has been a recent surge of interest in
using recurrent neural networks and their extensions for solving NLP tasks
such as machine translation and question answering (yih2015stagg, ;
weston2015memory, ). These methods share the same spirit of end-to-end
utterance-to-behavior learning as executable semantic parsing, but they do not
explicitly separate parsing from execution. This makes them architecturally
simpler than semantic parsers, but they are more data hungry and it is unclear
whether they can learn to perform complex logical reasoning in a generalizable
way. Nonetheless, it is quite likely that these methods will play an important
role in the story of language understanding.

Outlook. This is an exciting time for semantic parsing and natural language
understanding. As natural language interfaces (e.g., Siri) become more
ubiquitous, the demand for deep understanding will continue to grow. At the
same time, there is a fresh wave of ambition pushing the limits of what can be
machine learnable. The confluence of these two factors will likely generate
both end-user impact as well as new insights into the nature of language and
learning.

## References

  * [1] Y. Artzi and K. L. L. Zettlemoyer.  Broad-coverage CCG semantic parsing with AMR.  In Empirical Methods in Natural Language Processing (EMNLP), 2015\.
  * [2] Y. Artzi and L. Zettlemoyer.  Weakly supervised learning of semantic parsers for mapping instructions to actions.  Transactions of the Association for Computational Linguistics (TACL), 1:49–62, 2013.
  * [3] L. BanaRescu, C. B. S. Cai, M. Georgescu, K. Griffitt, U. Hermjakob, K. Knight, P. Koehn, M. Palmer, and N. Schneider.  Abstract meaning representation for sembanking.  In 7th Linguistic Annotation Workshop and Interoperability with Discourse, 2013.
  * [4] J. Berant, A. Chou, R. Frostig, and P. Liang.  Semantic parsing on Freebase from question-answer pairs.  In Empirical Methods in Natural Language Processing (EMNLP), 2013\.
  * [5] J. Berant and P. Liang.  Semantic parsing via paraphrasing.  In Association for Computational Linguistics (ACL), 2014.
  * [6] J. Berant and P. Liang.  Imitation learning of agenda-based semantic parsers.  Transactions of the Association for Computational Linguistics (TACL), 0, 2015.
  * [7] P. Blackburn and J. Bos.  Representation and Inference for Natural Language: A First Course in Computational Semantics.  CSLI Publishers, 2005.
  * [8] K. Bollacker, C. Evans, P. Paritosh, T. Sturge, and J. Taylor.  Freebase: a collaboratively created graph database for structuring human knowledge.  In International Conference on Management of Data (SIGMOD), pages 1247–1250, 2008.
  * [9] S. Branavan, H. Chen, L. S. Zettlemoyer, and R. Barzilay.  Reinforcement learning for mapping instructions to actions.  In Association for Computational Linguistics and International Joint Conference on Natural Language Processing (ACL-IJCNLP), pages 82–90, 2009\.
  * [10] E. Brill, S. Dumais, and M. Banko.  An analysis of the AskMSR question-answering system.  In Association for Computational Linguistics (ACL), pages 257–264, 2002.
  * [11] Q. Cai and A. Yates.  Large-scale semantic parsing via schema matching and lexicon extension.  In Association for Computational Linguistics (ACL), 2013.
  * [12] D. L. Chen and R. J. Mooney.  Learning to interpret natural language navigation instructions from observations.  In Association for the Advancement of Artificial Intelligence (AAAI), pages 859–865, 2011.
  * [13] J. Clarke, D. Goldwasser, M. Chang, and D. Roth.  Driving semantic parsing from the world’s response.  In Computational Natural Language Learning (CoNLL), pages 18–27, 2010.
  * [14] J. Duchi, E. Hazan, and Y. Singer.  Adaptive subgradient methods for online learning and stochastic optimization.  In Conference on Learning Theory (COLT), 2010.
  * [15] J. Krishnamurthy and T. Kollar.  Jointly learning to parse and perceive: Connecting natural language to the physical world.  Transactions of the Association for Computational Linguistics (TACL), 1:193–206, 2013.
  * [16] J. Krishnamurthy and T. Mitchell.  Weakly supervised training of semantic parsers.  In Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP/CoNLL), pages 754–765, 2012.
  * [17] N. Kushman and R. Barzilay.  Using semantic unification to generate regular expressions from natural language.  In Human Language Technology and North American Association for Computational Linguistics (HLT/NAACL), pages 826–836, 2013.
  * [18] T. Kwiatkowski, E. Choi, Y. Artzi, and L. Zettlemoyer.  Scaling semantic parsers with on-the-fly ontology matching.  In Empirical Methods in Natural Language Processing (EMNLP), 2013\.
  * [19] T. Kwiatkowski, L. Zettlemoyer, S. Goldwater, and M. Steedman.  Inducing probabilistic CCG grammars from logical form with higher-order unification.  In Empirical Methods in Natural Language Processing (EMNLP), pages 1223–1233, 2010.
  * [20] P. Liang.  Lambda dependency-based compositional semantics.  arXiv, 2013.
  * [21] P. Liang, M. I. Jordan, and D. Klein.  Learning dependency-based compositional semantics.  In Association for Computational Linguistics (ACL), pages 590–599, 2011.
  * [22] P. Liang and C. Potts.  Bringing machine learning and compositional semantics together.  Annual Reviews of Linguistics, 1(1):355–376, 2015.
  * [23] C. Matuszek, N. FitzGerald, L. Zettlemoyer, L. Bo, and D. Fox.  A joint model of language and perception for grounded attribute learning.  In International Conference on Machine Learning (ICML), pages 1671–1678, 2012.
  * [24] S. Miller, D. Stallard, R. Bobrow, and R. Schwartz.  A fully statistical approach to natural language interfaces.  In Association for Computational Linguistics (ACL), pages 55–61, 1996.
  * [25] R. Montague.  The proper treatment of quantification in ordinary English.  In Approaches to Natural Language, pages 221–242, 1973.
  * [26] P. Pasupat and P. Liang.  Compositional semantic parsing on semi-structured tables.  In Association for Computational Linguistics (ACL), 2015.
  * [27] S. Reddy, M. Lapata, and M. Steedman.  Large-scale semantic parsing without question-answer pairs.  Transactions of the Association for Computational Linguistics (TACL), 2(10):377–392, 2014.
  * [28] M. Steedman.  The Syntactic Process.  MIT Press, 2000.
  * [29] S. Tellex, T. Kollar, S. Dickerson, M. R. Walter, A. G. Banerjee, S. J. Teller, and N. Roy.  Understanding natural language commands for robotic navigation and mobile manipulation.  In Association for the Advancement of Artificial Intelligence (AAAI), 2011.
  * [30] Y. Wang, J. Berant, and P. Liang.  Building a semantic parser overnight.  In Association for Computational Linguistics (ACL), 2015.
  * [31] J. Weston, S. Chopra, and A. Bordes.  Memory networks.  In International Conference on Learning Representations (ICLR), 2015\.
  * [32] T. Winograd.  Understanding Natural Language.  Academic Press, 1972.
  * [33] Y. W. Wong and R. J. Mooney.  Learning synchronous grammars for semantic parsing with lambda calculus.  In Association for Computational Linguistics (ACL), pages 960–967, 2007.
  * [34] W. A. Woods, R. M. Kaplan, and B. N. Webber.  The lunar sciences natural language information system: Final report.  Technical report, BBN Report 2378, Bolt Beranek and Newman Inc., 1972\.
  * [35] W. Yih, M. Chang, X. He, and J. Gao.  Semantic parsing via staged query graph generation: Question answering with knowledge base.  In Association for Computational Linguistics (ACL), 2015.
  * [36] M. Zelle and R. J. Mooney.  Learning to parse database queries using inductive logic programming.  In Association for the Advancement of Artificial Intelligence (AAAI), pages 1050–1055, 1996.
  * [37] L. S. Zettlemoyer and M. Collins.  Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars.  In Uncertainty in Artificial Intelligence (UAI), pages 658–666, 2005.
  * [38] L. S. Zettlemoyer and M. Collins.  Online learning of relaxed CCG grammars for parsing to logical form.  In Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP/CoNLL), pages 678–687, 2007.
  * [39] L. S. Zettlemoyer and M. Collins.  Learning context-dependent mappings from sentences to logical form.  In Association for Computational Linguistics and International Joint Conference on Natural Language Processing (ACL-IJCNLP), 2009.
  * [40] K. Zhao and L. Huang.  Type-driven incremental semantic parsing with polymorphism.  In North American Association for Computational Linguistics (NAACL), 2015.


# Bridging the Gap: Transfer Learning from English PLMs to Malaysian English

Mohan Raj1, Lay-Ki Soon1∗, Ong Huey Fang1, and Bhawani Selvaretnam2
1School of Information Technology, Monash University Malaysia
{mohan.chanthran, soon.layki<EMAIL_ADDRESS>
2Valiantlytix
<EMAIL_ADDRESS>

###### Abstract

Malaysian English is a low resource creole language, where it carries the
elements of Malay, Chinese, and Tamil languages, in addition to Standard
English. Named Entity Recognition (NER) models underperform when capturing
entities from Malaysian English text due to its distinctive morphosyntactic
adaptations, semantic features and code-switching (mixing English and Malay).
Considering these gaps, we introduce MENmBERT and MENBERT, a pre-trained
language model with contextual understanding, specifically tailored for
Malaysian English. We have fine-tuned MENmBERT and MENBERT using manually
annotated entities and relations from the Malaysian English News Article (MEN)
Dataset. This fine-tuning process allows the PLM to learn representations that
capture the nuances of Malaysian English relevant for NER and RE tasks.
MENmBERT achieved a 1.52% and 26.27% improvement on NER and RE tasks
respectively compared to the bert-base-multilingual-cased model. Although the
overall performance of NER does not have a significant improvement, our
further analysis shows that there is a significant improvement when evaluated
by the 12 entity labels. These findings suggest that pre-training language
models on language-specific and geographically-focused corpora can be a
promising approach for improving NER performance in low-resource settings. The
dataset and code published in this paper provide valuable resources for NLP
research work focusing on Malaysian English.

**footnotetext: Corresponding Author.

Bridging the Gap: Transfer Learning from English PLMs to Malaysian English

Mohan Raj1, Lay-Ki Soon1∗, Ong Huey Fang1, and Bhawani Selvaretnam2 1School of
Information Technology, Monash University Malaysia {mohan.chanthran,
soon.layki<EMAIL_ADDRESS>2Valiantlytix<EMAIL_ADDRESS>

##  1 Introduction

With the recent proliferation of Large Language Models (LLMs), the usage of
Pre-trained Language Models (PLMs) like BERT (Devlin et al., 2019), RoBERTa
(Liu et al., 2019), XLM-RoBERTa (Conneau et al., 2020), ALBERT (Lan et al.,
2020) has been overshadowed. However, PLM has shown some significant
improvements when further pre-trained in domain specific (Chalkidis et al.,
2020; Lee et al., 2019; Araci, 2019; Huang et al., 2020) or language specific
corpus (Martin et al., 2020; Chan et al., 2020; Antoun et al., 2021; Vamvas et
al., 2023), and subsequently fine-tuned on NLP tasks. The adaptability of pre-
trained language model to diverse languages and dialects has enabled its
application to specific linguistic contexts, including Malaysian English, a
unique and culturally rich variant of the English language.

Malaysian English is also categorized as a creole language due to its distinct
characteristics that include loanwords, compound words, and the derivation of
new terms from Malay, Chinese, and Tamil, in addition to the Standard English
(Chanthran et al., 2024). Existing state-of-the-art (SOTA) solutions do not
produce satisfactory outcomes for downstream tasks performed in Malaysian
English (Chanthran et al., 2024). This is mainly due to the morphosyntactic
and semantical adaption nature of Malaysian English. Hence, there is a need to
improve this SOTA in order to support effective processing of Malaysian
English texts.

This work investigates the effectiveness of English pre-trained language
models (PLMs) to the low resource language like Malaysian English, in
downstream task, particularly Named Entity Recognition (NER), and Relation
Extraction (RE) performed on Malaysian English. Our findings contribute to the
growing body of research on promoting inclusivity in NLP by exploring the
applicability of PLMs to non-standard English varieties.

The contributions of this paper are as follows:

  1. 1.

Multilingual Pre-trained Model for Malaysian English: We introduce MENmBERT, a
BERT-based model pre-trained on a Malaysian English News (MEN) Corpus. MEN
Corpus comprises 14,320 articles, facilitating research on applying PLMs to
Malaysian English (Chanthran et al., 2024). This model will be made public to
develop resources and facilitate NLP research in Malaysian English. The code
for this experiment, and dataset have been published in
https://github.com/mohanraj-nlp/MEN-Dataset/tree/pretrained-lm.

  2. 2.

Fine-Tuned and Evaluated NER and RE on Malaysian English: We evaluated the
effectiveness of fine-tuned MENmBERT for NER and RE tasks on a benchmark MEN-
Dataset. MEN-Dataset contains 200 news articles with 6,061 annotated entities
and 4,095 relation instances (Chanthran et al., 2024). This analysis
demonstrates the applicability of transfer learning from English PLMs to
Malaysian English NLP tasks.

This paper is structured as follows. Section 2 explores the utilization of
pre-trained language models in both English and non-English scenarios. In
Section 3, we dive deeper into the pre-training and fine-tune methodologies of
MENmBERT and MENBERT. Section 4 we will discuss on the results of fine-tuned
NER and RE. Finally, Section 5 concludes the work and shares potential
enhancement as the future work.

##  2 Related Work

###  2.1 Pre-Trained Language Model for Non-English Context

Language-specific Pre-trained Language Models (PLMs) are essential to handle
complex languages and improve performance on downstream NLP task specific to
particular language. Considering this, AraBERT has been proposed to address
the morphological and syntactic differences in the Arabic language compared to
other languages, as Arabic shares very little with Latin-based languages and
has unique characteristics (Antoun et al., 2021). Multilingual models to learn
representations for multiple languages simultaneously resulted in little data
representation and small language-specific vocabulary for Arabic, hindering
performance compared to a single-language model. (Antoun et al., 2021)
overcome these challenges, the researchers pre-trained AraBERT specifically
for the Arabic language to capture the contextualized representations needed
for Arabic NLP tasks. By customizing the model for Arabic and optimizing
factors such as data size, vocabulary size, and pre-processing techniques,
AraBERT was able to achieve state-of-the-art performance on various Arabic NLP
tasks. The pre-training has been completed with 70 million sentences and
around 24GB of textual Arabic data. AraBERT performs better in downstream
tasks like Sentiment Analysis, Named Entity Recognition (NER), and Question
Answering (Antoun et al., 2021). Antoun et al. (2021) compared the AraBERT
fine-tuned model to SOTA and M-BERT, the results shows AraBERT performing
better than mBERT or SOTA.

Following the success of AraBERT in Arabic NLP, similar approaches can be
applied to other low-resource languages with unique characteristics. One such
example is KinyaBERT, a recent model specifically designed to address the
challenges of Natural Language Processing (NLP) tasks in Kinyarwanda
(Nzeyimana and Niyongabo Rubungo, 2022). KinyaBERT has been implemented with a
two-tier BERT architecture that token-level morphology encoder and
sentence/document level encoder. By dividing the model’s processing into these
two tiers, KinyaBERT aims to effectively capture both the fine-grained
morphological details of individual tokens and the broader contextual
information present in the input text. The pre-training task has been
completed with 16 million and 2.4GB of Kinyarwanda language texts. KinyaBERT
is evaluated on NLP downstream tasks such as NER, News Categorization Task
(NEWS) and Machine-Translated GLUE Benchmark. From the evaluation, KinyaBERT
has outperformed the baseline model like BERT Base Pre-trained on Kinyarwanda
Corpus (BERT BPE), BERT Tokenized by Morphological Analyzer (BERT MORPHO) and
XLM-R.

Similarly to AraBERT and KinyaBERT, SwissBERT has been proposed specifically
for national languages of Switzerland (Vamvas et al., 2023). SwissBERT trained
using a combination of domain adaptation, language adaptation, and
multilingual approaches. SwissBERT has been fine-tuned for NER task and it was
able to outperform the baseline model which has been further pre-trained.
SwissBERT has undergone several key adaptations and additions to make it
impact for processing Switzerland-related text, this includes:

  1. 1.

Multilingual Adaptation: SwissBERT is trained on a corpus of more than 21
million Swiss news articles in the national languages of Switzerland,
including German, French, Italian, and Romansh Grischun.

  2. 2.

Custom Language Adapters: SwissBERT utilizes custom language adapters in each
layer of the transformer encoder for the four national languages of
Switzerland.

  3. 3.

Switzerland-Specific Subword Vocabulary: To further enhance its performance on
Switzerland-related text, SwissBERT is equipped with a Switzerland-specific
subword vocabulary.

SwissBERT’s superior performance in Switzerland-related tasks, such as Named
Entity Recognition and Stance Detection, highlights its efficacy in handling
diverse linguistic content specific to Switzerland.

Devlin et al. (2019) has further pre-trained multilingual BERT (M-BERT) with
104 languages Wikipedia corpus. Wang et al. (2020) suggests enhancing M-BERT
by pre-training with low-resource corpora, as 11 languages are not covered in
the current 104 languages. The research found that fine-tuning M-BERT
(E-MBERT) on low-resource language corpora enhanced NER task performance. Wu
and Dredze (2020) found that multilingual BERT (mBERT in Wu and Dredze (2020))
may not perform well in low-resource languages. This inconsistency may be
attributed to the fact that mBERT is trained with a multitude of languages.
Pre-training mBERT for a low resource language model can negatively impact the
performance compared to training a monolingual BERT model for that language.
Findings in related works have inspired us to establish better evaluation and
model selection criteria for the development of a pre-trained language model
for Malaysian English.

##  3 MENmBERT and MENBERT

###  3.1 Overview

Chalkidis et al. (2020) discussed about two possible further pre-training
strategies:

  1. 1.

Continued / Further Pre-training (FP): FP creates domain- or language-specific
BERT models. FP utilises pre-trained model parameters, saving time and data
(Kalyan et al., 2021).

  2. 2.

Pre-Training from Scratch (SC): Pre-training from scratch lets you train the
model with a lot of data. Pre-training the model from scratch will utilise
existing language model architecture and parameters (Kalyan et al., 2021).

The difference between two approach is, FP has been pre-trained with generic
corpora like BookCorpus, and English Wikipedia (Devlin et al., 2019) while SC
has not been pre-trained with any corpus. With this in mind, we proposed to
explore several further pre-training strategies:

  1. 1.

MENBERT-FP: Further pre-train bert-base-cased model with MEN-Corpus

  2. 2.

MENmBERT-FP: Further pre-train bert-base-multilingual-cased model with MEN-
Corpus

  3. 3.

MENBERT-SC: We pre-train bert-base-cased from scratch with MEN-Corpus

Malaysian English features loan words, compound blend and derivations of new
terms from multiple languages local language, making multilingual BERT an
effective model for understanding the contexts of news articles. The need to
further pre-train BERT, mBERT and train BERT from scratch stems from our
hypothesis that PLM with rich language-based understanding will improve the
performance of NER and RE after being fine-tuned. Section 3.2 presents the
pre-training setup and hyperparameters used, while Section 3.3 explains the
model fine-tuning tasks.

###  3.2 Further Pre-Training MENmBERT and MENBERT

Python library Transformers (Wolf et al., 2020) has been used to pre-train
BERT. For MENmBERT and MENBERT we have used bert-base-multilingual-cased and
bert-base-cased respectively. Since MENBERT-SC was trained from scratch, we
used bert-base-cased architecture. We generated our own vocabulary using
BertWordPieceTokenizer for MENBERT-SC, meanwhile for MENmBERT and MENBERT we
have used BertTokenizer. We selected the hyperparameter combination with the
lowest training loss. Table 1 lists the important hyperparameters used to
train the models. Section 4 details the pre-training results and some
analyses.

Hyperparameters | MENBERT-FP | MENmBERT-FP | MENBERT-SC
---|---|---|---
epoch | 30 | 30 | 30
batch_size | 32 | 16 | 32
learning_rate | 5e-5 | 5e-5 | 5e-5
weight_decay | 0.001 | 0.001 | 0.001
max_sequence_length | 512 | 512 | 512

Table 1: Hyperparameters used to train MENBERT-FP, MENmBERT-FP, MENBERT-SC

###  3.3 Fine-Tuning MENmBERT and MENBERT

Fine-Tune is an adaptation method to train pre-trained model for any NLP
downstream task (Kalyan et al., 2021). Three pre-trained model MENBERT-FP,
MENmBERT-FP, and MENBERT-SC were fine-tuned for NER and RE using MEN-Dataset.
Additionally, we also fine-tuned pre-trained models bert-base-cased and bert-
base-multilingual-cased. Fine-tuning on pre-trained and further pre-trained
models helps us to compare the performance and validate our hypothesis (see
Section 3.1).

####  3.3.1 Named Entity Recognition

We used the Python library Transformers Wolf et al. (2020), specifically the
BertForTokenClassification module, for fine-tuning. As suggested by Devlin et
al. (2019), we went through hyperparameter optimization to find an optimal
hyperparameter for fine-tuning. We leveraged on WandB (Biewald, 2020) for
hyperparameter optimization and logging. We used [2e-5, 5e-5] for the
learning_rate, [10, 20, 30] for num_train_epochs, [0.01, 0.001, 0.0001] for
weight_decay, and finally [4, 8, 16] for the per_device_train_batch_size.

We used a grid-based search approach to find an optimal hyperparameter with
maximum F1-Score and minimum evaluation loss. Table 2 provides the
hyperparameters used to fine-tune for NER. The MEN-Dataset is split into
training (75%), test (10%) and validation (15%), with total entities of 5065,
453 and 618 respectively. Models fine-tuned with optimal hyperparameter were
evaluated using the validation set, and discussed in the following Section
4.2.1.

Hyperparameters | epoch | batch_size | learning_rate | weight_decay
---|---|---|---|---
bert-based-cased | 20 | 4 | 5e-5 | 0.0001
MENBERT-FP | 30 | 4 | 5e-5 | 0.01
mbert-based-cased | 30 | 4 | 5e-5 | 0.01
MENmBERT-FP | 30 | 4 | 5e-5 | 0.01
MENBERT-SC | 30 | 4 | 5e-5 | 0.01

Table 2: Optimal Hyperparameters used to fine-tune bert-base-cased, bert-base-
multilingual-cased, MENBERT-FP, MENmBERT-FP, MENBERT-SC for NER Figure 1:
Precision, recall, and F1-score calculated for NER on the MEN-Dataset
validation set.

####  3.3.2 Relation Extraction

Hyperparameters | epoch | batch_size | learning_rate | weight_decay
---|---|---|---|---
bert-based-cased | 30 | 4 | 5e-5 | 0.1
MENBERT-FP | 30 | 4 | 5e-5 | 0.1
mbert-based-cased | 30 | 4 | 5e-5 | 0.1
MENmBERT-FP | 30 | 4 | 5e-5 | 0.1
MENBERT-SC | 30 | 4 | 5e-5 | 0.1

Table 3: Optimal Hyperparameters used to fine-tune bert-base-cased, bert-base-
multilingual-cased, MENBERT-FP, MENmBERT-FP, MENBERT-SC for RE

To efficiently fine-tune PLM for RE on the MEN-Dataset, we leveraged existing
fine-tuning code for document-level relation extraction111DocRED_Bert Github
Link. We then carefully modified this code to accommodate the specific
characteristics and labeling scheme of the MEN-Dataset. Similarly like NER we
went through hyperparameter optimization to find an optimal hyperparameter for
fine-tuning. We used [2e-5, 5e-5] for the learning_rate, [10, 20, 30] for
num_train_epochs, [0.01, 0.001, 0.0001] for weight_decay, and finally [4, 8,
16] for the per_device_train_batch_size. Table 3 provides the hyperparameters
used to fine-tune for RE.

MEN-Dataset has relation labels adapted from prominent RE dataset like DocRED
(Yao et al., 2019) and ACE-2005 (Walker, 2005). There are 84 relation labels
that are adapted from DocRED, and 16 relation labels from ACE-2005. For this
study, we concentrated on the relation labels originating from the DocRED
dataset. One of the reason for our decision are due to Label Distribution. The
DocRED labels constitute the majority within the MEN-Dataset. Focusing on
these prevalent labels allows the model to learn robust representations for
the most frequently occurring relation types, leading to potentially better
performance on tasks involving these relations. Apart from that, we have also
excluded a special relation label "NO_RELATION", as they are used is to
indicate entities that might have a relation but not captured by the
predefined relation set (Chanthran et al., 2024). Including "NO_RELATION"
could introduce noise or ambiguity during model training.

MEN-Dataset has a total of 2,237 relation instances adapted from DocRED
relation labels, distributed across training (1,693), testing (267), and
validation (277) sets. To ensure a comprehensive representation of relation
labels during training, we employed a stratified sampling approach on the MEN-
Dataset. While the original split allocates 75%, 10%, and 15% for training,
validation, and testing, respectively, our stratified sampling guarantees that
all relation labels are present in the training data. The result and analysis
of fine-tuned PLM for RE have been discussed in Section 4.2.2.

##  4 Experiment Result and Analysis

###  4.1 Further Pre-trained Language Model

Masked Sentence | Masked Token | Pretrained Model
---|---|---
| bert-based
---
-cased
| mbert-based
---
-cased
menbert-fp | menbert-sc | menmbert-fp
| There are three levels of disaster management,
---
the first involves a locality in a district,
secondly when more than two districts of a
state is involved and the third involves
two or three states. So everyone is aware of this,”
he told a press conference at <MASK> Sri
Muda today.
Taman | the | Sri | the | ’ | Taman
| On the Perlindungan Tenang Voucher,
---
he said all eight million recipients
of <MASK> Prihatin Rakyat are eligible to
receive the voucher worth RM50 announced
in Budget 2021 for the benefit of
the B40 group.
Bantuan | the | the | the | the | Anugerah
| Ismail Sabri also hoped that ties between
---
Umno and PAS in Bera would remain
strong and despite the harsh statement issued
by the top leaders of the two parties, priority
should be given to unite the Malay
Muslims and <MASK>.
Bumiputera | Christians | Muslims | Muslims | , | Muslims
| The Ministry of <MASK> Territories
---
(KWP) while ensuring the flood
management in Kuala Lumpur is
proceeding well.
Federal | New | Protected | Federal | ##am | Federal
| Hamzah said he had discussed the issue with
---
Inspector-General of Police Tan Sri
Acryl Sani <MASK> Sani.
Abdullah | - | . | Abdullah | ser | Abdullah
| Hamzah also said police had set up
---
a Tactical Command Centre in <MASK>
Langat district in Selangor to coordinate
flood relief operations of all units.
Hulu | the | the | Kuala | , | Hulu

Table 4: Some sentences from the MEN-Dataset were used to predict masked
tokens using various PLMs. Bold tokens indicate correctly predicted tokens
when compared to the ground truth.

(Salazar et al., 2020; Kauf and Ivanova, 2023) suggests a "pseudo-log-
likelihood" score calculated by masking tokens individually. The score is
computed by summing the log-losses at the different masked positions. However,
we are more interested in how accurately the models predict the masked token.
This will help us to understand PLM models understanding and contextual
awareness. We have collected 100 sentences from Malaysian English news article
platform and we have done validation to ensure those sentence not part of our
pre-training MEN-Corpus. In the first 70 sentences, one token from the local
language, such as Bahasa Malaysia, has been randomly masked. In the remaining
30 sentences, one token of Standard English has been masked. We employ
accuracy metrics to assess each model’s effectiveness, providing a clear
differentiation in their performance. The results demonstrate that additional
pretraining on language-specific data significantly enhances the models’
predictive capabilities, underscoring the importance of tailored training for
improved language understanding.

Figure 2: Accuracy of different pre-trained model predicting masked tokens in
50 Malaysian English Sentence.

For each pre-trained model, we calculated the accuracy of correctly predicted
masked tokens. We used bert-based-cased and bert-base-multilingual-cased as
baseline to investigate the improvement made by further pre-trained model.
Based on the result in Figure 2, we have identified that MENmBERT-FP has
highest accuracy on predicting masked token from Malaysian English sentence.
In Table 4, we present some sample of sentences showcasing how different PLMs
perform in predicting masked tokens. Here are the findings obtained from the
sample result shown in Table 4:

  1. 1.

MENmBERT-FP: Even though the model did not predict the exact ground-truth
token, in some cases it identified semantically similar tokens. For instance,
in the Malaysian context, Bumiputera often refers to the Muslim community.
Here, MENmBERT-FP might predict a token related to ethnicity but not strictly
synonymous with Muslim. This highlights the model’s ability to capture
semantic nuances, even when encountering challenging cases.

  2. 2.

MENBERT-FP: When we compared the performance of MENBERT-FP with bert-based-
cased, we can understand that further-pretraining has improved the performance
of language model. The success ratio of bert-based-cased is 0, and once
further pre-trained, there is an improvement of +33%.

  3. 3.

bert-based-cased: Since bert-based-cased has only been trained with English
corpus, it was not able to unmask any tokens with compound blend correctly.

  4. 4.

MENBERT-SC: Based on our observation, MENBERT-SC has produced bad results when
unmasking the tokens. Once fine-tune, we will be able to understand better on
the performance of the model.

In Section 4.2 we have discussed the performance of pre-trained model once
fine-tune them for NER and RE.

Entity Label |  | Total Annotated
---
Entity in
MEN-Dataset
| Total Annotated
---
Entity in
Validation Set
bert-based-cased | mbert-based-cased | menbert-fp | menbert-sc | menmbert-fp
PERSON | 1646 | 108 | 0.74 | 0.84 | 0.79 | 0.17 | 0.86
LOCATION | 1157 | 150 | 0.86 | 0.88 | 0.87 | 0.48 | 0.91
ORGANIZATION | 1624 | 262 | 0.81 | 0.89 | 0.82 | 0.29 | 0.89
EVENT | 386 | 30 | 0.67 | 0.61 | 0.66 | 0.13 | 0.77
PRODUCT | 72 | 6 | 0.24 | 0.33 | 0.13 | 0 | 0.07
FACILITY | 208 | 27 | 0.24 | 0.11 | 0.47 | 0 | 0.25
ROLE | 485 | 35 | 0.39 | 0.4 | 0.37 | 0.35 | 0.6
NORP | 114 | 5 | 0.88 | 0.6 | 0.89 | 0.21 | 0.57
TITLE | 300 | 4 | 0.55 | 0 | 0.43 | 0.18 | 0.5
LAW | 62 | 5 | 0.15 | 0.12 | 0.17 | 0.1 | 0.13
LANGUAGE | 0 | 0 | 0 | 0 | 0 | 0 | 0
WORK_OF_ART | 7 | 2 | 0.1 | 0.15 | 0.12 | 0 | 0.16
|  |  |  |  |  |  |
Total Entities | 6061 | 634 |
| Overall
---
Micro F1-Score
| 0.763 | 0.819 | 0.778 | 0.316 | 0.831

Table 5: Fine-Tuned model performance (based on F1-Score) calculated based on
validation set for each entity labels.  Figure 3: Precision, recall, and
F1-score calculated for RE on the MEN-Dataset validation set.

###  4.2 Fine-Tuning Pre-Trained Model

####  4.2.1 Named Entity Recognition

Figure 1 shows the comparison of Precision, Recall, and F1-Score among five
different pre-trained models. To evaluate the performance of further pre-
trained models, we also fine-tuned pre-trained model (bert-base-cased, and
bert-base-multilingual-cased) as the baseline. Meanwhile in Table 5, we
detailed the performance of model by entity labels.

Referring to the results, we observe that MENmBERT-FP achieves the highest
F1-Score (0.831), while MENBERT-SC obtains the lowest F1-Score (0.316).
Nevertheless, Figure 1 demonstrates an improvement when further pre-training
the BERT model, which validated our hypothesis (discussed in Section 3.1). A
few other observations from this experiment:

  1. 1.

MENmBERT-FP has a higher F1-Score (0.831) than mBERT-base-cased (0.819). We
observe a +1.52% improvement. Although the improvement is not significant, but
when we analyse the F1-Score based on the entity label:

    1. (a)

There is an significant improvement for 6 out of 11 entity labels that are
evaluated.

    2. (b)

On average, the difference in F1-Score between MENmBERT-FP and mBERT-base-
cased is +10%.

    3. (c)

mBERT-base-cased is only able to achieve on-par in terms of F1-Score with
MENmBERT-FP, specifically for entity label ORGANIZATION.

This in-depth observation proves the significant performance of MENmBERT-FP.

  2. 2.

MENBERT-FP (F1-Score is 0.778) has a higher F1-Score than bert-based-cased
(F1-Score is 0.763), with an improvement of +1.94%. However, after further
investigated the performance based on entity labels:

    1. (a)

MENBERT-FP has only improved the performance of only 7 out of 11 entity
labels.

    2. (b)

On average, the improvement is only around +2%.

    3. (c)

When compared with MENmBERT-FP, MENBERT-FP is able to outperform in terms of
F1-Score for 4 out of 11 entity labels. These include entity labels PRODUCT,
FACILITY, NORP and LAW.

  3. 3.

MENBERT-SC has not shown any improvement in the performance of NER. Our
evaluation in Section 4.1 also shows it was not able to unmask the tokens
correctly.

The experimental results and findings conclude that our fine-tuned MENmBERT-FP
has achieved highest F1-Score compared to other pre-trained models. MENmBERT-
FP could be used to fine-tune for more NLP downstream tasks, involving
Malaysian English, for improved performance.

####  4.2.2 Relation Extraction

Figure 3 shows F1-Scores compared across five PLMs. Like with NER, we used
fine-tuned bert-base-cased and bert-base-multilingual-cased as baselines.

MENmBERT-FP achieved the highest F1-score (0.353), indicating a slight
improvement over our baseline PLMs. This suggests that further pre-training on
MEN-Dataset has been beneficial for RE on Malaysian English context. Here are
some of ur observations from the experiment:

  1. 1.

MENmBERT-FP has made +33.21% improvement in F1-Score compared to baseline
approach mBERT-base-cased. This has been proven significant. Our cross-
analysis of NER and RE predictions reveals that the TP relation instances have
entity pairs correctly classified by the fine-tuned NER model (from previous
analysis). This suggests a significant improvement in RE performance due to
the model’s enhanced entity prediction capabilities.

  2. 2.

MENBERT-FP (F1-Score is 0.3105) has a higher F1-Score than bert-based-cased
(F1-Score is 0.2602), with an improvement of +19.33%. The analysis of the
fine-tuned model’s predictions did not reveal any surprising or unexpected
patterns. This aligns with the observations from the previous point.
Meanwhile, for MENBERT-SC was performed badly when fine-tuned for RE task.

Apart from that, it’s important to note that our fine-tuned RE models achieved
lower performance compared to reported results on other document-level
relation extraction datasets like DocRED. For instance, prior work using a
fine-tuned BERT model on DocRED (38,269 relation instances) achieved an
F1-score of 54.16 (Dev) and 53.20 (Test) (Wang et al., 2019). Compared with
our finding, the overall F1-Score could be not significant due to the nature
of MEN-Dataset with a small set of annotation instance.

##  5 Conclusion

This work introduced MENmBERT, a contextualized language model pre-trained on
a Malaysian English corpus. Our experiments demonstrated that fine-tuning
MENmBERT on language-specific data significantly improves performance on NER
tasks with average of +1.74%. For RE, we have achieved average improvement of
+26.27% compare our MENmBERT and MENBERT with baseline PLM’s. However, the
fine-tuned RE models achieved lower performance compared to reported results
on other document-level relation extraction datasets. This suggests that while
MENmBERT’s entity prediction capabilities benefit RE tasks, further
exploration is needed to optimize RE performance in the context of our
dataset. This has gaps has suggested us for future work, explore several
avenues to improve RE performance. One of the approach involve investigating
data augmentation techniques to expand our dataset and improve model training.
Beyond that, we will extend our experiment do several other downstream NLP
task.

##  6 Acknowledgements

Part of this project was funded by the Malaysian Fundamental Research Grant
Scheme (FRGS) FRGS/1/2022/ICT02/MUSM/02/2.

## References

  * Antoun et al. (2021) Wissam Antoun, Fady Baly, and Hazem Hajj. 2021.  Arabert: Transformer-based model for arabic language understanding.  _Preprint_ , arXiv:2003.00104.
  * Araci (2019) Dogu Araci. 2019.  Finbert: Financial sentiment analysis with pre-trained language models.  _Preprint_ , arXiv:1908.10063.
  * Biewald (2020) Lukas Biewald. 2020.  Experiment tracking with weights and biases.  Software available from wandb.com.
  * Chalkidis et al. (2020) Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos. 2020.  Legal-bert: The muppets straight out of law school.  _Preprint_ , arXiv:2010.02559.
  * Chan et al. (2020) Branden Chan, Stefan Schweter, and Timo Möller. 2020.  German’s next language model.  _Preprint_ , arXiv:2010.10906.
  * Chanthran et al. (2024) Mohan Raj Chanthran, Lay-Ki Soon, Huey Fang Ong, and Bhawani Selvaretnam. 2024.  Malaysian english news decoded: A linguistic resource for named entity and relation extraction.  _Preprint_ , arXiv:2402.14521.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  _Preprint_ , arXiv:1911.02116.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _Preprint_ , arXiv:1810.04805.
  * Huang et al. (2020) Kexin Huang, Jaan Altosaar, and Rajesh Ranganath. 2020.  Clinicalbert: Modeling clinical notes and predicting hospital readmission.  _Preprint_ , arXiv:1904.05342.
  * Kalyan et al. (2021) Katikapalli Subramanyam Kalyan, Ajit Rajasekharan, and Sivanesan Sangeetha. 2021.  Ammus : A survey of transformer-based pretrained models in natural language processing.  _Preprint_ , arXiv:2108.05542.
  * Kauf and Ivanova (2023) Carina Kauf and Anna Ivanova. 2023.  A better way to do masked language model scoring.  _Preprint_ , arXiv:2305.10588.
  * Lan et al. (2020) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020.  Albert: A lite bert for self-supervised learning of language representations.  _Preprint_ , arXiv:1909.11942.
  * Lee et al. (2019) Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019.  BioBERT: a pre-trained biomedical language representation model for biomedical text mining.  _Bioinformatics_ , 36(4):1234–1240.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _Preprint_ , arXiv:1907.11692.
  * Martin et al. (2020) Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suá rez, Yoann Dupont, Laurent Romary, Éric de la Clergerie, Djamé Seddah, and Benoît Sagot. 2020.  CamemBERT: a tasty french language model.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics.
  * Nzeyimana and Niyongabo Rubungo (2022) Antoine Nzeyimana and Andre Niyongabo Rubungo. 2022.  KinyaBERT: a morphology-aware Kinyarwanda language model.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 5347–5363, Dublin, Ireland. Association for Computational Linguistics.
  * Salazar et al. (2020) Julian Salazar, Davis Liang, Toan Q. Nguyen, and Katrin Kirchhoff. 2020.  Masked language model scoring.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics.
  * Vamvas et al. (2023) Jannis Vamvas, Johannes Graën, and Rico Sennrich. 2023.  Swissbert: The multilingual language model for switzerland.  _Preprint_ , arXiv:2303.13310.
  * Walker (2005) Christopher Walker. 2005.  _Multilingual Training Corpus LDC2006T06. Web Download. Philadelphia: Linguistic Data Consortium_.
  * Wang et al. (2019) Hong Wang, Christfried Focke, Rob Sylvester, Nilesh Mishra, and William Wang. 2019.  Fine-tune bert for docred with two-step process.  _Preprint_ , arXiv:1909.11898.
  * Wang et al. (2020) Zihan Wang, Karthikeyan K, Stephen Mayhew, and Dan Roth. 2020.  Extending multilingual BERT to low-resource languages.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 2649–2656, Online. Association for Computational Linguistics.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Wu and Dredze (2020) Shijie Wu and Mark Dredze. 2020.  Are all languages created equal in multilingual bert? In _Workshop on Representation Learning for NLP_.
  * Yao et al. (2019) Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou, and Maosong Sun. 2019.  DocRED: A large-scale document-level relation extraction dataset.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 764–777, Florence, Italy. Association for Computational Linguistics.