#  Debugging Frame Semantic Role Labeling
_Towards robust error analysis of statistical models for automatic frame
semantic structure extraction_

Alexandre Kabbach

(2017)

BlueBox VZ14 BlueBox subsubsection french myruled myruled myruled myruled
myruled myruled0 myruled0 myruled

Debugging Frame Semantic Role Labeling
_Towards robust error analysis of statistical models for automatic frame
semantic structure extraction_

Memoir for the Certificate of Specialization in Linguistics

University of Geneva

Faculty of Humanities

Department of Linguistics

December 2017

### Abstract

We propose a quantitative and qualitative analysis of the performances of
statistical models for frame semantic structure extraction. We report on a
replication study on FrameNet 1.7 data and show that preprocessing toolkits
play a major role in argument identification performances, observing gains
similar in their order of magnitude to those reported by recent models for
frame semantic parsing. We report on the robustness of a recent statistical
classifier for frame semantic parsing to lexical configurations of predicate-
argument structures, relying on an artificially augmented dataset generated
using a rule-based algorithm combining valence pattern matching and lexical
substitution. We prove that syntactic preprocessing plays a major role in the
performances of statistical classifiers to argument identification, and
discuss the core reasons of syntactic mismatch between dependency parsers
output and FrameNet syntactic formalism. Finally, we suggest new leads for
improving statistical models for frame semantic parsing, including joint
syntax-semantic parsing relying on FrameNet syntactic formalism, latent
classes inference via split-and-merge algorithms and neural network
architectures relying on rich input representations of words.

###### Table of Contents

  1. 0 Introduction
  2. 1 Frame Semantic Parsing
    1. 1 A brief history of frame semantics theory
      1. 1 Historical motivations
      2. 2 The case for case
      3. 3 From case grammar to frame semantics
    2. 2 Introduction to FrameNet
      1. 1 Core concepts
      2. 2 Data and statistics
      3. 3 Frame and frame element relations
      4. 4 Frame element attributes
        1. Core types
        2. Null instantiations
        3. Semantic types
    3. 3 FrameNet and alternative lexical resources
      1. 1 WordNet
      2. 2 PropBank
      3. 3 VerbNet
      4. 4 Others
    4. 4 The frame semantic parsing task
      1. 1 Definition
      2. 2 Datasets
      3. 3 Evaluation
        1. Scoring
        2. Statistical significance
  3. 2 Related Work
    1. 1 A historical overview of frame semantic parsing
    2. 2 Challenges of frame semantic parsing
      1. 1 A rich underlying taxonomy
      2. 2 Necessary approximations
      3. 3 Annotation data scarcity and domain specificity
    3. 3 Shortcomings of past error analysis
    4. 4 Using FrameNet for automatic paraphrase generation
  4. 3 Models and Experimental Setup
    1. 1 Model
      1. 1 Paraphrastic data augmentation
        1. Philosophy
        2. Implementation
        3. Robustness
        4. Filtering
      2. 2 Argument identification
        1. Model
        2. Features
        3. Learning
        4. Decoding
    2. 2 Experimental setup
      1. 1 Datasets
      2. 2 Toolkits
        1. Querying valence patterns
        2. From Semafor to Rofames
        3. Generating paraphrastic examples and debugging Rofames
      3. 3 Hyperparameters
  5. 4 Baseline and Replication
    1. 1 Validating the preprocessing pipeline
    2. 2 Impact of part-of-speech taggers
    3. 3 Impact of dependency parsers
    4. 4 Impact of exemplar annotation
    5. 5 Impact of the hierarchy feature
    6. 6 Baseline
  6. 5 Results
    1. 1 No filtering setup
    2. 2 Introducing multi-tokens filtering
    3. 3 Introducing semantic filtering
      1. 1 Semantic filtering on nouns
      2. 2 Semantic filtering on adjectives
      3. 3 Semantic filtering on verbs
      4. 4 Semantic filtering on nouns and verbs
      5. 5 Comparing random vs. top filtering
      6. 6 Comparing predictive and count-based distributional models
      7. 7 Measuring the impact of exemplar data
  7. 6 Error analysis
    1. 1 Baseline error analysis
      1. 1 Frequency effects on Frame Elements
      2. 2 Frame elements bearing targets
      3. 3 Lexical challenges
        1. Lexical diversity
        2. Compensating lexical diversity
        3. Semantic considerations
      4. 4 Syntactic challenges
      5. 5 Characterizing _easy_ frame elements
      6. 6 Characterizing _hard_ frame elements
    2. 2 Comparative analysis
      1. 1 Coverage
      2. 2 Distribution
  8. 7 Discussion
    1. 1 Impact of syntactic preprocessing
    2. 2 Consistency of FrameNet annotation
    3. 3 The more the merrier?
    4. 4 Learning conceptual representations from limited data
  9. 8 Conclusion
  10. A Frame Semantic Parsing History: Detailed Overview
    1. 1 srlgildeajurafsky2002
    2. 2 johansson2007
    3. 3 dasetal2014
    4. 4 hermann2014
    5. 5 tack2015
    6. 6 kshirsagaretal2015
    7. 7 rothlapata2015
    8. 8 fitzgerald2015
    9. 9 roth2016
    10. 10 swayamdipta2017
    11. 11 yangmitchell2017

###### List of Figures

  1. 1 SemEval evaluation script output with frames scores
  2. 2 Modified SemEval evaluation script output without frames scores
  3. 1 pFN paraphrastic data generation pipeline
  4. 2 A word lattice showing all paraphrasing candidates output by the pFN system for the sentence: _You contribution to Goodwill will mean more than you may know_. Original targets are represented in the upper edges and paraphrastic candidates are represented in the lower edges
  5. 3 Argument identification feature templates as of dasetal2014
  6. 2 Distribution
  7. 2 Distribution

###### List of Tables

  1. 1 Raw FrameNet data statistics for releases 1.3, 1.5 and 1.7
  2. 2 Computed FrameNet data statistics for releases 1.3, 1.5 and 1.7
  3. 3 Multilingual FrameNet data statistics according to bakeretal2015
  4. 1 FrameNet 1.7 datasets metrics with development and testing sets. The training set does not include the development set
  5. 2 Metrics comparison between FrameNet 1.5 and FrameNet 1.7 datasets with no development set
  6. 1 Argument identification results with gold frames output by the SEM and ACL scoring scripts. Train and test splits are generated from FrameNet 1.5 fulltext XML data. Train split is produced by Rofames, test split is that of dasetal2014 as-is. All data are part-of-speech tagged with MXPOST and dependency parsed with the MSTParser
  7. 2 Argument identification results with gold frames. Train, dev and test splits are part-of-speech tagged with MXPOST or NLP4J and dependency parsed with the MSTParser. Statistical significance is indicated by $\dagger$ for $p<0.05$ and $\ast$ for $p<0.10$
  8. 3 Argument identification results with gold frames. Train, dev and test splits are part-of-speech tagged with NLP4J and dependency parsed with the MSTParser or the BIST parser in its BMST or BARCH variant
  9. 4 Argument identification results with gold frames. Rofames is trained on fulltext (FT) and exemplar (EX) data
  10. 5 Argument identification results with gold frames. Rofames is trained on fulltext (FT) and exemplar (EX) data, with and without the hierarchy feature (H) of kshirsagaretal2015. Statistical significance is indicated by $\ddagger$ for $p<0.01$, $\dagger$ for $p<0.05$ and $\ast$ for $p<0.10$
  11. 6 Baseline scores for argument identification with gold frames on dev and test splits, scored with the ACL script. Train, dev and test splits are part-of-speech tagged with NLP4J and dependency parsed with the BIST BMST parser
  12. 1 Argument identification results with a model trained on FrameNet fulltext (FT) augmented with pFN data generated from FT filtered by part-of-speech (POS). Results are given for nouns (N) and verbs (V) without additional filtering on pFN candidates
  13. 2 Impact of multi-tokens filtering (MWE = true) on argument identification with a model trained on FrameNet fulltext (FT) augmented with pFN data generated from verbs
  14. 3 Impact of various word2vec semantic filtering of pFN candidates on argument identification with a model trained on FrameNet fulltext (FT) augmented with pFN data generated from nouns (N)
  15. 4 Impact of various word2vec semantic filtering of pFN candidates on argument identification with a model trained on FrameNet fulltext (FT) augmented with pFN data generated from adjectives (A)
  16. 5 Impact of various word2vec semantic filtering of pFN candidates on argument identification with a model trained on FrameNet fulltext (FT) augmented with pFN data generated from verbs (V). None of the improvements reported proved statistically significant ($p>0.1$)
  17. 6 Impact of various word2vec semantic filtering of pFN candidates on argument identification with a model trained on FrameNet fulltext (FT) augmented with pFN data generated from nouns and verbs (N +V). In both settings (top and threshold), $p>0.1$
  18. 7 Impact of top-n word2vec semantic filters _versus_ n-random filters for pFN data generated from verbs. Improvements reported are not statistically significant ($p>0.1$ except for XP#29 with $p=0.08$)
  19. 8 Comparing predictive and count-based distributional models for paraphrastic candidates filtering. Impact is measured on argument identification. word2vec is the traditional predictive model trained on Google News and used with the word2vec toolkit. dissect-pred is a word2vec predictive model extracted from a different dataset than Google News, and used with the dissect toolkit. dissect-rcount is a count-based model extracted from the same dataset than dissect-pred and used with the dissect toolkit
  20. 9 Impact of using exemplar data for training Rofames (XP#12), or for generating artificial data via pFN (XP#41). A top-2 word2vec semantic filter is applied to limit the size of the paraphrastic dataset generated by pFN. Improvements in XP#30 and XP#41 are not statistically significant ($p>0.1$), while improvements in XP#12 are ($p<0.01$)
  21. 1 Top 10 most frequent frame elements, with occurrence counts in training and development sets
  22. 1 Frequency effects on Frame Elements
  23. 2 Frame elements bearing targets
  24. 6 Argument identification scores by syntactic realizations of the frame elements
  25. 1 Coverage
  26. 9 Overlap between training and development sets, with a training set composed of FrameNet fulltext data and pFN XP#30 data

##  Chapter 0 Introduction

Frame semantics (fillmore1982) is an approach to the study of lexical meaning
where schematic representations of events, relations or entities called
_frames_ provide semantic background for interpreting the meaning of words and
illustrating their syntactic valence. Its implementation for English through
the FrameNet lexicon has proven greatly beneficial for the field of natural
language processing over the past twenty years. Indeed, FrameNet
(baker1998berkeley) provides a fine-grained classification of predicate-
argument structures in English, relying on an exhaustive taxonomy of inter-
connected semantic classes and roles, which is particularly suited for drawing
useful semantic inferences from text while abstracting away from surface
syntactic structures. FrameNet has been used in tasks as diverse as paraphrase
recognition (pado2005cause), question answering (shen2007using), stock price
movements prediction (xie2013), student essay clarity modeling (persing2013),
knowledge extraction from Twitter (sogaard2015using) and event detection
(liuetal2016; spilioetal2017).

In order to make use of FrameNet annotation for the tasks at hand, the
aforementioned systems have typically relied on statistical models to perform
automatic extraction of frame semantic structures – a task also referred to as
_frame semantic parsing_ – so as to project frame semantic annotation on
custom data for which no FrameNet annotation was available. However, the low
performances of frame semantic parsers, especially in out-of-domain settings
(croceetal2010; hartmannetal2017), have persistently limited the exploitation
of FrameNet’s full potential.

Past research (dasetal2014; fitzgerald2015; hartmann2016) have argued that
frame semantic parsing suffers from the sparsity of FrameNet annotated data.
Yet, previous attempts at augmenting training data, either manually
(kshirsagaretal2015), or through automatic generation via annotation
projection (furstenau2012) and transfer from linked lexical resources
(hartmann2016) have resulted in limited improvements. The limited improvements
of kshirsagaretal2015 are particularly worrisome, as they relied on previously
unused gold FrameNet annotated data, which augmented training data by an order
of magnitude. It is all the more concerning that they did try to compensate
for out-of-domainness effects – the fact that data used to evaluate a
statistical model may contain patterns absent from the training data used to
generate the model – following remarks of dasetal2014, but to no avail. Two
questions arise from those observations:

  1. 1.

Are prediction failures due to a _lack of annotation_ , which translates to
the absence in the training data of linguistic patterns necessary for _any_
machine learning model to properly predict the frame semantic structures
included in the evaluation data?

  2. 2.

Does failure to capture structural generalization principles necessary to
correctly predict frame semantic structures lie with the probabilistic models
used so far?

Past results already offer preliminary leads which can guide reflection:
several research have successfully attempted to incorporate features relating
to information hard-coded in FrameNet – such as frame relations
(kshirsagaretal2015), or frame element semantic types (rothlapata2015) – to
statistical models for frame semantic parsing, suggesting that improvements
can still be made on the modeling side, as current statistical models fail to
capture structural information available in FrameNet. It is therefore more
than ever crucial to understand exactly _what_ statistical models learn, and
_what_ they _fail_ to learn. In addition, it is important to understand
whether statistical models’ failures to capture structural generalizations
correlate with properties of language and linguistic phenomena at large, or
whether they lie in engineering approximations and optimization constraints.
The question of what statistical models do learn in the end is all the more
relevant today as recent approaches to frame semantic parsing have relied on
neural networks (fitzgerald2015; roth2016; swayamdipta2017; yangmitchell2017),
which are notoriously hard to debug. Careful error analysis are still lacking
to understand _qualitatively_ rather than _quantitatively_ what those models
have contributed to and why they do seem to learn better than previous models.

Although it is likely true that statistical models for frame semantic parsing
can still be improved, it is evenly likely that the performances of those
models remain bounded by domain-specific limitations and annotation
redundancies (croceetal2010; hartmannetal2017). Redundancies are particularly
problematic given the cost of FrameNet annotation which requires extensive
linguistic expertise and time. If such redundancies do exist, they should be
clearly identified to prevent annotators from wasting time and energy. Given
those considerations, the purpose of this memoir is threefold:

  1. 1.

Demonstrate our understanding of frame semantics theory, FrameNet and the
literature on frame semantic parsing;

  2. 2.

Formalize the challenges posed by frame semantic parsing, replicate past
results and rationalize experimental setups;

  3. 3.

Report on the preliminary results of a model used to artificially augment
FrameNet annotated data in order to ultimately perform qualitative analysis of
statistical models for frame semantic parsing.

More specifically, we propose a model to artificially augment FrameNet
annotated data with paraphrastic examples generated via a rule-based algorithm
combining lexical substitution and valence pattern111defined as a specific
combination of syntactic realizations of the arguments of a predicate, see
Section 2 matching. For example, given the sentence _John gave an apple to
Mary_, our system could generate the sentence _John handed out an apple to
Mary_ given that both _give_ and _hand out_ predicates bear semantic and
argumental similarities. The core idea of the model is to generate data
reflecting at the surface level222i.e. at the sentence level, versus, encoded
as a high-level feature structural information encoded in FrameNet through
valence patterns. In theory, such artificial data, highly consistent with the
underlying logic of FrameNet annotation, should not contribute significantly
(if not at all) to improving performances of (good) statistical models for
frame semantic parsing. Indeed, our model would only provide _new_ information
in terms of _lexical configurations_ of predicate-argument structures – the
association of a given predicate with a given set of arguments in a given
syntactic configuration, as previously exemplified in the sentence _John
handed out an apple to Mary_. It _would not_ introduce predicates, arguments
or predicate-argument configurations previously _absent_ from the data.
Therefore, if the performances of the aforementioned statistical models were
to improve when trained on an artificially augmented dataset, it would only
demonstrate that such models fail to properly generalize predicate-argument
structures in FrameNet when exposed to a limited set of gold data. Our
paraphrastic data augmentation approach could offer an easy alternative to
compensate for the shortcomings of such models, and a simple way to understand
under which conditions those models fail to generalize and what kind of
additional data, in terms of lexical predicate-argument configurations, those
models require to better _learn_.

The structure of the memoir is as follows: in Chapter 1 we introduce in more
details frame semantics theory, FrameNet and the frame semantic parsing task.
In Chapter 2, we go over the literature on frame semantic parsing and
critically analyze challenges posed by current approaches to frame semantic
parsing as well as the limitations of past error analysis. In Chapter 3, we
introduce our paraphrastic data augmentation model and the experimental setup
used to generate artificial data. In Chapter 4 we replicate the results of
several past studies on frame semantic parsing and detail the baseline used
for quantifying the contribution of our data augmentation model. We report our
results in Chapter 5, and analyze them using different metrics in Chapter 6.
Finally, we discuss possible improvements in Chapter 7 and conclude in Chapter
8.

##  Chapter 1 Frame Semantic Parsing

_Frame semantics_ (fillmore1982; backgroundfn2003; fillmore2009;
fillmoreinterview2010) is an approach to the study of lexical meaning where
schematic representations of events, relations or entities called _frames_
provide semantic background for interpreting the meaning of words and
illustrating their syntactic valence. _Frame semantic parsing_ is the task of
automatically predicting predicate-argument structures following the
classification framework of frame semantics. Pioneered by
srlgildeajurafsky2002, it was more formally defined during the SemEval 2007
shared task 19 (fnsemeval2007) on _Frame Semantic Structure Extraction_.

In this Chapter we present an introduction to the theory of frame semantics,
its implementation into the FrameNet lexicon, and the frame semantic parsing
task. In Section 1 we introduce the theory of frame semantics, its historical
development, and its relation to other linguistic theories such as _case
grammar_. In Section 2 we introduce the FrameNet lexicon, an implementation of
frame semantics theory for English. We present its core concepts, statistics
on annotated data, and several implementation specifications crucial to the
task of frame semantic parsing. In Section 3 we briefly compare FrameNet to
other lexical resources for English such as WordNet, PropBank or VerbNet.
Finally, in Section 4, we introduce the frame semantic parsing task as defined
during the SemEval 2007 shared task, its datasets and evaluation protocols.

###  1 A brief history of frame semantics theory

####  1 Historical motivations

The motivations underpinning the frame semantics enterprise can be traced back
to Fillmore’s early work on the classification of English verbs (fillmore1961;
fillmore1963) and to the fundamental idea that discoveries in the behavior of
particular classes of words could lead to discoveries in the structure of the
grammar of English. Fillmore’s approach at the time was resolutely
transformationalist – influenced by work such as (chomsky1957), (lees1960) or
(chomsky1965) – in that it attempted to classify verbs according to _surface-
syntactic frames_ and _grammatical behavior_ defined as sensitivity of certain
classificatory structures to particular grammatical transformations.

####  2 The case for case

Later on, in his seminal work on _case grammar_ (case1968; fillmore1977a),
Fillmore proposed a more semantically-grounded theory where verbs could be
classified according to the semantic roles of their associated arguments.
Fillmore’s work was then largely influenced by _dependency grammar_ and
_valence theory_ (tesniere1959), notably in its treating all arguments of the
predicate equally, without special consideration for the subject, contrary to
previous transformationalist approaches. Still motivated by considerations
over the universality of the deep syntactic structure of clauses, he
introduced a set of six (potentially) universal semantic role categories
called _deep cases_.111Agentive, instrumental, dative, factitive, locative and
objective Shared _semantic valence_ – defined as a set of co-occurrence
restrictions over the arguments of predicates according to their semantic
roles – allowed for grouping verbs into what Fillmore called _case frames_ ,
which provided the necessary background for understanding the formation of
valid minimal clauses across languages. Abstracting over surface syntactic
configurations, case grammar made it possible to account for cases where
semantic roles occupy different syntactic positions, as in:222 In all our
examples containing semantic roles annotation, we follow FrameNet annotation
style: we rely on a phrase structure grammar rather than a dependency grammar,
i.e., we tag the whole constituents rather than just the head words of the
constituents

[John]Agent broke the window

[A hammer]Instrument broke the window

[John]Agent broke the window [with a hammer]Instrument In sentence 2 the item
marked with the Agent case is the subject whereas in sentence 2, the
Instrument is the subject. Both Agent and Instrument cases may appear in the
same sentence as in 2 but only the Agent is subject.

The theory allowed for interesting predictions: the fact that the subject in
(2) and (2) have different deep cases explain why the combined meaning of the
two sentences could not be produced by co-joining the subjects, as in:

######  Example 1.1.

*John and a hammer broke the window.

According to Fillmore, additional restrictions which may hold between cases
and lexical features (e.g. between Agent and animateness) could also explain
why certain sentences are unacceptable, such as:

######  Example 1.2.

*A hammer broke the glass with a chisel.

Both co-occurrence restrictions over deep cases and constraints between cases
and lexical features are fundamental properties of language that every parser
aiming at automatically predicting predicate-argument structure must capture
in order to properly identify and classify semantic roles. As we will see in
the following chapters, this poses major challenges to the task of frame
semantic parsing in general.

####  3 From case grammar to frame semantics

The theory of _frame semantics_ was formalized in 1982 to overcome the
shortcomings of case grammar which, according to Fillmore, failed to provide
the details needed for semantic description, especially for verbs in
particular limited domains. The concept of a (cognitive) semantic frame which
would provide background for semantic analysis was already salient in case
grammar, where Fillmore considered _case frames_ to characterize small
abstract _scenes_ or _situations_ , so that to understand the semantic
structure of a verb it was necessary to understand the properties of such
schematized scenes.

Some of the core concepts of frame semantics theory were introduced in
(fillmore1977b) focusing on the characterization of the cognitive _scene_
related to the notion of a _commercial event_. In his work, Fillmore argued
that an important set of English verbs could be seen as semantically related
to each other given that they _evoked_ the same general scene. The _commercial
event_ schematic scene, or _frame_ , were to include concepts such as Buyer,
Seller, Goods and Money. Verbs such as buy.v would be said to focus on the
actions of the Buyer with respect to the Goods, backgrounding the Seller and
the Money, allowing sentences such as:

[John]Buyer bought [a car]Goods

[John]Buyer bought [a car]Goods [from Mary]Seller

[John]Buyer bought [a car]Goods [from Mary]Seller [for $5,000]Money while
disallowing sentences such as:

*[John]Buyer bought [from Mary]Seller

*[John]Buyer bought [for $5,000]Money

Similarly, the verb sell.v would be said to focus on the actions of the Seller
with respect to the Goods, backgrounding the Buyer and the Money, while the
verb pay.v would be said to focus on the actions of the Buyer with respect to
both the Money and the Seller, backgrounding the Goods. The point of the
description was to argue that nobody could be said to know the meanings of
these verbs who did not know the details of the kind of scenes that these
words could represent.

In the following section we introduce the FrameNet implementation of frame
semantics theory for English.

###  2 Introduction to FrameNet

####  1 Core concepts

FrameNet (baker1998berkeley; backgroundfn2003; fillmore2009; rupetal2016) is a
computational lexicography project whose purpose is to provide reliable
descriptions of the syntactic and semantic combinatorial properties of each
word in the lexicon, and to assemble information about alternative ways of
expressing concepts in the same conceptual domain. Its output takes the form
of a database of corpus-extracted sentences annotated in terms of frame
semantics. FrameNet is articulated around the following core concepts:

  * •

frames: schematic representations of events, relations or entities. They
provide semantic background for understanding the meaning of words;

  * •

lexical units: words paired with meaning. A lexical unit corresponds to a
specific word sense and is formalized as a link between a _lemma_ and a
_frame_. The lexical unit is said to _evoke_ the frame it belongs to. A
polysemous word will be formalized as a single lemma with references to
multiple lexical units in distinct frames;

  * •

frame elements: frame-specific semantic roles. Frame elements are said to be
_core_ if they are obligatorily expressed, and _non-core_ if they are not (see
examples 2.1 and 2.2 as well as Section 4 for additional details on frame
elements categorization in FrameNet). Frame element names generalize over
frames, i.e. different frames with similar frame elements are considered to
share semantic information. Frame elements can also be connected via frame
element-to-frame element relations, to indicate semantic relations between
them (see Section 3);

  * •

valence units: syntactic realizations of frame elements. A valence unit is
represented as a triplet FE.PT.GF of frame element (FE), phrase type (PT) and
grammatical function (GF);

  * •

valence patterns: the range of combinatorial possibilities of valence units
for each lexical unit. A single valence pattern is usually represented as a
string of multiple valence units FE.PT.GF separated by whitespaces.

Let us now turn to a concrete example illustrating the aforementioned
concepts. In FrameNet, the Commerce_buy frame is defined as follows:

> These are words describing a basic commercial transaction involving a Buyer
> and a Seller exchanging Money and Goods, taking the perspective of the
> Buyer. The words vary individually in the patterns of frame element
> realization they allow. For example, the typical pattern for the verb BUY:
> Buyer buys Goods from Seller for Money.

It contains six lexical units: buy.v, buyer.n, client.n, purchase [act].n,
purchase.v and purchaser.n. It also contains two core frame elements, Buyer
and Goods, which are necessary to form grammatical clauses with lexical units
belonging to the frame, as in:

######  Example 2.1.

[John]Buyer bought [a car]Goods

It also contains thirteen non-core frame elements, such as, e.g., Money,
Place, Seller, Recipient or Time which may or may not be expressed in a given
clause without altering its grammaticality, as in:

######  Example 2.2.

[John]Buyer bought [Susan]Recipient [a car]Goods [yesterday]Time [for
$5,000]Money

The two core frame elements Buyer and Goods, occur in valence patterns such
as:333In the following examples, ‘NP’ is a noun phrase, ‘Ext’ an external
argument (the subject), ‘Obj’ an object, ‘PP’ a prepositional phrase, ‘AJP’
and adjective phrase and ‘Dep’ a dependent

  * •

Buyer.NP.Ext Goods.NP.Obj, as in the simple direct object construction:

######  Example 2.3.

[John]Buyer.NP.Ext bought [a car]Goods.NP.Obj

  * •

Buyer.PP.Dep Goods.NP.Ext, for the passive construction:

######  Example 2.4.

[A car]Goods.NP.Ext was bought [by John]Buyer.PP.Dep

  * •

Buyer.NP.Ext Goods.AJP.Dep for the adjectival construction:

######  Example 2.5.

[John]Buyer.NP.Ext bought [American]Goods.AJP.Dep

####  2 Data and statistics

FrameNet mostly annotates the British National Corpus (BNC) and the American
National Corpus (ANC). FrameNet annotation is divided into two sets:

  1. 1.

exemplar: also referred to as _lexicographic annotation_. Exemplars are the
core of the FrameNet lexicographic project. Each exemplar sentence exemplifies
the use of a valence pattern for a single lexical unit;

  2. 2.

fulltext: developed with NLP applications in mind, fulltext annotations cover
FrameNet annotation for entire corpus-extracted texts where all possible
lexical units and subsequent valence patterns of each sentence are annotated.

Table 1 and Table 2 below provide interesting statistics regarding three major
releases of the English FrameNet data: 1.3, 1.5 and 1.7.444See the release
notes on https://framenet.icsi.berkeley.edu Release 1.3 formed the basis of
the SemEval 2007 shared task (see Section 4), release 1.5 was used in most
recent state-of-the-art work on frame semantic parsing (see Section 1), while
release 1.7, the latest, is used throughout our work (see Chapter 4).

|  | R1.3 | R1.5 | R1.7
---|---|---|---|---
Frames | 795 | 1 019 | 1 221
Frame Elements | (FEs) | 7 124 | 8 884 | 10 503
Lexical Units | (LUs) | 10 195 | 11 829 | 13 572
Valence Units | (VUs) | – | 22 734 | 26 563
Valence Patterns | (VPs) | – | 44 499 | 51 470
Annotation Sets | (AnnoSets) | 151 110 | 173 018 | 202 225
in exemplars |  | 139 439 | 149 931 | 174 017
in fulltext |  | 11 671 | 23 087 | 28 208
Table 1: Raw FrameNet data statistics for releases 1.3, 1.5 and 1.7 |  | R1.3 | R1.5 | R1.7
---|---|---|---|---
FEs per Frame | 9,88 | 9,78 | 9,7
LUs per Frame | 14,14 | 13,03 | 12,5
LUs with AnnoSets |  66,8% |  65,2% |  62%
AnnoSets per LU | 20,5 | 19,5 | 20,73
Table 2: Computed FrameNet data statistics for releases 1.3, 1.5 and 1.7

FrameNet is also being developped in other languages such as Spanish
(subirats2003), Swedish (borin2010) or Japanese (ohara2004), for which we
provide comparative annotation statistics in Table 3.

| Spanish | Swedish | Japanese | English
---|---|---|---|---
Nouns | 271 | 28 891 | 2 043 | 5 299
Verbs | 856 | 5 398 | 908 | 5 141
Adjectives | 99 | 3 293 | 134 | 2 347
Adverbs | 16 | 322 | 89 | 214
Other | 26 | 124 | 231 | 420
Total Lexical Units | 1 268 | 38 028 | 3 405 | 13 421
Total Annotation Sets |  11k |  9k |  73k |  200k
Table 3: Multilingual FrameNet data statistics according to bakeretal2015

####  3 Frame and frame element relations

Frames in FrameNet form a network of interconnected entities, connected to
each other via eight types of _frame-to-frame relations_. Frame elements are
similarly connected to each other via _frame element-to-frame element
relations_. Frame relations include:

  1. 1.

Inheritance: a _child_ frame inherits from a _parent_ frame if its equally of
more specific than its parent. For example, the Commerce_buy frame inherits
from the Getting frame, which can be understood intuitively by the fact that
_buying_ something is _getting_ it, with the additional assumption that the
thing received was exchanged for money. Hence, the Buyer frame element of the
Commerce_buy frame is related to the Recipient frame element of the Getting
frame but is semantically more specific.

  2. 2.

[John]Recipient acquired [a house]Theme

  3. 3.

[John]Buyer bought [a house]Goods

  4. 4.

Perspective_on: a given frame is in a Perspective_on relationship to a
_neutral_ frame if it provides a specific perspective on the neutral frame.
For example, both the Commerce_buy frame and the Commerce_sell frame are in a
Perspective_on relationship to the Commerce_goods-transfer frame: the
Commerce_buy frame provides the point-of-view of the buyer while the
Commerce_sell frame provides the point-of-view of the seller, although both
frames relate to the same _scene_ , as shown by the following sentences with
the lexical units Commerce_buy.buy.v and Commerce_sell.sell.v:

  5. 5.

[John]Buyer bought [a car]Goods [from Mary]Seller [for $5,000]Money

  6. 6.

[John]Seller sold [a car]Goods [to Mary]Buyer [for $5,000]Money

  7. 7.

Using: a _target_ frame is in a Using relationship to a _source_ frame if a
part of the scene evoked by the target frame refers to the source frame. Note
that a frame can be in a Using relationship with multiple other frames. For
example, the Judgment_communication frame is in a Using relationship to both
the Judgment frame and the Statement frame. It does not Inherit from the
Judgment frame as it is not a simple sub-type of a purely cognitive state.
Similarly, it does not Inherit from the Statement frame as its frame element
corresponding to the Statement frame’s Message frame element is actually split
in two separate frame elements: Evaluee and Reason;

  8. 8.

SubFrame: a _target_ frame is in a SubFrame relationship to a _source_ frame
if the source frame refers to a sequence of _states_ and _transitions_ , of
which the target frame is a particular instance. For example, the Arrest,
Arraignment, Trial, Sentencing and Appeal frames are all in a SubFrame
relationship with the Criminal_process frame;

  9. 9.

Precedes: a _target_ frame is in a Precedes relationship to a _source_ frame
if both target and source frames refer to specific states in a sequence of
states and if the target frame has temporal or ordinal precedence to the
source frame. For example, the aforementioned Arrest frame is in a Precedes
relationship to the Arraignment frame;

  10. 10.

Causative_of and Inchoative_of: a _target_ frame is in a Causative_of,
Inchoative_of relationship to a stative _source_ frame if it describes the
causative / inchoative view on a given stative scene. Causative frames will
typically also inherit from the Transitive_action frame and contain an Agent
frame element as in “John broke the window”, and inchoative frames will
typically inherit from the Event, State or Gradable_attribute frame;

  11. 11.

Metaphor: a _target_ frame is in a Metaphor relationship to a _source_ frame
if many of the lexical units in the target frame are to be understood at least
partially in terms of the source frame. Consider the Metaphor relation between
the Cause_motion frame and the Suasion frame, as in:

######  Example 2.6.

The judge was not moved by the lawyer’s argument

In 2.6, “moved” would be annotated in the Cause_motion frame, with the
annotation set marked with the Metaphor label;

  12. 12.

See_also: a _target_ frame is in a See_also relationship to a _source_ frame
if it is similar to the source frame but should be carefully differentiated,
compared and contrasted. This is intended to help human users better make
sense of the FrameNet frame differentiation.

We will detail in Section 1 and Section 2 how frame and frame element
relations have been successfully used to improve automatic predictions of
frame element labels, in work such as (kshirsagaretal2015) or
(matsubayashi2014).

####  4 Frame element attributes

##### Core types

FrameNet distinguishes between core and non-core frame elements, following the
traditional distinction between core arguments and peripheral adjuncts (such
as _time_ , _place_ and _manner_). Those frame element categories are referred
to in FrameNet as core and peripheral. FrameNet also contains two additional
frame element categories:

  1. 1.

Extra-thematic frame elements, which are frame elements that introduce
information which is not a necessary part of the description of the central
frame they belong to. For example, the Recipient frame element in the
Commerce_buy frame is extra-thematic: it introduces a concept incorporated
from the Giving (and subsequent) frame which is not core in the Commerce_buy
frame, as shown in the following sentences:

  2. 2.

[Mary]Buyer bought [a car]Goods

  3. 3.

[Mary]Buyer bought [a car]Goods [for John]Recipient The difference between
extra-thematic and peripheral frame elements is that peripheral frame elements
do not uniquely characterize a frame, and can be instantiated in any
semantically appropriate frame;

  4. 4.

Core-unexpressed frame elements, which are frame elements that behave like
core frame elements in the frame they belong to but are not necessarily listed
among the frame elements in the descendant frames. This can be explained by
cases where information bear by the frame element is _included_ in the lexical
units in the descendant frame, and cannot be separately expressed. This is
exemplified in the Choosing frame which inherits from the Intentionally_act
frame containing a core-unexpressed Act frame element, which cannot be passed
on to the Choosing frame without forming ungrammatical sentences as in 6:

  5. 5.

[I]Agent will do [the cooking]Act

  6. 6.

*[I]Cognizer will choose [decision]Act [to cook]Chosen

##### Null instantiations

There are many occurrences in corpora where core frame elements do not appear
in given sentences. Those cases are referred to in FrameNet as _null
instantiations_. FrameNet categorizes three cases of null instantiation:

  1. 1.

Indefinite Null Instantiation (INI): refers to cases where an argument is
omitted for intransitive lexical units. FrameNet marks transitive and
intransitive lexical units as both transitive and, in intransitive cases,
marks the core argument as INI, as in:

  2. 2.

[I]Ingestor ate [a cake]Ingestibles

  3. 3.

[I]Ingestor ate Ingestibles.INI

  4. 4.

Definite Null Instantiation (DNI): refers to cases where an argument is
omitted under lexically licensed zero anaphora, where all parties in a
conversation are assumed to know what the argument is, as in:

  5. 5.

[We]Competitor won! Competition.DNI

  6. 6.

When did [they]Theme arrive? Goal.DNI

  7. 7.

Construction Null Instantiation (CNI): refers to cases where the grammar
considerations require or permit the omission of some arguments in particular
structures. This is typically the case for imperative sentences which omit the
subject, and passive sentences which omit the agent, as in:

  8. 8.

Get out! Theme.CNI

  9. 9.

We have been robbed Perpetrator.CNI

Null instantiations exhibit different linguistic phenomena which have been
shown to be particularly hard to learn by machine learning algorithms used for
frame semantic parsing. Indeed, frame semantic parsers have a tendency to
wrongly predict core frame elements when confronted to null instantiations,
negatively affecting recall metrics (see (chen2010) detailed in Section 2).

##### Semantic types

Frame element semantic types in FrameNet are designed primarily to aid frame
parsing and automatic frame element recognition. They provide a general and
limited set of categories which characterize the basic typing of fillers of
frame elements. For example, the Cognizer frame element is marked as
‘Sentient’, the Place frame element as ‘Locative_relation’, and the Purpose or
Means frame elements as ‘State_of_affairs’. We will detail in Section 2 how
matsubayashi2014 successfully used frame elements semantic types to improve
automatic predictions of frame element labels.

###  3 FrameNet and alternative lexical resources

####  1 WordNet

WordNet (miller1995wordnet) is a large electronic lexical database of English,
which was inspired by pycholinguistics research investigating how and what
type of information is stored in the human mental lexicon. WordNet groups
words into sets of synonyms called _synsets_ and records a number of relations
among lexical entries, such as synonymy/antonymy, hyponymy/hypernymy,
meronymy/holonymy, troponymy or entailment. It has a large coverage, with more
than 200,000 word-sense pairs documented in four morphosyntactic categories:
nouns, verbs, adjectives and adverbs. Contrary to FrameNet, WordNet contains
little syntagmatic information. Its purpose is not to document predicate-
argument structures in general. It does, however, offer more fine-grained
distinctions across lexical entries, especially for nouns, and is therefore
complementary to FrameNet.

####  2 PropBank

PropBank (propbank2005) is a resource which annotates predicate-argument
structures on top of the phrase structure annotation of the Penn TreeBank
(marcus1993). Predicates senses are classified according to frames, as in
FrameNet, but, unlike FrameNet, PropBank postulates predicate-specific roles
which are then generalized to a set of 5 core roles classes (ARG0 to
ARG4)555ARG0: agent, ARG1: patient, ARG2: instrument, benefactive, attribute,
ARG3: starting point, benefactive, attribute, ARG4: ending point and 18
adjunct/modifier classes (ARGM-*)666Such as ARGM-LOC for location and ARGM-TMP
for time. PropBank annotates verbs exclusively and does not offer the same
granularity of semantic roles and frames as in FrameNet. It also does not
provide semantic relations across frames and roles. Its generalized roles set
does however make the task of automatic semantic role labeling simpler and it
has been made it a popular resource in parsing and computational linguistics
consequently.

####  3 VerbNet

VerbNet (verbnet2005) is a digital database that provides predicate-argument
structures documentation for English verbs, relying on a hierarchical
classification of verbs inspired by Levin’s verb classes (levin1993).
Arguments in VerbNet consist of a list of 23 generic thematic roles. Each
(syntactic) frame groups together verbs that have similar semantic and
syntactic characterizations, and VerbNet provides documentation for frame-
specific thematic role syntactic realizations and co-occurence constraints.
VerbNet has a much lower coverage than FrameNet, with only 4526 senses of 3769
verbs annotated, and, unlike FrameNet, does not encompass (semantic) relations
across its entities.

####  4 Others

Several other resources also aim at documenting predicate-argument structures
for English: the Valency Dictionary of English (herbst2004valency), which,
unlike previously mentioned resources, is not free and not entirely machine-
readable; VALEX (korhonen2006large), a large valency subcategorization lexicon
for English verbs, which is, unlike previously mentioned resources, partially
automatically annotated and therefore somehow noisy; and Abstract Meaning
Representation (AMR) (amr2013), the newest resources of all. AMR offers a
graph-based representation of lexical concepts and typed relations between
those concepts, denoted by an English sentence. It covers PropBank predicate-
argument semantics, entity linking, coreference, modality, negation,
questions, relations between nominals and canonicalization of content words.
Its graph-based representation allows for abstracting away grammatical
specificities, and its single-structure representation is designed to support
rapid corpus annotation and data-driven natural language processing.

###  4 The frame semantic parsing task

####  1 Definition

Frame semantic parsing is the task of automatically predicting frame-evoking
words, frames, and corresponding frame element spans and labels in a given
sentence. The task, formally defined in the SemEval 2007 shared task 19 on
_Frame Semantic Structure Extraction_ (fnsemeval2007), was divided into three
subtasks:

  1. 1.

target identification: identify all frame evoking words in a given sentence;

  2. 2.

frame identification: label all frames of pre-identified targets in a given
sentence;

  3. 3.

argument identification: identify all frame-specific frame element spans and
labels for pre-identified lexical units in a given sentence.

####  2 Datasets

Data were based on an extended version of the FrameNet 1.3 data release.
Training data comprised about 2,000 annotated sentences with about 11,000
annotation sets in fulltext documents. Testing data comprised about 3
annotated fulltexts composed of 120 sentences and about 1,000 annotation sets
(See details in (dasetal2014)). Texts were extracted from the American
National Corpus (ANC) and the NTI website.777http://www.nti.org/

In this work, we first attempt to replicate past studies and therefore rely on
the original FrameNet 1.5-based training and testing sets of dasetal2014 (see
Chapter 4 and Section 1). However, we also report results on FrameNet
1.7-based datasets, where the testing set contains the same list of fulltext
documents as the original list of dasetal2014, but contains more annotated
data as those documents were further annotated in the 1.7 release (see Table
2) and therefore produce a more robust testing set.

####  3 Evaluation

##### Scoring

The original task was evaluated globally, each subtask taking as input the
predicted output of the previous subtask and global precision recall and
$F_{1}$ scores were computed for frames and arguments microaveraged across a
concatenation of the test set sentences.

In this work, we focus mostly on the subtask of argument identification and
therefore evaluate predictions of arguments spans and labels given gold
targets and gold frames. We use the modified evaluation script of the SemEval
shared task introduced by kshirsagaretal2015. The modified evaluation script
does not give extra credit for gold frames, unlike the standard SemEval
script. This allows for better evaluation of parsers’ improvements in argument
identification, as it does not dilute argument identification scores by
blending them with (gold) frame scores, necessarily 100% accurate. For
example, Figure 1 which shows the original SemEval evaluation script output
for the sentence _The college’s students were forced to take shelter in the
gymnasium_ , contains full M/S/G scores on the fourth line of the output
script, corresponding to the frame (FR) Education_teaching evoked by the
target ranging from character 20 to character 27. A contrario, Figure 2, which
shows the modified evaluation script output for the same sentence, does not
contain any M/S/G scores for the same line.

The evaluation script scores each frame and argument independently. In our
configuration, argument spans and labels are scored jointly. Core frame
elements are given one point and non-core frame elements are given half a
point. The ‘Match’ column M is assigned a value each time a correct argument
is identified (true positive); the ‘Score’ column ‘S’ is assigned a value each
time an argument is wrongly predicted (false positive); and the ‘Gold’ column
G is assigned a value each time the corresponding span and label is gold
annotation.

Precision is then computed by summing over all true positives and false
positives for all predicted labels in a given sentence:

$P=\frac{\sum{t_{p}}}{\sum{t_{p}}+\sum{f_{p}}}=\frac{\sum{M}}{\sum{S}}$ (1)

Recall is computed in a similar fashion with true positives and false
negatives:

$R=\frac{\sum{t_{p}}}{\sum{t_{p}}+\sum{f_{n}}}=\frac{\sum{M}}{\sum{G}}$ (2)

$F_{1}$ score is then given by:

$F_{1}=2\cdot\frac{P\cdot R}{P+R}$ (3)

Figure 1: SemEval evaluation script output with frames scores Figure 2:
Modified SemEval evaluation script output without frames scores

##### Statistical significance

Finally, we measure the statistical significance of all our results relying on
the bootstrapping approach of bergkirkpatrick2012: let us consider a system
$A$ which we compare to a baseline system $B$. Let us consider additionally a
test set $x=x_{1},\ldots,x_{n}$ on which $A$ beats $B$ by $\delta(x)$. $b$
bootstrap samples $x^{(i)}$ of size $n$ are drawn by sampling with replacement
from $x$. For each $x^{(i)}$, a variable $s$, initialized at 0, is incremented
if and only if $\delta(x^{(i)})>2\delta(x)$. The $p$ value is then given by:

$p(x)=\frac{s}{b}$ (4)

##  Chapter 2 Related Work

###  1 A historical overview of frame semantic parsing

Early work on frame semantic parsing has relied almost exclusively on
statistical classifiers based on an exhaustive list of carefully handcrafted
features (srlgildeajurafsky2002; johansson2007; dasetal2014; hermann2014;
kshirsagaretal2015; rothlapata2015; tack2015). Such approaches usually require
extensive feature engineering which can prove particularly time-consuming, in
addition to risking biasing systems for the task at hand (rothlapata2016).
Moreover, those approaches have typically made use of heavy preprocessing, be
it through part-of-speech tagging, dependency-parsing, or coreference
resolution (e.g. dasetal2014; rothlapata2015). However, preprocessing has been
shown to significantly impact the overall performances of statistical models
for frame semantic parsing (see Chapter 4 for details), and even to strictly
bound the performances of those models below an unsatisfactory threshold
(dasetal2014; tack2015). On top of this, previous research have relied on
different preprocessing toolkits, and those inconsistent experimental setups
have prevented fair comparison between models, as they do not allow
discounting the contribution of the preprocessing toolkit from the
contribution of the statistical model itself.

Given those considerations, recent neural-network-based approaches to frame
semantic parsing appear very promising (fitzgerald2015; roth2016;
swayamdipta2017; yangmitchell2017), given the limited number of core features
they require as input. In this regard, the most exciting approach is probably
that of swayamdipta2017, which proposed two very interesting models: one
requiring no syntactic preprocessing and one acquiring syntactic
representations alongside other distributional representations in a multi-task
setting. Both models yielded competitive if not above state-of-the-art
results.

Recent approaches have also called into question how the frame semantic
parsing task itself has been formalized, confirming what intuition long
suggested: frame and argument identification performed jointly yield better
results than performed sequentially (yangmitchell2017).

State-of-the-art results on frame identification are reported in (hermann2014)
at 88.41% $F_{1}$ score; on argument identification with gold frames, in
(swayamdipta2017) at 68.9% $F_{1}$ and on argument identification with
predicted frames in (yangmitchell2017) at 76.6% $F_{1}$. Readers can refer to
Appendix A for a more exhaustive historical overview of supervised and semi-
supervised methods for frame semantic parsing.

###  2 Challenges of frame semantic parsing

Some important challenges posed by frame semantic parsing are deeply rooted in
its underlying theoretical formalization. Indeed, frame semantics theory
operates many fine-grained distinctions within predicate-argument structures
classification, many of which are semantically motivated and do not translate
into linguistic phenomena that can easily be captured at the surface syntactic
level (croceetal2010). Concretely, those considerations have turned FrameNet
into an exhaustive taxonomy which has rendered the task of frame semantic
structure prediction very complex (see Section 1).

The richness of FrameNet’s classification, which is also its main asset
compared to other lexical resources, has lead researchers to operate multiple
approximations in order to be able to solve the task at hand. Those
approximations dealt with the definition of the frame semantic parsing task
itself, and with its resolution, both in terms of modeling and optimization
(see Section 2).

In addition, the fined-grained nature of FrameNet annotation has rendered its
production extremely costly, resulting in limited annotated data compared to
what is usually required for machine learning algorithms to learn the kind of
generalizations necessary to properly predict frame semantic structures,
within and across domains (see Section 3).

In this section we skim through the existing literature on each of those
challenges and detail some of the solutions that have been proposed to tackle
them.

####  1 A rich underlying taxonomy

The FrameNet taxonomy, albeit rich and complex, is also well structured. From
this observation, matsubayashi2014 proposed to exploit the structural
information available in FrameNet – such as frame and frame element relations
and semantic types – combined with other lexical resources such as PropBank
and VerbNet, to infer generalizations on frame elements which would reduce the
number of classes for the argument identification task, and thereby its
complexity. Although they did show significant improvements, with a 19.16%
error reduction in terms of total accuracy, their approach can be viewed as a
simplification of the task of frame semantic parsing in general, a
simplification which discards many interesting distinctions made by the
FrameNet framework. It does however support in an indirect fashion the
underlying annotation of FrameNet, as it shows that generalizations motivated
by theoretical considerations from frame semantics theory generate systematic
linguistic phenomena in as they can be captured by a machine learning system.

Other forms of linguistic knowledge annotated in FrameNet have proven harder
to capture through automatic means, whether directly or indirectly.
dasetal2014 for instance, showed that a traditional log-linear model relying
on a latent variable over frame and argument representations failed to capture
the kind of structural constraints on frame elements which are hard-coded in
FrameNet. Such constraints included a vast majority of non-overlapping frame
element labels, – which the system violated 441 times111About 10% of the time,
given the size of the test set – and other forms of constraints, such as
argument pairwise requirement/exclusion, a constraint exemplified in the
following sentences:

[A blackberry]Entity_1 resembles [a loganberry]Entity_2

[Most berries]Entities resemble each other The frame elements Entity_1 and
Entity_2 _require_ one another and _exclude_ the Entities frame element and
vice versa.

In a similar vein chen2010 showed that the frame semantic parser of
dasetal2014 often failed to correctly identify null instantiations of core
frame elements. As for kshirsagaretal2015, they showed that the same parser
also failed to capture linguistic knowledge encoded in frame relations, and
that such information could only be adequately captured through active
supervision.

####  2 Necessary approximations

In order to simplify automatic frame semantic structure extraction, past
research have proceeded to multiple approximations.

The first approximation deals with the definition of the task itself, which
has been artificially split into the three subtasks of _target identification_
, _frame identification_ and _argument identification_. Nothing in the theory
of frame semantics requires this subdivision of labor, which is more commanded
by requirements in terms of evaluation for natural language processing. On the
contrary, it is often useful in order to perform frame identification for a
given predicate, to process the arguments of the predicates _in context_. The
recent work of yangmitchell2017 has confirmed this intuition, achieving a
significant improvement in both frame and argument identification tasks by
performing those two tasks jointly rather than sequentially.

The second approximation deals with input processing, which has systematically
been performed at the sentence level, despite theoretical considerations in
frame semantics stressing the importance of processing language at a higher
level than the mere sentence (fillmore1982). Again, this constraint is due to
computational considerations, and the recent work of rothlapata2015 who
improved argument identification performance by incorporating discourse
features through, e.g., coreference resolution, confirms the limitations posed
by this approximation.

Other approximations have been performed to ease the modeling and better fit
it into a machine learning pipeline. Such approximations include that of
dasetal2014 on the nature and the systematicity of structural constraints
which operate on frame elements. Those approximations, which often have no
theoretical grounds, allow formulating the problem of argument identification
as a constrained optimization problem, turning a complex structured prediction
problem into a more standard optimization problem, easier to process with,
e.g., linear programming solvers.

Finally, work such as (dasetal2014; tack2015; fitzgerald2015) have
demonstrated how the performances of traditional approaches to frame semantic
parsing, relying on heavy syntactic preprocessing, could be bounded by the
performances of syntactic parsers. This is particularily striking in the case
of argument span identification methods, which, in the aforementioned work,
have relied on heuristics-based algorithms exploiting syntactic features
derived from syntactic parsers, and which have bounded arguments span
identification recall to a low 80%.

The impact of preprocessing on frame semantic parsing will be further detailed
in Chapter 4, where we will discuss how inconsistent preprocessing pipelines
across research and experimental setups may pose problems for replicating past
results and quantifying systems contributions.

####  3 Annotation data scarcity and domain specificity

It has frequently been argued that FrameNet lacked the necessary amount of
annotated data for machine learning systems to properly predict frame semantic
structure (palmercoverage2010; dasetal2014; das2014; fitzgerald2015), or
generalize across domains (croceetal2010; dasetal2014; hartmannetal2017).

The negative impact of coverage gaps has been extensively documented by
palmercoverage2010, albeit limited to the task of frame identification, which
had a knock-on effect on argument identification for frame semantic parsing
pipelines processing tasks sequentially.

While johansson2007 noted a 20 $F_{1}$ loss on argument identification when
tested on a different domain than that of the training set, dasetal2014 argued
further that exemplar sentences could not successfully be used for training
frame semantic parsers on the SemEval data. According to them, possible
explanations included the fact that exemplars were not representative as a
sample, did not have complete annotations, and were not from a domain similar
to the fulltext test data.

Yet, recent work of kshirsagaretal2015 and yangmitchell2017 have shown a few
$F_{1}$ points gain when training on both fulltexts and exemplars of the
FrameNet 1.5 dataset, with and without domain adaptation techniques. Those
improvements, however, remain limited considering that the exemplar sentences
represent more than three times the amount of annotated data available in
fulltext data.

Many approaches have been tried to overcome the problems of data sparsity,
coverage gaps and domain specificities. croceetal2010 and hartmannetal2017
proposed to rely on distributional representations of lexical features to
better generalize in out-of-domain settings. furstenau2012 augmented the
FrameNet annotation set by projecting FrameNet role annotation to
syntactically similar sentences, while hartmann2016 augmented the set of
annotation for both FrameNet senses and roles using distant supervision to
transfer annotation for linked lexical resources. rastogiandvandurme2014 and
pavlicketal2015 proposed to augment FrameNet annotation automatically via
paraphrasing, while hongbaker2011, fossatietal2013, chang2015scaling and
chang2016linguistic proposed to do so via crowdsourcing.

However, the aforementioned methods have either yielded limited improvements,
either not been tested in traditional frame semantic parsing setups for proper
comparison, either not been tested at all. It is therefore difficult to
measure the true impact of FrameNet data augmentation on frame semantic
parsing. Even if such improvements were to be attested, its impact would
remain limited without a proper error analysis which would help us understand
exactly _what kind of linguistic knowledge_ additional data helped capture.

###  3 Shortcomings of past error analysis

Although much work has been done over the past ten years in the field of frame
semantic parsing, very little has been done in terms of error analysis to
understand exactly the contribution of each new models to the tasks at hand,
as well as what systems still fail to capture in terms of linguistic
phenomena.

The seminal work of srlgildeajurafsky2002 provided extensive error analysis,
including performance of the system broken down by feature and roles,
measuring the contribution of each feature to the identification and labeling
of each semantic role. Their work also provided accuracy, recall and $F_{1}$
scores for most frequent frame elements, as well as performance measures of
each learning algorithm used. This provided a detailed overview of what their
system was good at (e.g., identifying and labeling Agent frame elements), and
what it was bad at (e.g., identifying Manner and Location frame elements).
Their analysis of each feature’s coverage and contribution to frame element
identification and labeling helped craft a carefully selected set of features
(highly motivated and backed by experimental data and analysis) which has been
used almost as-is (though extended) in subsequent studies of semantic role
labeling.

Since then, however, very little has been done in terms of error analysis. If
work such as (palmercoverage2010) or (kshirsagaretal2015) do provide coverage
and frequency metrics of lexical units and/or frame elements, potentially
exhibiting correlations between coverage gaps, low frequency frame elements
and low performance of system, we still lack the kind of analysis done on
PropBank work (e.g. woodsend2014), in order to identify linguistic knowledge
which proves systematically hard to predict by frame semantic parsers. This is
especially true for recent approaches relying on neural networks. If work such
as fitzgerald2015 or rothlapata2016 have provided very useful 2D-graphics of
the representations learned by their model, projections of learned
representations exemplifying how models could operate classification of
semantic roles and group together similar semantico-syntactic configurations,
such as nested agents, relative clauses, complements of nominal predicates,
etc., error analysis are often absent, or limited to the mere comparison of
performance based on sentence length (yangmitchell2017).

A notable exception to this absence of error analysis is work by
rothlapata2015. They were indeed among the only ones to provide an exhaustive
qualitative analysis of the improvements of their system, based on discourse
and context features. They showed for instance that adding contextual features
helped the model capture semantic properties of unseen tokens, such as being
able to properly label ‘Dec. 1’ with the Time frame element. Similarly, their
system proved able to capture certain semantic categorizations, such as the
fact that _aunt_ , _uncle_ and _grandmother_ were all of the _human_ semantic
type and therefore could more likely fill roles such as Recipient than that of
Goal. They additionally showed that their system, and more specifically the
contextual features on semantic types and the discourse feature on salience,
better captured the _agency_ property of certain frame elements, which
translated into an increase in recall from 56% to 78%. They also showed that
contextual word representations lead to better identification of Time frame
elements, especially when the label was filled by an infrequent adverb.

###  4 Using FrameNet for automatic paraphrase generation

A major question that arises from previous sections is whether data in
FrameNet contain many _redundancies_ and whether they are _well-balanced_ for
parsers to properly learn useful generalizations, or whether the fault lies in
machine learning systems used so far which prove unable to capture useful
generalizations from the available training data.

In order to test the robustness and ability of statistical models for frame
semantic parsing to extract information from FrameNet annotated data, we
propose, in the next chapters, a model to artificially augment FrameNet
annotation with _paraphrastic_ examples generated via a rule-based algorithm
combining lexical substitution and valence pattern matching. The core idea of
the model is to generate data reflecting at the surface level – via lexical
configurations of predicate-argument structures – structural information
encoded in FrameNet through valence patterns. By recombining information
already latently present in FrameNet, our paraphrastic data augmentation
approach should provide a simple way to understand under which conditions
statistical models for frame semantic parsing fail to generalize and what kind
of additional data, in terms of lexical predicate-argument configurations,
those models require to better learn.

Our paraphrastic data augmentation model uses FrameNet-internal knowledge to
generate near-paraphrases which are in turn used to try and improve frame
semantic parsing. It is worth mentioning that both ideas have antecedents in
the literature: the potential of FrameNet for paraphrasing has already been
thoroughly documented (ellsworth2007; coyne2009lexpar; hasegawaetal2011), and
several work have argued that paraphrasing could prove to be beneficial to
frame semantic parsing (rastogiandvandurme2014; pavlicketal2015). However, no
open-source system exists today to generate paraphrases with FrameNet, and no
previous work have actually _proven_ paraphrases to quantifiably improve frame
semantic parsing performances.

Several other work have attempted to augment training data, sometimes relying
on paraphrase generation, be it for Propbank (woodsend2014), or FrameNet
(furstenau2012), but none of those approaches used resource-internal knowledge
to generate paraphrases and/or augment the training set. Moreover, they have
both exhibited limited improvements, while significantly increasing the size
of the training set. With our approach, we hope that a system relying on
FrameNet-internal logic to augment the training set will prove more consistent
with the gold annotation and lead to better performances ultimately.

##  Chapter 3 Models and Experimental Setup

In this chapter we introduce two models: (1) a rule-based model for
paraphrastic data augmentation used for augmenting our training set of
FrameNet annotated data with artificial exemples generated from the FrameNet
database via a valence patterns matching algorithm; and (2) a statistical
model for argument identification as originally defined by dasetal2014 and
refined by kshirsagaretal2015.

In Section 1 we introduce the paraphrastic data augmentation model, detail its
core motivations and implementation specifications, and discuss its
robustness. In Section 2 we introduce the argument identification model, its
features, and its learning and decoding strategies. Additionally, in Section
2, we detail the experimental setup used throughout this work, in terms of
training, development and testing datasets, toolkits and hyperparameters used
for all the models.

###  1 Model

####  1 Paraphrastic data augmentation

##### Philosophy

Our data augmentation model is grounded in frame semantics theory which argues
that lexical units sharing similar valence patterns should be considered as
semantically equivalent (fillmore1982; rupetal2016). This fundamental idea is
at the core of past research which have relied on FrameNet to generate
sentential paraphrases (ellsworth2007; hasegawaetal2011). For example, both
buy.v and purchase.v lexical units in the Commerce_buy frame are compatible
with the Buyer.NP.Ext Goods.NP.Obj valence pattern. Therefore, the following
sentences should be considered as _grammatically_ correct and _semantically_
equivalent:111In all the following examples boldface indicates _targets_ ,
i.e. frame-evoking words in context

[John]Buyer.NP.Ext bought [a car]Goods.NP.Obj

[John]Buyer.NP.Ext purchased [a car]Goods.NP.Obj

Note that the semantic equivalence is not strict, as FrameNet may classify in
the same frame lexical units with different semantics – sometimes even
antonyms – provided that those lexical units share some commonalities in the
semantico-syntactic realizations of their arguments. For example, both make.v
and lose.v lexical units belong to the same Earnings_and_losses frame and are
compatible with the Earner.NP.Ext Earnings.NP.Obj valence pattern, as in:

[I]Earner.NP.Ext like working and making [money]Earnings.NP.Obj

[I]Earner.NP.Ext like working and losing [money]Earnings.NP.Obj

As lexical units with diverging (or even opposite) meanings may share common
valence patterns in FrameNet, the lexical units generated by our valence
pattern matching algorithm will not necessarily form _true_ or _valid_
paraphrases of the original target. Hence, we call them _paraphrastic_
candidates and describe our data augmentation method as _paraphrastic data
augmentation_.

##### Implementation

Our paraphrastic data augmentation is implemented in a system called pFN,
which workflow is as follows:

  1. 1.

Extract all annotation sets from FrameNet XML files and unmarshall to a list
of standardized annotation sets for easy manipulation;

  2. 2.

Filter the list of standardized annotation sets according to the filtering
options (detailed in Section 1);

  3. 3.

For each filtered annotation set, extract the corresponding valence pattern
and fetch all the lexical units in training data compatible with the source
valence pattern. The list of lexical units output by the system, enriched with
the set of annotation labels extracted from the original annotation set, will
constitute the list of paraphrastic candidates;

  4. 4.

Filter the list of paraphrastic candidates according to the filtering options
(detailed in Section 1);

  5. 5.

Generate all possible sentences from the list of filtered candidates, project
FrameNet annotation label to the new sentences, and export all newly generated
sentences to FrameNet XML format.

Figure 1 presents the overall pipeline of the pFN system. The generation step
is where the list of paraphrastic candidates is generated for a given source
annotation set. To provide a concrete example, consider the following
sentence:

######  Example 1.1.

Your contribution to Goodwill will mean more than you may know

This sentence has exactly six annotation sets in FrameNet (one per
target):222for clarity we only show the frame element labels, omitting phrase
types and grammatical functions

[Your]Donor contribution [to Goodwill]Recipient will mean more than you may
know

[Your contribution]Trajector to [Goodwill]Landmark will mean more than you may
know

[Your contribution to Goodwill]Means will mean [more than you may know]Value

Your contribution to Goodwill will mean [more [than you may
know]Class]Added_set

Your contribution to Goodwill will mean more than [you]Hypothetical_event may
[know]Hypothetical_event

Your contribution to Goodwill will mean more than [you]Cognizer may know The
pFN system will return paraphrastic candidates for only four of those targets:
_contribution_ , _to_ , _may_ and _know_. Targets for which no paraphrastic
candidates are returned usually indicate that they are the sole lexical units
found in the input valence patterns. The list of paraphrastic candidates can
be represented as a word lattice, as shown in Figure 2, following traditional
studies on sentencial paraphrase (e.g. madnanidorr2010).

Figure 1: pFN paraphrastic data generation pipeline

Figure 2: A word lattice showing all paraphrasing candidates output by the pFN
system for the sentence: _You contribution to Goodwill will mean more than you
may know_. Original targets are represented in the upper edges and
paraphrastic candidates are represented in the lower edges

pFN will perform _loose_ valence pattern matching, i.e., two valence patterns
will be considered matching as long as all their _core_ valence units333 the
syntactic realizations of the core frame elements do match. For example, the
valence patterns Earner.NP.Ext Earnings.NP.Obj and Earner.NP.Ext
Earnings.NP.Obj Time.AVP.Dep will be considered matching as the Time frame
element is peripheral in the Earner.NP.Ext Earnings.NP.Obj Time.AVP.Dep
valence pattern. This approach has the advantage of augmenting the pool of
candidates generated by the system when only a limited number of valence
patterns has been annotated for a given lexical unit and when those valence
patterns contain peripheral frame elements.

The number of possible paraphrastic sentences $N_{p}$ output by the pFN system
for a given sentence $s$ containing $N_{a}$ annotation sets is given by:

$N_{p}=\prod_{j}^{N_{a}}\binom{N_{c_{j}}+1}{1}-1$ (1)

where $N_{c_{j}}$ is the number of paraphrastic candidates generated by pFN
for the the $j$-th annotation set. $N_{p}$ simplifies to:

$N_{p}=\prod_{j}^{N_{a}}(N_{c_{j}}+1)-1$ (2)

For sentence 1.1, and without any filtering, pFN will therefore generate 47
paraphrastic sentences:

$N_{p}=2\cdot 2\cdot 1\cdot 1\cdot 3\cdot 4-1=47$ (3)

##### Robustness

Our approach for augmenting FrameNet annotation via paraphrastic sentences is
more robust than what previous approaches to automatic FrameNet data
augmentation have attempted, as it guarantees strong consistency with the
underlying FrameNet annotation framework. Indeed, past research have often
combined word-based semantic similarity with dependency-tree-based syntactic
similarity to project annotation to new data, be it on FrameNet
(furstenau2012), or PropBank (woodsend2014).

However, the challenge of such approaches is to be able to reproduce the
structure of FrameNet annotation. Consider first the following example:

[John]Buyer.NP.Ext bought [a car]Goods.NP.Obj this morning

Mary acquired a house yesterday Although both Commerce_buy.buy.v and
Getting.acquire.v lexical units share some apparent syntactic and semantic
similarities, FrameNet clearly distinguishes them and assigns them to
different frames with distinct frame elements in order to account explicitly
for the semantic specificity of the lexical unit buy.v which implies an
exchange of money for the thing acquired. Consider then the following example:

[I]Ingestor.NP.Ext ate [cheese]Ingestibles.NP.Obj

[I]Ingestor.NP.Ext drank [wine]Ingestibles.NP.Obj Although both eat.v and
drink.v lexical units share syntactic similarities, they appear somehow
slightly more distant semantically than buy.v and acquire.v. They are
nonetheless annotated within the same Ingestion frame in FrameNet, and are
both compatible with the Ingestor.NP.Ext Ingestibles.NP.Obj valence pattern.

The dilemma for systems combining word-based semantic similarity and
dependency tree-based syntactic similarity is therefore to be able to handle
_both_ eat/drink and buy/
acquire cases, by including one while excluding the other. However, a quick
test with a cosine similarity calculated on distributional representations
computed with word2vec444See Section 1 for details gives us a 0.57 similarity
score between _buy_ and _acquire_ and a 0.5 similarity score between _eat_ and
_drink_. We see directly how such a measure would lead to filtering out both
cases or including them both in the augmented data, given that the output
hierarchy of similarity is the inverse of that of FrameNet.

Although our approach does overcome those limitations, as it generates
artificial data fully compliant with FrameNet annotation _by construction_ ,
it still faces two major shortcomings:

  1. 1.

as previous approaches, and without filtering, it generates a huge number of
paraphrastic sentences, which tend to explode the training set beyond what is
computationally reasonable;

  2. 2.

due to FrameNet’s annotating dissimilar lexical units with similar valence
patterns, the pFN system may generate sentences which are unlikely to be seen
in a corpus, as:

######  Example 1.2.

? I ate wine

The key question is therefore whether or not the statistical classifier used
for performing argument identification is sensitive to lexical diversity at
the argument level – that is, to lexical co-ocurrence (as suggested by the
lexical features detailed in Figure 3) – which is what our paraphrastic data
augmentation brings. We hypothesized two different scenarios:

  1. 1.

the statistical classifier relies primarily on a latent representation of the
arguments, and is less sensitive to predicate-argument lexical configurations,
in which case we should be able to observe an improvement in global argument
identification scores when training on our augmented training set, even
without filtering;

  2. 2.

the statistical classifier is sensitive to predicate-argument lexical
configurations, to the point that bias introduced by unlikely lexical
configurations does not compensate for the lexical diversity brought at the
argument level. In this case we should observe a decrease in global argument
identification scores when training on our augmented training set without
filtering.

Following our first results which confirmed scenario 2 (see Chapter 5), we
decided to implement an extra layer of filtering in order to filter out
unlikely lexical configurations.

##### Filtering

The pFN pipeline includes two different kinds of filtering:

  1. 1.

annotation sets filtering: source annotation sets are filtered based on the
part of speech or number of tokens of their lexical units. Filtering out items
according to part of speech allows for evaluating the contribution of
paraphrastic data augmentation for each morphosyntactic category separately.
Filtering out multi-lexeme lemmas allows removing configurations not supported
by the Semafor parser which we use for argument identification;555 Semafor
does not support distant multi-lexeme lemmas such as phrasal verbs separated
by nouns or adjectives, e.g. “give the answer away”

  2. 2.

paraphrastic candidates filtering: paraphrastic candidates are filtered based
on the cosine similarity measure between their distributional representation
and the distributional representation of the source lexical unit.

The aforementioned _distributional representations_ refer to vectorial
representations of the meaning of words where the dimensions of the vectors
record contextual information about the co-occurrence of target words with
their neighboring words as observed in large text corpora. The distributional
representations used for paraphrastic candidates filtering are computed from
_predictive models_ and _count-based models_. Predictive models estimate word
vectors via a supervised task where the weights of the word vector are set to
maximize the probability of the contexts in which the word is observed in the
corpus. Count-based models compute word vectors by extracting co-occurrence
counts of target words and their contexts. The _cosine similarity_ measure
used is defined as the dot product of two vectors normalized by the product of
vectors length. Given two distributional representations
$\overrightarrow{u}=\langle u_{1},\dots,u_{n}\rangle$ and
$\overrightarrow{v}=\langle v_{1},\dots,v_{n}\rangle$, the cosine similarity
between $\overrightarrow{u}$ and $\overrightarrow{v}$ is formally defined as:

$cos(\overrightarrow{u},\overrightarrow{v})=\frac{\sum_{i}{u_{i}\cdot
v_{i}}}{||\overrightarrow{u}||\cdot||\overrightarrow{v}||}$ (4)

The semantic filter component of the paraphrastic candidates filter has three
possible configurations:

  1. 1.

random which selects $n$ random paraphrastic candidates in the list;

  2. 2.

word2vec which filters out paraphrastic candidates based on a word2vec
predictive model (mikolovetal2013) trained on the GoogleNews corpus;666See
details at https://code.google.com/archive/p/word2vec/

  3. 3.

dissect which filters out paraphrastic candidates using the dissect toolkit
(dissect2013) with two different models: a word2vec predictive model trained
on a concatenation of the ukWaC, English Wikipedia, and BNC corpora, and a
(reduced) count-based model generated from the same concatenated corpus (see
baronietal2014, for details).

Both word2vec and dissect configurations have two additional setups:

  1. 1.

top: which selects the top $n$ elements from a list of paraphrastic candidates
ranked in a decreasing order according to the cosine similarity between their
distributional representations and the distributional representation of the
original target;

  2. 2.

threshold which selects all paraphrastic candidates for which the cosine
similarity between their distributional representations and the distributional
representation of the original target is above a specified value.

####  2 Argument identification

For argument identification we rely on the log-linear model of dasetal2014,
modified by kshirsagaretal2015 to reduce training time and enable training on
exemplar sentences.

##### Model

The model performs argument span identification and argument role labeling
jointly: all possible spans are pre-identified with a rule-based algorithm and
the system then assigns a role to each span from the set of frame elements
belonging to the target predicate’s frame, to which has been added the null
role $\varnothing$ in case no role is to be assigned to the span.

Argument span identification is performed by identifying all continuous spans
in a sentence that contain a single word or comprise a valid subtree of a word
and all its descendants in the dependency parse produced by the dependency
parser. This bounds recall on argument identification as it covers only 80% of
arguments in the FrameNet dataset.

Argument role labeling is performed with a log-linear model as follows: Let
$x$ be a dependency-parsed sentence, $p$ the target predicate and $f$ the
frame evoked by $p$. Let $spans(x,p,f)$ be the set of candidate spans
previously identified. For each candidate argument $a\in spans(x,p,f)$ and
each role (frame element) $r$, a binary feature vector $\phi(a,x,p,f,r)$ is
extracted. Let $w$ be the weight matrix. Each argument $a$ is given a real-
valued score by a linear model:

$score_{w}(a|x,p,f,r)=w^{\top}\phi(a,x,p,f,r)$ (5)

##### Features

Figure 3 shows the list of feature templates used by the model. Every feature
template has a $\Circle$ version that does not take into account the role
being filled (so as to incorporate overall biases). The $\LEFTcircle$ symbol
indicates that the feature template also has a variant that is conjoined with
the frame element name being filled, and $\CIRCLE$ indicates that the feature
template additionally has a variant that is conjoined with both the frame
element name and the frame name.

Figure 3: Argument identification feature templates as of dasetal2014

##### Learning

The model is trained using a local objective where each span-role pair is
treated as an independent training instance. Consider the following squared
structured hinge loss function $L_{w}$ given for a training example $i$:

$L_{w}(i)=\bigg{(}\max_{a^{\prime}}(w^{\top}\cdot\phi(a^{\prime},x,p,f,r)+1\\{a^{\prime}\neq
a\\})-w^{\top}\cdot\phi(a,x,p,f,r)\bigg{)}^{2}$ (6)

The model parameters are learned by minimizing the $l_{2}$-regularized average
loss using the online optimization method AdaDelta (zeiler2012):

$w^{\star}=\operatorname*{arg\,min}_{w}\frac{1}{N}\sum_{i=1}^{N}L_{w}(i)+\frac{1}{2}\lambda\|w\|_{2}^{2}$
(7)

##### Decoding

Decoding is done using beam search which produces a set of k-best hypotheses
of sets of span-role pairs. The approach enforces multiple constraints
including the fact that a frame element label may be assigned to at most one
span and that spans of overt arguments must not overlap.

###  2 Experimental setup

####  1 Datasets

In this work we diverge from previous approaches in that we rely on the latest
FrameNet 1.7 data release instead of FrameNet 1.5. We split data in three
sets: train, dev, test for training, development and testing. For testing, we
use the same list of fulltext documents as dasetal2014 used for testing on
FrameNet 1.5. Using FrameNet 1.7 has the benefit of almost doubling the size
of the testing set, providing thereby a more robust set to test model
performances. Table 1 shows the number of sentences and annotation sets in the
train/dev/test setting on FrameNet 1.7. Table 2 shows the same metrics on
FrameNet 1.5 and 1.7 in a train/test setting where the content of the
development set is included in the training set. Additional data such as
counts and coverage metrics for lexical units, frame elements, valence units
and valence patterns are provided in Chapter 6.

Framenet 1.7 dataset metrics
---
| #sentences | #annotationsets
train (fulltext + exemplar) | 168 237 | 184 050
train (fulltext) | 2 804 | 16 215
dev | 887 | 5 715
test | 1 247 | 6 517
Table 1: FrameNet 1.7 datasets metrics with development and testing sets. The
training set does not include the development set Framenet 1.5 and 1.7
datasets metrics
---
| #sentences | #annotationsets
train FT 1.5 | 2 983 | 18 228
train FT 1.7 | 3 691 | 20 886
train FT+EX 1.5 | 150 735 | 168 357
train FT+EX 1.7 | 169 124 | 189 553
test 1.5 | 875 | 3 702
test 1.7 | 1 246 | 6 040
Table 2: Metrics comparison between FrameNet 1.5 and FrameNet 1.7 datasets
with no development set

####  2 Toolkits

Over the course of our research, we had to develop and entire toolkit so as to
replicate past results, implement our models and compare ourselves to previous
approaches. The following sections provide a short description of this
toolkit.

##### Querying valence patterns

As valence patterns were not accessible out-of-the-box in FrameNet, due to
their being deeply embedded in the structure of FrameNet XML and split into
lists of valence units, we had to implement a set of tools in order to be able
to efficiently search for valence patterns in the FrameNet dataset. For
performance reasons, we decided to rely on a NoSQL MongoDB database which
provided an optimal architecture with indexed valence units and valence
patterns against which FrameNet data could be queried directly. We named our
application to import FrameNet XML data to MongoDB NoFrameNet.777available at:
https://github.com/akb89/noframenet We then implemented a search engine API to
query FrameNet valence patterns in MongoDB. The API would receive HTTP GET
requests and return JSON-formatted FrameNet data. We named this API Valencer
888available at: https://github.com/akb89/valencer and published it as a
system demonstration during the COLING 2016 conference (see kabbach2016). To
be able to easily visualize the output of our API, we also implemented a web
client on top of the Valencer API. We named it myValencer 999available at:
https://github.com/akb89/myvalencer and published it at the eLex 2017
conference as a system demonstration (see kabbach2017).

##### From Semafor to Rofames

We forked the acl2015 branch of the Semafor parser of dasetal2014,
corresponding to the version of the parser used in kshirsagaretal2015. We re-
implemented the whole preprocessing pipeline to generate the train, dev and
test splits from FrameNet XML data as it was no longer available in that
version of Semafor. This enabled us to train and test on FrameNet 1.7 data in
addition to 1.5. We added a branch for the system where we removed the
hierarchy feature introduced by kshirsagaretal2015 in order to have a version
of Semafor corresponding to previous baselines but we kept the changes in
training as it reduced training time by about one order of magnitude and made
it possible to train on exemplar sentences as well as fulltext. We named our
forked version Rofames, the semordnilap of Semafor.

##### Generating paraphrastic examples and debugging Rofames

We implemented a Python wrapper, named pyFN, to handle FrameNet XML data in a
more systematic fashion. This proved particularly helpful as we discovered
that FrameNet XML data contained many corrupted entries (such as annotation
with partially missing label indexes), which bugged our paraphrastic data
augmentation system pFN (see Chapter 4 for details). In addition to pyFN and
pFN, we also implemented a debugging application named dFN to compute several
metrics such as coverage and precision/recall/$F_{1}$ scores per frame element
given FrameNet data and the model output by Rofames.

####  3 Hyperparameters

In all our experiments, Rofames is used with $\lambda=1e^{-6}$ and a beam of
100. Data are tokenized and part-of-speech tagged with NLP4J (choi2016) and
dependency-parsed with BMST (kiperwasser2016). The Valencer API throws
requests at a MongoDB database containing the exact same data as in the
training set. We experiment pFN configurations with semantic filtering with
word2vec predict and dissect reduced count models, all in random mode with
random values 2,3,4, in top mode 2,3,4 and in threshold mode 0.5 and 0.7. We
filter part of speech to experiment data augmentation on nouns only, verbs
only, adjectives only and nouns plus verbs.

##  Chapter 4 Baseline and Replication

In this chapter we report on our attempts to replicate several past results of
argument identification with our FrameNet 1.7 datasets. We also discuss
several crucial experimental considerations regarding the production of a
robust baseline for argument identification on FrameNet. In Section 1 we
report on results validating our preprocessing implementation in Rofames. In
Section 2 and Section 3, we discuss the impact of part-of-speech taggers and
dependency parsers on argument identification. Next, in Section 4 and Section
5 we report on our attempts at replicating results from (kshirsagaretal2015)
with exemplar data and the hierarchy feature. Finally, in Section 6 we
summarize our results and compile them into a list of recommendations
providing a robust baseline against which we will be able to compare our
paraphrastic data augmentation approach in Chapter 5.

###  1 Validating the preprocessing pipeline

As we re-implemented the preprocessing pipeline of Rofames, we first tested
the robustness of our implementation by replicating previous baseline results
based on the FrameNet 1.5 splits of dasetal2014. We trained Rofames on our
train split but tested on the test split of dasetal2014. Our train split
differed from that of dasetal2014 in that we applied strict filtering of
potentially erroneous annotation in order not to bias the parser: we removed
all annotation sets which contained missing or invalid start/end indexes
(except for null instantiations). Splits were tokenized using Robert
MacIntyre’s sed script, part-of-speech tagged with MXPOST (mxpost1996) and
dependency-parsed with the MSTParser (mst2006), following the original
pipeline of dasetal2014. For scoring, we used the two evaluation scripts
mentioned in Section 3:

  1. 1.

SEM: the SemEval 2007 evaluation script as-is (fnsemeval2007)

  2. 2.

ACL: the modified SemEval 2007 evaluation script used by kshirsagaretal2015,
which does not give extra credits for (gold) frames and therefore better
accounts for argument identification performances (refer to Section 3 for
details)

Scores, detailed in Table 1, validate our implementation. Results output by
the SEM scoring script are consistent with the baseline reported in
(rothlapata2015). Regarding results output by the ACL script, we found a 1
point $F_{1}$ gain over the baseline reported in (kshirsagaretal2015). Both
approaches used the train and test splits of dasetal2014. Note that we found
the decrease in precision to be consecutive to the increase in recall.

Validating Rofames preprocessing pipeline on FrameNet 1.5
---
SEM | P | R | F1
#1 ROFAMES + MXPOST + MST | 76.9 | 74.4 | 75.6
rothlapata2015 | 78.4 | 73.1 | 75.7
ACL | P | R | F1
#1 ROFAMES + MXPOST + MST | 64.3 | 56.4 | 60.1
kshirsagaretal2015 | 65.6 | 53.8 | 59.1
Table 1: Argument identification results with gold frames output by the SEM
and ACL scoring scripts. Train and test splits are generated from FrameNet 1.5
fulltext XML data. Train split is produced by Rofames, test split is that of
dasetal2014 as-is. All data are part-of-speech tagged with MXPOST and
dependency parsed with the MSTParser

###  2 Impact of part-of-speech taggers

Past studies on frame semantic parsing have used various preprocessing
toolkits. In this section, we focus on the impact of part-of-speech tagger
performances on argument identification, comparing MXPOST (mxpost1996)
originally used in (dasetal2014) with NLP4J (choi2016) used in (roth2016).
Results, reported in Table 2, show a systematic $F_{1}$ gain in argument
identification when relying on the NLP4J part-of-speech tagger. Improvements
gains vary from 0.3 $F_{1}$ in test to 0.7 $F_{1}$ in dev, with a systematic
improvement in recall. In subsequent experiments, we rely systematically on
NLP4J for part-of-speech tagging.

Comparing pos taggers on FrameNet 1.7 |
---|---
|  dev | P | R | F1
#7 | MXPOST + MST | 56.3 | 51.0 | 53.5
#8 | NLP4J + MST | 56.6 | 52.0 | $54.2^{\dagger}$
|  test | P | R | F1
#7 | MXPOST + MST | 57.8 | 54.1 | 55.9
#8 | NLP4J + MST | 57.7 | 54.7 | $56.2^{\ast}$
Table 2: Argument identification results with gold frames. Train, dev and test
splits are part-of-speech tagged with MXPOST or NLP4J and dependency parsed
with the MSTParser. Statistical significance is indicated by $\dagger$ for
$p<0.05$ and $\ast$ for $p<0.10$

###  3 Impact of dependency parsers

Similarly, past studies on frame semantic parsing have used various dependency
parsers. Here, we compare the MSTParser originally used dasetal2014 to the
BIST parser (kiperwasser2016) in two of its variants: the transition-based
BARCH parser and the graph-based BMST parser. The BIST parser was originally
used by roth2016 for frame semantic parsing. Results, presented in Table 3,
show a systematic improvement on argument identification when preprocessing
data with the BIST parser rather than the MSTParser. $F_{1}$ gains vary from
1.3 to 2.6 depending on the scoring set and the BIST parser variant used. In
subsequent experiments, we rely systematically on the BMST variant of the BIST
parser for dependency parsing. We favor it over the BARCH variant as it is
faster and provides only slighly lower performances.

Comparing dependency parsers on FrameNet 1.7
---
|  dev | P | R | F1
#8 | NLP4J + MST | 56.6 | 52.0 | 54.2
#9 | NLP4J + BMST | 59.4 | 52.6 | 55.8
#10 | NLP4J + BARCH | 58.9 | 52.5 | 55.5
|  test | P | R | F1
#8 | NLP4J + MST | 57.7 | 54.7 | 56.2
#9 | NLP4J + BMST | 60.5 | 55.6 | 58.0
#10 | NLP4J + BARCH | 60.9 | 56.8 | 58.8
Table 3: Argument identification results with gold frames. Train, dev and test
splits are part-of-speech tagged with NLP4J and dependency parsed with the
MSTParser or the BIST parser in its BMST or BARCH variant

###  4 Impact of exemplar annotation

In the next two sections, we report on our attempts to replicate results from
kshirsagaretal2015. We first include exemplar data in the training set.
Results, displayed in Table 4, show a systematic 1.2 $F_{1}$ gain over
training on fulltext data only. This is, however, significantly lower that the
2.8 $F_{1}$ gain on FrameNet 1.5 data reported by kshirsagaretal2015. In all
the following sections, we found $F_{1}$ gains on our FrameNet 1.7 splits to
be systematically lower that those reported in (kshirsagaretal2015). We
hypothesize that those may be due to the increased size of the development and
testing sets, the reduced size of the training set when testing with the
development set, and the increased robustness of our train, dev and test
splits which do not contain duplicate annotation sets likely to bias scores,
and no annotation sets with missing or invalid indexes.

Adding exemplar data on FrameNet 1.7
---
|  dev | P | R | F1
#9 | FT | 59.4 | 52.6 | 55.8
#12 | FT + EX | 58.4 | 55.7 | 57.0
|  test | P | R | F1
#9 | FT | 60.5 | 55.6 | 58.0
#12 | FT + EX | 59.5 | 58.9 | 59.2
Table 4: Argument identification results with gold frames. Rofames is trained
on fulltext (FT) and exemplar (EX) data

###  5 Impact of the hierarchy feature

Additionally, still following kshirsagaretal2015, we trained Rofames on
FrameNet 1.7 data with the hierarchy feature comprising Inheritance and
SubFrame relations. Results, displayed in Table 5, show a $F_{1}$ gain of 0.5
to 0.7 when using the hierarchy feature, again lower than the 1.3 gain
reported by kshirsagaretal2015 on FrameNet 1.5. When incorporating both the
hierarchy feature and exemplar data to the setup, we show a 2.1 to 2.4 $F_{1}$
gain, again significantly lower than the 4 points gain reported by
kshirsagaretal2015.

Adding the hierarchy feature on FrameNet 1.7
---
|  dev | P | R | F1
#9 | FT | 59.4 | 52.6 | 55.8
#13 | FT + H | 60.0 | 53.5 | $56.5^{\dagger}$
#14 | FT + H + EX | 59.2 | 57.2 | $58.2^{\ddagger}$
|  test | P | R | F1
#9 | FT | 60.5 | 55.6 | 58.0
#13 | FT + H | 61.0 | 56.3 | $58.5^{\ast}$
#14 | FT + H + EX | 60.3 | 60.0 | $60.1^{\ddagger}$
Table 5: Argument identification results with gold frames. Rofames is trained
on fulltext (FT) and exemplar (EX) data, with and without the hierarchy
feature (H) of kshirsagaretal2015. Statistical significance is indicated by
$\ddagger$ for $p<0.01$, $\dagger$ for $p<0.05$ and $\ast$ for $p<0.10$

###  6 Baseline

All subsequent experiments in Chapter 5 rely on the baseline scores shown in
Table 6, computed from FrameNet 1.7 splits. We built on results of previous
sections in order to produce the most robust baseline:

  * •

for each train/dev/test split, all annotation sets with missing or invalid
start/end indexes are removed (except for null instantiations);

  * •

all duplicate annotation sets and sentences are removed from the test split;

  * •

incomplete annotation – targets with annotated frames but no annotated frame
elements – are removed from the training set;

  * •

data are part-of-speech tagged with the NLP4J tagger (choi2016);

  * •

data are dependency-parsed with the BIST BMST parser (kiperwasser2016).

Argument identification baseline on FrameNet 1.7
---
|  | P | R | F1
#9 |  dev | 59.4 | 52.6 | 55.8
#9 |  test | 60.5 | 55.6 | 58.0
Table 6: Baseline scores for argument identification with gold frames on dev
and test splits, scored with the ACL script. Train, dev and test splits are
part-of-speech tagged with NLP4J and dependency parsed with the BIST BMST
parser

##  Chapter 5 Results

In this chapter, we report on the contribution of our paraphrastic data
augmentation model described in Chapter 3 to argument identification,
evaluated against the baseline described in Chapter 4. Our preliminary
results, obtained by augmenting nouns and verbs without filtering paraphrastic
candidates (see Section 1), confirm the need for additional (semantic)
filtering: both approaches lead to a systematic decrease in $F_{1}$ score
compared to the baseline, despite the significant increase in size of the
datasets produced, almost one order of magnitude higher than the baseline in
the case of augmented verbs.

Our semantic filtering approach, detailed in Section 3, shows mixed results:
on augmented nouns and adjectives, it yields no significant improvements, but
it does not lead to significant decreases in performance either. On augmented
verbs, it does yield limited improvements, in the order of .5 $F_{1}$ gain in
the best scenario (XP#41). However, none of the reported improvements prove
statistically significant in the end, with $p$ values systematically above
0.1, except for XP#29 ($p=0.08$).

If, overall, our results tend to demonstrate the smartness of the Rofames
parser, the marginal improvements shown upon training on augmented verbs,
albeit not statistically significant, suggest that there could be specific
patterns that Rofames fails to extract from gold data alone.

Similarly, our results tend to support the robustness of our paraphrastic data
augmentation approach: almost none of the approaches relying on filtering of
paraphrastic candidates lead to a decrease in performances, suggesting that
the core pFN logic, combined with the right candidates filtering, rarely
introduces noise in the data.

However, those results should again lead us to tread carefully, as no-
filtering approaches do show significant decreases in performance, suggesting
that paraphrastic approaches relying on FrameNet annotation logic alone have a
natural tendency to generate (noisy) unlikely predicate-argument lexical
configurations, to which the Rofames parser is not robust.

It would be tempting at first to conclude from our results that semantic
filtering of paraphrastic candidates, motivated by semantic proximity measures
computed from distributional representations of words, is capable of
compensating for the natural tendency of pFN to generate unlikely lexical
configurations. However, semantic filtering approaches do not actually prove
any better than random filtering approaches, as shown in Section 5.

What is to be made of the observations hereof? We postulate two
interpretations, motivated by two distinct phenomena: (1) there exists a
latent variable, not necessarily semantically motivated, or at least not in
ways distributional models can capture, that determines, among the set of
unfiltered pFN candidates, what is noise from what is not; or (2) data
generated by pFN are actually always somehow noisy, but the Rofames parser is
robust enough that it can compensate for the bias introduced by noisy
artificial data, provided that those noisy data remain limited in size.

All in all, the results and preliminary explanations presented in this chapter
do not suffice to arbitrate, in light of the intricacies that arise. Further
systematic analysis of datasets and errors are required in order to properly
guide reflection. Those analysis will be presented in Chapter 6, and further
discussed in Chapter 7.

###  1 No filtering setup

In the first two experiments we augmented the FrameNet fulltext training set
with pFN-generated data produced with minimal filtering: we kept only noun and
verb annotation sets and did not apply any semantic filtering on candidates.
Our preliminary results for augmented nouns and verbs are shown in Table 1. In
all subsequent tables, FT refers to FrameNet fulltext data and EX to FrameNet
exemplar data.

No filtering setup
---
| train | filters |  |  |  |  |
XP | FN | pFN | POS | #sent | #anno | P | R | F1
baseline | FT | – | – | 2 804 | 15 389 | 59.4 | 52.6 | 55.8
#16 | FT | FT | N | 10 386 | 34 734 | 60.3 | 51.5 | 55.6
#18 | FT | FT | V | 36 234 | 126 436 | 60.4 | 50.9 | 55.3
Table 1: Argument identification results with a model trained on FrameNet
fulltext (FT) augmented with pFN data generated from FT filtered by part-of-
speech (POS). Results are given for nouns (N) and verbs (V) without additional
filtering on pFN candidates

Results confirm the second scenario formulated in Section 1: the statistical
classifier used for argument identification seems to be sensitive to
predicate-argument lexical configurations, to the point that bias introduced
by unlikely lexical configurations does not compensate for the lexical
diversity brought at the argument level. Note, in addition, that the size of
the training set increases significantly with the paraphrastic data
augmentation approach, due to the number of paraphrastic candidates extracted
by the valence pattern matching algorithm.

###  2 Introducing multi-tokens filtering

The Rofames system fails to process cases where annotated lexical units
contain non-continuous tokens, such as phrasal verbs separated by a noun (e.g.
_He gave the answer away_). We therefore tried and filtered out all multi-
token targets and candidates to see whether or not it could positively impact
the performances of the classifier. Results, displayed in Table 2, show a
marginal difference compared to the no-filtering approach. We nonetheless
applied mutli-tokens (or _multiword expressions_) filtering in all subsequent
experiments in order to limit the pool of candidates and generate higher
quality artificial data.

Multi-tokens filtering
---
| train | filters |  |  |  |  |
XP | FN | pFN | POS | MWE | #sent | #anno | P | R | F1
baseline | FT | – | – | – | 2 804 | 15 389 | 59.4 | 52.6 | 55.8
#18 | FT | FT | V | false | 36 234 | 126 436 | 60.4 | 50.9 | 55.3
#19 | FT | FT | V | true | 34 119 | 119 418 | 60.9 | 50.5 | 55.2
Table 2: Impact of multi-tokens filtering (MWE = true) on argument
identification with a model trained on FrameNet fulltext (FT) augmented with
pFN data generated from verbs

###  3 Introducing semantic filtering

####  1 Semantic filtering on nouns

Following the observations of Section 1, we first experimented on filtering
noun candidates with a word2vec semantic filter. Overall results, displayed in
Table 3, show a marginal, though not statistically significant ($p=0.3$),
improvement in one scenario (XP#21), and a systematic decrease in recall
otherwise, suggesting that the pFN system mostly introduced noise in the
training set.

Semantic filtering on nouns
---
| train | filters |  |  |  |  |
XP | FN | pFN | POS | MWE | SEM | #sent | #anno | P | R | F1
baseline | FT | – | – | – | – | 2 804 | 15 389 | 59.4 | 52.6 | 55.8
#16 | FT | FT | N | – | – | 10 386 | 34 734 | 60.3 | 51.5 | 55.6
#20 | FT | FT | N | – | random-3 | 6 615 | 24 209 | 59.7 | 52.0 | 55.6
#21 | FT | FT | N | – | top-3 | 6 537 | 24 050 | 60.4 | 52.0 | 55.9
#22 | FT | FT | N | – | threshold-0.5 | 3 021 | 15 666 | 59.2 | 52.5 | 55.7
#23 | FT | FT | N | – | threshold-0.7 | 2 844 | 15 434 | 60.0 | 52.2 | 55.8
Table 3: Impact of various word2vec semantic filtering of pFN candidates on
argument identification with a model trained on FrameNet fulltext (FT)
augmented with pFN data generated from nouns (N)

####  2 Semantic filtering on adjectives

Similar observations apply for adjectives, which exhibit not significant
improvements overall, regardless of the filtering configuration used (see
Table 4).

Semantic filtering on adjectives
---
| train | filters |  |  |  |  |
XP | FN | pFN | POS | MWE | SEM | #sent | #anno | P | R | F1
baseline | FT | – | – | – | – | 2 804 | 15 389 | 59.4 | 52.6 | 55.8
#33 | FT | FT | A | – | random-3 | 6 000 | 21 811 | 59.7 | 52.3 | 55.8
#34 | FT | FT | A | – | top-3 | 5 984 | 21 799 | 59.5 | 52.6 | 55.8
#35 | FT | FT | A | – | threshold-0.5 | 3 490 | 16 221 | 59.4 | 52.3 | 55.6
Table 4: Impact of various word2vec semantic filtering of pFN candidates on
argument identification with a model trained on FrameNet fulltext (FT)
augmented with pFN data generated from adjectives (A)

####  3 Semantic filtering on verbs

Experiments with semantic filtering on verbs, reported in Table 5, show a
relative increase in recall over the baseline, especially compared to no-
filtering approaches. Overall, results tend to show that filtering candidates
by keeping those _closest semantically_ to the original target is beneficial,
especially when keeping two to three candidates per target only. No
significant different is observed between _top_ and _threshold_ filtering. The
most interesting experiment is probably XP#30 which shows a .4 $F_{1}$ gain
over the baseline with a less that doubled dataset. Note, however, that none
of the experiments proved statistically significant ($p>0.1$) and that the
best improvements in precision seen in XP#31 are incidental to the decrease in
recall.

Semantic filtering on verbs
---
| train | filters |  |  |  |  |
XP | FN | pFN | POS | MWE | SEM | #sent | #anno | P | R | F1
bas. | FT | – | – | – | – | 2 804 | 15 389 | 59.4 | 52.6 | 55.8
#19 | FT | FT | V | true | – | 34 119 | 119 418 | 60.9 | 50.5 | 55.2
#30 | FT | FT | V | true | top-2 | 8 213 | 28 772 | 60.2 | 52.8 | 56.2
#25 | FT | FT | V | true | top-3 | 11 099 | 37 491 | 60.1 | 52.7 | 56.1
#31 | FT | FT | V | true | top-4 | 13 548 | 45 227 | 61.1 | 52.0 | 56.2
#26 | FT | FT | V | true | threshold-0.5 | 4 720 | 18 826 | 59.2 | 52.7 | 55.8
#27 | FT | FT | V | true | threshold-0.7 | 2 969 | 15 568 | 59.7 | 52.5 | 55.9
Table 5: Impact of various word2vec semantic filtering of pFN candidates on
argument identification with a model trained on FrameNet fulltext (FT)
augmented with pFN data generated from verbs (V). None of the improvements
reported proved statistically significant ($p>0.1$)

####  4 Semantic filtering on nouns and verbs

We also attempted to augment data in two dimensions at once, namely nouns and
verbs, in order to increase lexical diversity and potentially bring new
information to the statistical classifier. Experiments were all performed with
a semantic filter on top, to reduce the pool of candidates, for both quality
and feasability reasons. Results, displayed in Table 6 show a significant drop
in recall (XP#32) in a top-n setting, with an explosion of the size of the
augmented dataset generated, suggesting very noisy and low quality data.
Results in a threshold setting (XP#36) did not bring statistically significant
improvements, and generated a limited number of paraphrastic sentences.

Semantic filtering on nouns and verbs
---
| train | filters |  |  |  |  |
XP | FN | pFN | POS | MWE | SEM | #sent | #anno | P | R | F1
bas. | FT | – | – | – | – | 2 804 | 15 389 | 59.4 | 52.6 | 55.8
#32 | FT | FT | N+V | true | top-3 | 34 367 | 153 293 | 60.4 | 49.4 | 54.3
#36 | FT | FT | N+V | true | threshold-0.5 | 5 118 | 19 807 | 60.0 | 52.5 | 56.0
Table 6: Impact of various word2vec semantic filtering of pFN candidates on
argument identification with a model trained on FrameNet fulltext (FT)
augmented with pFN data generated from nouns and verbs (N +V). In both
settings (top and threshold), $p>0.1$

####  5 Comparing random vs. top filtering

We then tried to quantify the real benefits brought by a semantic filter based
on proximity measures between two distributional representations, compared to
a control filter selecting candidates randomly. The top-n filter, choosing the
$n$ paraphrastic candidates closest to the original target from a given list
output by pFN, did not actually perform better than the random-n filter
choosing n candidates randomly (see Table 7).

Random vs. top semantic filtering
---
| train | filters |  |  |  |  |
XP | FN | pFN | POS | MWE | SEM | #sent | #anno | P | R | F1
#28 | FT | FT | V | true | random-2 | 8 278 | 28 885 | 60.3 | 52.6 | 56.1
#30 | FT | FT | V | true | top-2 | 8 213 | 28 772 | 60.2 | 52.8 | 56.2
#24 | FT | FT | V | true | random-3 | 11 188 | 37 632 | 60.4 | 52.5 | 56.2
#25 | FT | FT | V | true | top-3 | 11 099 | 37 491 | 60.1 | 52.7 | 56.1
#29 | FT | FT | V | true | random-4 | 13 563 | 45 143 | 60.9 | 52.2 | 56.2
#31 | FT | FT | V | true | top-4 | 13 548 | 45 227 | 61.1 | 52.0 | 56.2
Table 7: Impact of top-n word2vec semantic filters _versus_ n-random filters
for pFN data generated from verbs. Improvements reported are not statistically
significant ($p>0.1$ except for XP#29 with $p=0.08$)

####  6 Comparing predictive and count-based distributional models

The previous experiment comparing _top_ and _random_ semantic filters raised
the question of the quality of the underlying word2vec distributional model
and toolkit used to measure semantic proximity between paraphrastic candidates
and targets. Past research (baronietal2014) have shown that the choice of
distributional model can significantly impact performances on tasks such as
_semantic relatedness_ and _synonym detection_. We therefore experimented with
two additional distributional models and toolkits, namely dissect-pred and
dissect-rcount, on top of word2vec.111see Section 1 for details However, our
experimental results, displayed in Table 8, showed only marginal differences
across models and toolkits.

Predictive vs. count-based distributional models
---
XP | POS | MWE | SEM | #sent | #anno | P | R | F1
#25 | V | true | word2vec | top-3 | 11 099 | 37 491 | 60.1 | 52.7 | 56.1
#37 | V | true | dissect-pred | top-3 | 11 006 | 37 259 | 59.8 | 52.9 | 56.1
#38 | V | true | dissect-rcount | top-3 | 11 026 | 37 281 | 60.4 | 52.3 | 56.0
#26 | V | true | word2vec | threshold-0.5 | 4 720 | 18 826 | 59.2 | 52.7 | 55.8
#39 | V | true | dissect-pred | threshold-0.5 | 3 758 | 16 855 | 59.5 | 52.7 | 55.9
#40 | V | true | dissect-rcount | threshold-0.5 | 6 350 | 23 656 | 60.1 | 52.4 | 56.0
Table 8: Comparing predictive and count-based distributional models for
paraphrastic candidates filtering. Impact is measured on argument
identification. word2vec is the traditional predictive model trained on Google
News and used with the word2vec toolkit. dissect-pred is a word2vec predictive
model extracted from a different dataset than Google News, and used with the
dissect toolkit. dissect-rcount is a count-based model extracted from the same
dataset than dissect-pred and used with the dissect toolkit

####  7 Measuring the impact of exemplar data

Finally, we experimented with a pFN model performing valence pattern matching
on the FrameNet fulltext plus exemplar dataset. Our purpose was to analyze
whether such a model could actually beneficially incorporate information
available in the exemplar dataset, or, failing that, be able to create an
augmentated artificial dataset of higher quality given that paraphrastic
candidates would be chosen from a larger pool. Results, displayed in Table 9,
showed no significant improvements over the pFN model based on fulltext alone.
Note that improvements reported in XP#30 and XP#41 did not prove statistically
significant ($p>0.1$), while improvements on XP#12 did ($p<0.01$). This shows
that training on gold fulltext and exemplar data leads to much more robust
improvements than training on our artificial dataset.

Impact of exemplar data
---
| train | filters |  |  |  |  |
XP | FN | pFN | POS | MWE | SEM | #sent | #anno | P | R | F1
base. | FT | – | – | – | – | 2 804 | 15 389 | 59.4 | 52.6 | 55.8
#12 | FT+EX | – | – | – | – | 168 237 | 184 050 | 58.4 | 55.7 | 57.0
#30 | FT | FT | V | true | top-2 | 8 213 | 28 772 | 60.2 | 52.8 | 56.2
#41 | FT | FT+EX | V | true | top-2 | 14 588 | 53 536 | 60.3 | 52.9 | 56.3
Table 9: Impact of using exemplar data for training Rofames (XP#12), or for
generating artificial data via pFN (XP#41). A top-2 word2vec semantic filter
is applied to limit the size of the paraphrastic dataset generated by pFN.
Improvements in XP#30 and XP#41 are not statistically significant ($p>0.1$),
while improvements in XP#12 are ($p<0.01$)

##  Chapter 6 Error analysis

In this chapter we discuss both _quantitative_ and _qualitative_ performances
of the statistical classifier used for argument identification. We focus on
trying to identify (linguistic) phenomena and/or specific (linguistic)
patterns which may account for the performances reported in Chapter 5. Our
analysis is articulated around two axis: in Section 1, we present an
exhaustive error analysis of the baseline system, proposing several angles to
tackle the key question of what constitutes _easy_ and _hard_ frame elements,
from the perspective of statistical learning. In Section 2 we discuss the
respective contributions of incorporating exemplar and paraphrastic data into
the training set, both in terms of coverage and how it affects data
distribution. In the following sections we use _exemplar approach_ to refer to
the approach consisting in training on fulltext and exemplar data, and
_paraphrastic approach_ to refer to the approach corresponding to experiment
30, augmented on verbs with a word2vec top-2 filter.

###  1 Baseline error analysis

The purpose of this section is to discuss both qualitative and quantitative
performances of the statistical classifier at a more fined-grained level than
the micro-averaged results presented in Chapter 5. We report on the robustness
of the parser to frequency effects (Section 1), analyze the impact of _frame
elements bearing targets_ on argument identification (Section 2), and discuss
both lexical (Section 3) and syntactic challenges (Section 4) posed to
statistical learning.

####  1 Frequency effects on Frame Elements

The first observation that can be made when looking at precision, recall and
$F_{1}$ measure _per_ frame element, is that the Rofames parser seems
relatively robust to frequency effects. Indeed, Table 1 shows that, among the
ten most frequent frame elements in the development set, $F_{1}$ measures do
not seem to correlate well with frequency and/or distribution in either
training or development sets. The lack of pattern is confirmed when
classifying frame elements by $F_{1}$ measure, such as in Table 3 or Table 3:
we find certain frequent frame elements to have relatively low $F_{1}$ scores,
such as Descriptor, Place or Figure, and conversely, certain less frequent
frame elements to have higher $F_{1}$ scores, such a Count, Part, or Material.

Top 10 Most frequent FEs
---
| train | dev | P | R | F1
Agent | 1 391 | 347 | 68.5 | 64.6 | 66.5
Entity | 1 204 | 318 | 59.4 | 55.7 | 57.5
Locale | 957 | 317 | 87.3 | 89.0 | 88.1
Weapon | 608 | 308 | 67.8 | 66.2 | 67.0
Time | 791 | 275 | 54.3 | 34.5 | 42.2
Theme | 796 | 232 | 56.2 | 56.5 | 56.3
Message | 541 | 215 | 60.3 | 53.0 | 56.4
Event | 670 | 211 | 44.9 | 31.3 | 36.9
Speaker | 598 | 201 | 72.0 | 71.6 | 71.8
Descriptor | 527 | 172 | 35.3 | 14.0 | 20.0
Table 1: Top 10 most frequent frame elements, with occurrence counts in
training and development sets

Top 10 Best FEs
---
| train | dev | P | R | F1
Number | 207 | 59 | 96.6 | 96.6 | 96.6
Substance | 239 | 125 | 91.7 | 88.8 | 90.2
Locale | 957 | 317 | 87.3 | 89.0 | 88.1
Person | 304 | 70 | 84.9 | 88.6 | 86.7
Count | 164 | 82 | 73.5 | 74.4 | 73.9
Part | 116 | 72 | 73.2 | 72.2 | 72.7
Unit | 382 | 146 | 64.2 | 82.2 | 72.1
Speaker | 598 | 201 | 72.0 | 71.6 | 71.8
Material | 106 | 61 | 84.4 | 62.3 | 71.7
Cognizer | 537 | 129 | 72.5 | 67.4 | 69.9

Table 2: Top 10 best frame elements among the top 50 most frequent frame
elements in dev set, ranked by $F_{1}$ score

Top 10 Worst FEs
---
| train | dev | P | R | F1
Name | 246 | 54 | 27.3 | 11.1 | 15.8
Manner | 251 | 89 | 29.4 | 11.2 | 16.3
Means | 144 | 54 | 42.9 | 11.1 | 17.6
Descriptor | 527 | 172 | 35.3 | 14.0 | 20.0
Place | 354 | 99 | 47.6 | 20.2 | 28.4
Figure | 464 | 83 | 31.0 | 32.5 | 31.8
Cause | 231 | 53 | 43.8 | 26.4 | 32.9
State_of_affairs | 145 | 153 | 42.4 | 27.5 | 33.3
Use | 310 | 112 | 40.7 | 29.5 | 34.2
Hypothetical_event | 215 | 57 | 50.0 | 28.1 | 36.0

Table 3: Top 10 worst frame elements among the top 50 most frequent frame
elements in dev set, ranked by $F_{1}$ score

####  2 Frame elements bearing targets

Looking closer at Table 3, we see that a limited number of frame elements,
such as Number, Substance, Locale and Person, have significantly higher
$F_{1}$ scores, systematically above 86% when the overall $F_{1}$ on the top
50 most frequent frame elements is at 56.2% and the average $F_{1}$ measure at
52.1%. Those frame elements have in common to be realized by their respective
targets, in what is called _frame elements bearing targets_ configurations, as
in:

######  Example 1.1.

These are surrounded by areas of limestone formations, scrub and grassland,
coral cliffs, and fine [sand]Substance beaches

In example 1.1, _sand_ is both the target evoking the Shapes frame and the
frame element Substance. Such configurations appear particularly easy for the
parser to acquire as it requires learning a correlation between a frame
element, a frame and a frame-evoking target without special considerations for
the frame element spans or syntactic realization. To better account for such
cases we introduce the febar ratio, which, for a given frame element, is
defined as:

$\texttt{febar}(f_{e})=\frac{N_{febc}}{N_{c}}$ (1)

where $N_{febc}$ is the number of frame element bearing configurations for the
frame element $f_{e}$ and ${N_{c}}$ is the total number of configurations for
the frame element $f_{e}$.

As shown in Table 5 and Table 5, we find the febar ratio to correlate nicely
with the $F_{1}$ score of a given frame element: the higher the febar ratio,
the more a given frame element will be realized in a frame element bearing
configuration, and the easier it will be for the parser to predict it. There
are three notable exceptions which constitute interesting cases: the Count and
Speaker frame elements which have high $F_{1}$ scores but low febar ratios,
meaning that they are easy to predict despite being realized _outside_ of
their respective targets; and the Weapon frame element which has a high febar
ratio but a $F_{1}$ significantly lower than frame elements with comparable
febar ratios, by almost 20 points of $F_{1}$ score. Those examples actually
illustrate interesting lexico-syntactic patterns that we will discuss in the
following sections.

Top 10 FEs by $F_{1}$
---
| febar | P | R | F1
Number | 100.0 | 96.6 | 96.6 | 96.6
Substance | 95.2 | 91.7 | 88.8 | 90.2
Locale | 96.5 | 87.3 | 89.0 | 88.1
Person | 92.9 | 84.9 | 88.6 | 86.7
Count | 1.2 | 73.5 | 74.4 | 73.9
Part | 34.7 | 73.2 | 72.2 | 72.7
Unit | 81.5 | 64.2 | 82.2 | 72.1
Speaker | 1.5 | 72.0 | 71.6 | 71.8
Material | 54.1 | 84.4 | 62.3 | 71.7
Cognizer | 9.3 | 72.5 | 67.4 | 69.9

Table 4: Top 10 best frame elements among the top 50 most frequent frame
elements in dev set, with febar ratio and ranked by $F_{1}$ score

Top 10 FEs by febar
---
| febar | P | R | F1
Number | 100.0 | 96.6 | 96.6 | 96.6
Locale | 96.5 | 87.3 | 89.0 | 88.1
Substance | 95.2 | 91.7 | 88.8 | 90.2
Weapon | 94.5 | 67.8 | 66.2 | 67.0
Person | 92.9 | 84.9 | 88.6 | 86.7
Unit | 81.5 | 64.2 | 82.2 | 72.1
Origin | 67.6 | 56.4 | 62.0 | 59.1
Leader | 63.0 | 64.3 | 66.7 | 65.5
Material | 54.1 | 84.4 | 62.3 | 71.7
Text | 45.1 | 71.1 | 52.9 | 60.7

Table 5: Top 10 frame elements among the top 50 most frequent frame elements
in dev set, ranked by febar ratio

####  3 Lexical challenges

##### Lexical diversity

As mentioned in Chapter 2, lexical coverage and out-of-vocabulary effects play
a determining role in frame semantic parsing in general. On argument
identification, the impact of lexical coverage is probably best exemplified by
the Name frame element, which covers mostly proper nouns, as in Sentence 1.2,
and as such is typically affected by out-of-vocabulary effects.

######  Example 1.2.

Your Georgia O’Keeffe membership, please note, includes free admission to all
art-related activities of the IMA’s [Young Friends of Art]Name group

As shown in Table 3, the Name frame element has the lowest performances among
all frame elements predicted in the development set, with a $F_{1}$ score of
15.8%, significantly below both micro-averaged (56.2%) and averaged (52.1%)
$F_{1}$ scores on all frame elements. The task of identifying Name labels is
actually a harder task than mere Named Entity Recognition, considering that
Name frame elements can be realized in various syntactic patterns, beyond noun
phrases, as in:

######  Example 1.3.

The island [of Jamaica]Name will be near the top of the list for anyone
planning an idyllic holiday getaway

The difficulty posed by the Name frame element is encompassed in the larger
phenomenon of _lexical diversity_ : if a frame element is realized in a wide
range of lexical patterns, the frame element may prove harder to predict,
especially if it is realized in a lexical pattern that has not been previously
seen in the training set. This is exactly what is happening for the Name frame
element: as there is a multitude of possible lexical patterns that can be
labeled with Name, they will prove almost impossible to predict by the parser
if previously unseen in training, as there will be no way for the parser to
compensate for its lack of lexical knowledge.

##### Compensating lexical diversity

However, lexical diversity itself is not enough to characterize the difficulty
to predict a given frame element. Indeed, high lexical diversity effects can
be compensated by systematic syntactic patterns. In other words, a frame
element which is realized lexically in a wide range of patterns may prove easy
to predict as long as it is realized in clear and systematic syntactic
patterns. This is typically the case of frame elements such as Agent or
Speaker, which are overwhelmingly realized as subjects in nouns phrases
(NP.Ext) – 75% of the time for Agent and 85% of the time for Speaker
respectively. Moreover, the NP.Ext syntactic pattern has the benefit of
translating well into a single arc (_subj_) between the predicate and the
argument, in the dependency tree produced by the syntactic parser. Therefore,
the statistical classifier used for argument identification can adequately
learn a correlation between certain predicates and the arguments connected to
them by a _subj_ arc in order to properly predict frame element labels such as
Agent or Speaker. Such phenomena explain why $F_{1}$ scores in frame elements
such as Speaker (71.8%) can be significantly higher than the micro-averaged
and averaged $F_{1}$ measures on all frame elements, despite their low
respective febar ratio.

Such compensating effects become all the more obvious when syntactic
representations between output dependencies and FrameNet valence units do not
align. In such cases, the statistical classifier clearly fails to properly
classify corresponding spans with the adequate frame element label, as in:

######  Example 1.4.

The Defense Department has awarded [BioPort Corporation]Agent a contract to
manufacture, test, bottle and store [anthrax vaccine]Theme, the company has
announced.

In this particular case _BioPort Corporation_ is labeled by the syntactic
parser as being in an _obj_ – and not _subj_ – dependency to the _awarded_ –
and not _store_ – predicate, while it is marked as NP.Ext in FrameNet for the