# Discrete Event Simulation: It’s Easy with SimPy!

Dmitry Zinoviev  Suffolk University, Boston<EMAIL_ADDRESS>
(Date: February 2018)

###### Abstract.

This paper introduces the practicalities and benefits of using SimPy, a
discrete event simulation (DES) module written in Python, for modeling and
simulating complex systems. Through a step-by-step exploration of the
classical Dining Philosophers Problem, we demonstrate how SimPy enables the
efficient construction of discrete event models, emphasizing system states,
transitions, and event handling. We extend the scenario to introduce
resources, such as chopsticks, to model contention and deadlock conditions,
and showcase SimPy’s capabilities in managing these scenarios. Furthermore, we
explore the integration of SimPy with other Python libraries for statistical
analysis, showcasing how simulation results inform system design and
optimization. The versatility of SimPy is further highlighted through
additional modeling scenarios, including resource constraints and customer
service interactions, providing insights into the process of building,
debugging, simulating, and optimizing models for a wide range of applications.
This paper aims to make DES accessible to practitioners and researchers alike,
emphasizing the ease with which complex simulations can be constructed,
analyzed, and visualized using SimPy and the broader Python ecosystem.

##  1\. Introduction

Computer modeling and simulation (M&S) is the priceless art of bringing to
“life” systems and behaviors that would otherwise be prohibitively expensive,
unethical, or impossible to build, such as a Mars rover, a centaur or a cruel
autocratic regime based on slave ownership.

An M&S project consists of several standard steps (Banks2009DiscreteEvent, ):

  1. (1)

First, an M&S specialist builds a model. The most common model types are:

     * •

Mathematical models that represent the relationships between the system
variables as one or more linear or differential equations (Ohm’s law is an
example of a linear model), and

     * •

Discrete event models that represent the system as a collection of finite
states and transitions between them caused by internal or external events. The
simulation time “jumps” from one event to the next, and there is a timeless
void between events.

  2. (2)

Then, the model is simulated to obtain execution traces. The traces describe
the evolution of some or all system variables over time.

  3. (3)

The traces are often visualized because a picture is worth 1,000 words.

  4. (4)

Optimization is the next step if the project aims to discover the optimal
values of the system parameters. The system model is repeatedly simulated
during optimization with different parameters to minimize or maximize a
predefined goal function. The optimization routine may use constrained or
unconstrained functional optimization, curve fitting, simulated annealing, or
brutal force.

Once implemented, debugged, simulated, and optimized, a model can be used to
evaluate, build, and even replace the original system.

There is a plethora of general-purpose M&S software, both commercial and open
source, such as OpenModelica (openmodelica, ), Ptolemy (Ptolemaeus2014, ),
Simulink (MATLAB, ), and even Simula (kirkerud1989, )—a complete computer
programming language designed to support simulation. This narrowly focused
article shows how to do simple discrete event simulation with SimPy (simpy,
)—an M&S module written in pure Python.

##  2\. Installing SimPy

The easiest (and the only recommended) way to install SimPy is via pip:

⬇

pip install simpy

#> ...

#> Successfully installed simpy-4.1.1

At the time of writing, simpy-4.1.1 is the most recent version of SimPy and
will be used for all examples.

To check if SimPy was successfully installed, open a Python shell and import
simpy.

Assuming you see no error messages, the next goal is finding a suitable model
and simulation system. Why not revisit the good old Dining Philosophers?

##  3\. Dining Philosophers

In brief, five outstanding Philosophers sit around a table with an enormous
dish of freshly made spaghetti at the center. Most of the time, they think,
but now and then, a Philosopher becomes hungry. At that point, he collects two
forks—left and right—and starts eating the pasta. Having refueled himself, he
puts down the forks and returns to thinking.

Like all fine philosophers, the five dining philosophers barely make ends meet
and can afford only five forks, which prevents more than two of them from
eating simultaneously. In addition to being poor, the philosophers are
stubborn and uncooperative: once they pick up one of the forks, they only put
it down once they have a chance to eat. Think what happens if all five guys
pick up their left forks at about the same time. (A hint: they starve.)

Figure 1. The five dining philosophers

The five dining philosophers around the table.

Edsger Dijkstra introduced the dining philosophers problem in 1965, and it has
been a gold standard test framework for concurrent algorithm designers
(Silberschatz2018, ). The problem is simple and well-understood, which makes
it an excellent candidate for an introductory simulation project. The proposed
solution models each philosopher and the interactions between the philosophers
and the chopsticks.

From the discrete event simulation point of view, a philosopher is a finite
state machine with four states: Thinking (the initial state), Hungry, Hungry-
With-One-Chopstick, and Eating. The diagram has no final state, as the
philosophers are never supposed to break the cycle.

Figure 2. State transition diagram of a dining philosopher

State transition diagram of a dining philosopher.

The transitions between the states are caused by external or internal events.
A thinking philosopher becomes hungry after some time—on timeout. The time is
based on some internal philosopher’s considerations that are unknown to us. It
can be modeled as an exponentially distributed random variable111It is a
common and dangerous mistake to treat times as normally distributed random
variables—in the first place, due to the unbounded nature of a normal
distribution. with the mean value T0. The transition from the Eating state to
the Thinking state is triggered internally, too. It can be modeled as another
exponentially distributed random variable with the mean value T1.

The other two transitions in the state transition diagram result from external
events: the availability of the chopsticks. A philosopher becomes Hungry-With-
One-Chopstick when he manages to collect the first chopstick, and he becomes
Eating when the second chopstick becomes available.

Last, it is assumeed a philosopher spends some small but finite time DT in the
Hungry-With-One-Chopstick state even if the other chopstick is readily
available. If he picks up both chopsticks at once, the system never ends up in
a deadlock, which is practically desirable—but not very exciting from the S&M
point of view.

##  4\. Simulating the “Classical” Dining Philosophers

Enter SimPy.

At the core of any SimPy model, there is an environment (simpy.Environment).
The environment, among other things, provides functions for the continuous or
step-by-step simulation of the models, dispatches events, and track of the
current simulation time. A SimPy program starts by importing the necessary
modules and creating a new simulation environment:

⬇

import simpy

import random

env = simpy.Environment()

The environment will be passed to all other entities involved in the model—for
example, to the chopsticks.

Using the S&M terminology, a chopstick is a renewable resource. It is managed
by the environment and must be created before the first use. During the
simulation, a resource can be requested, allocated, and released. SimPy
provides three types of resources: “vanilla” resources (simpy.Resource,
simpy.PriorityResource, simpy.PreemptiveResource), containers
(simpy.Container), and stores (simpy.Store, simpy.FilterStore,
simpy.PriorityStore). The first model of a chopstick is the basic
simpy.Resource.

Each resource has a capacity that defines how many resource users can share
it. If the number of requests exceeds the capacity, SimPy blocks the
outstanding requests until some users release the resource. For sanitary and
convenience reasons, a chopstick should not be shared; its capacity is set to
1 (making it a mutually exclusive resource). Five chopsticks are created:

⬇

N = 5

chopsticks = [simpy.Resource(env, capacity=1) for i in range(N)]

The next step is to define the philosophers. Unlike the chopsticks, the
philosophers are active. In SimPy, active entities are known as processes.
(Not to be confused with the operating system processes!) A process is a
Python generator that yields discrete events. (If your knowledge of the yield
keyword is rusty, refresh it before reading further.) The simulation
environment has a method env.process() that registers the processes. The
following code declares the process function run_the_party() as a Python class
method and creates and registers an event generator after initializing the
class and instance variables. The purpose of highlighting is explained later.

⬇

class Philosopher():

T0 = 10 # Mean thinking time

T1 = 10 # Mean eating time

DT = 1 # Time to pick the other chopstick

def __init__(self, env, chopsticks, my_id, DIAG = False):

self.env = env

self.chopsticks = chopsticks

self.id = my_id

# START_HIGHLIGHT

self.waiting = 0

# END_HIGHLIGHT

self.DIAG = DIAG

# Register the process with the environment

env.process(self.run_the_party())

def get_hungry(self): # Request the resources

yield # Do nothing so far

def run_the_party(self): # Do everything...

yield # ...but do nothing so far

def diag(self, message): # Diagnostic routine

if self.DIAG:

print("P{} {} @{}".format(self.id, message, self.env.now))

The class constructor takes several parameters: the simulation environment,
the chopstick resources, and the philosopher’s id (for diagnostics). Also note
that the current simulation time is available as an attribute of the
environment, env.now.

The philosophers are dysfunctional and do not yield any events, but they can
already be instantiated and assigned two chopsticks to each:

⬇

philosophers = [Philosopher(env, (chopsticks[i], chopsticks[(i + 1) % N]), i)

for i in range(N)]

The event generator run_the_party() yields events of three types: timeout,
request, and release. A call to env.timeout(tau) blocks the generator for tau
time units. Delays are used to model the thinking and eating activities. A
call to resource.request() blocks the generator until the resource becomes
available, marks the resource as allocated to the caller, and returns the
request ID. Finally, a call to resource.release(rq) deallocates the resource
associated with the request id rq. It does not block the generator.

The following code is a translation of Figure 2 to the language of SimPy. It
consists of two methods that implement the main process and a subprocess.

⬇

class Philosopher():

...

def get_hungry(self):

# START_HIGHLIGHT

start_waiting = self.env.now

# END_HIGHLIGHT

self.diag("requested chopstick")

rq1 = self.chopsticks[0].request()

yield rq1

self.diag("obtained chopstick")

yield self.env.timeout(self.DT)

self.diag("requested another chopstick")

rq2 = self.chopsticks[1].request()

yield rq2

self.diag("obtained another chopstick")

# START_HIGHLIGHT

self.waiting += self.env.now - start_waiting

# END_HIGHLIGHT

return rq1, rq2

def run_the_party(self):

while True:

# Thinking

thinking_delay = random.expovariate(1 / self.T0)

yield self.env.timeout(thinking_delay)

# Getting hungry

get_hungry_p = self.env.process(self.get_hungry())

rq1, rq2 = yield get_hungry_p

# Eating

eating_delay = random.expovariate(1 / self.T1)

yield self.env.timeout(eating_delay)

# Done eating, put down the chopsticks

self.chopsticks[0].release(rq1)

self.chopsticks[1].release(rq2)

self.diag("released the chopsticks")

The method get_hungry() is a subprocess—a process within a process. It hides
the technicalities of getting hungry (get one chopstick, get another
chopstick, etc.), letting me concentrate on the party’s conceptual flow. The
main process is suspended until the subprocess yields all events. Note that
the subprocess method returns the chopstick request handlers used later to
release the resources.

The model is ready to run. It can be simulated step by step (or event by
event) by calling env.step(), or for the first t time units by calling
env.run(until=t), or until the simulation naturally stops by calling env.run()
without any parameters. Here is a sample output of the script:

⬇

env.run()

#> P1 requested chopstick @0.10843721582414197

#> P1 obtained chopstick @0.10843721582414197

#> P1 requested another chopstick @1.108437215824142

#> P1 obtained another chopstick @1.108437215824142

#> P0 requested chopstick @6.642119333001387

#> P0 obtained chopstick @6.642119333001387

#> P0 requested another chopstick @7.642119333001387

#> P2 requested chopstick @7.898070765597094

#> ...

#> P2 obtained chopstick @45632.71748485687

#> P1 requested another chopstick @45632.721825172666

#> P4 requested another chopstick @45632.94176319329

#> P3 requested another chopstick @45633.53086921637

#> P2 requested another chopstick @45633.71748485687

The simulation suddenly stops at this point: the system is in a deadlock state
where each philosopher holds one chopstick and waits for the other. There is
no standard way to detect deadlocks in SimPy. An indirect solution is to look
at the counts of requests or actual requests granted for each resource (stored
in the attributes resource.count and resource.users, respectively). If at
least two counts are greater than zero and the simulation stops, the system
has likely been deadlocked:

⬇

[f.count for f in chopsticks]

#> [1, 1, 1, 1, 1]

Incidentally, requests not yet granted can be found in the attribute
resource.queue.

One of the well-known solutions to a deadlock is to allocate resources in some
globally defined order (say, in the order of their numerical IDs or memory
references). The philosophers will never starve if a Python ID-based sorting
is added to the second line of the class initializer:

⬇

self.chopsticks = sorted(chopsticks, key=id)

##  5\. Gathering and Visualizing Statistics

The goal of an M&S project is usually to verify the correctness of the modeled
system and get a sense of its performance. The most common performance
characteristics include waiting time, turnaround time, response time, and
resource utilization. As an exercise, let’s estimate waiting time—the time a
process waits for the requested resources (ready to run but not running).

Unlike other M&S software, SimPy does not provide tools for gathering and
visualizing simulation statistics—and, honestly, it does not have to. As a
part of the Python ecosystem, SimPy can be easily integrated with statistics,
NumPy, Pandas, SciPy, matplotlib, and other plotting, statistical, and general
number-crunching packages.

The mean waiting time of the philosophers is measured by instrumenting the
Philosopher class with a time-counting variable waiting. The relevant
instrumentation code is highlighted in the listings above.

To compare the waiting times or any other performance metrics for different
systems (say, the dining parties with various numbers of philosophers), a
function is needed that prepares a model, makes a simulation run, and returns
the appropriate performance values:

⬇

def simulate(n, t):

"""

Simulate the system of n philosophers for up to t time units.

Return the average waiting time.

"""

env = simpy.Environment()

chopsticks = [simpy.Resource(env, capacity=1) for i in range(n)]

philosophers = [

Philosopher(env, chopsticks[i], chopsticks[(i + 1) % n], i)

for i in range(n)]

env.run(until=t)

return sum(ph.waiting for ph in philosophers) / n

This all-in-one function will be called for different values of n, and the
results will be plotted with matplotlib—the ubiquitous Python plotting engine.

⬇

# Set up the plotting system

import matplotlib.pyplot as plt

import matplotlib

matplotlib.style.use("ggplot")

# Simulate

N = 20

X = range(2, N)

Y = [simulate(n, 50000) for n in X]

# Plot

plt.plot(X, Y, "-o")

plt.ylabel("Waiting time")

plt.xlabel("Number of philosophers")

plt.show()

Figure 3 shows the simulation results. As expected, the waiting time for two
philosophers is shorter than for three. A less obvious result is that the
waiting time for three or more philosophers is almost constant with respect to
the party size.

Figure 3. Waiting time vs. the number of philosophers

Waiting time vs. the number of philosophers

Any interpretation of these facts is beyond the scope of this story.

##  6\. Adding a Container

No matter how you feel about the stubborn philosophers, the bottomless bowl of
rice at the center of the table is unrealistic. Real-life rice ends soon
because it is a consumable resource. The model can be made less fictitious by
treating the rice bowl as a container with finite capacity. The amount of rice
in the bowl will decrease by a fixed amount every time a philosopher eats from
it. (For convenience, let’s assume that the rice is atomically transferred to
the philosopher’s plate when he collects both chopsticks.) Let’s call the new
breed of philosophers “pseudo-philosophers” because genuine philosophers
surely do not eat rice or anything material but merely pretend to be eating.

A simpy.Container can yield two events of interest: put(amount) and
get(amount). Both can block the generator if the container is nearly full or
nearly empty, respectively. A container’s capacity and current level are
stored in the namesake attributes, container.capacity and container.level.

Since a Philosopher class already exists, the cheapest way to implement a
pseudo-philosopher is to add an optional bowl parameter to the constructor and
a request for a portion of rice to the get_hunry() method if the bowl has been
provided (the highlighted lines are responsible for the interaction with the
container):

⬇

class Philosopher():

T0 = 10 # Mean thinking time

T1 = 10 # Mean eating time

DT = 1 # Time to pick the other chopstick

# START_HIGHLIGHT

PORTION = 20 # Single meal size

# END_HIGHLIGHT

# START_HIGHLIGHT

def __init__(self, env, chopsticks, my_id, bowl = None, DIAG = False):

# END_HIGHLIGHT

self.env = env

self.chopsticks = sorted(chopsticks, key=id)

self.id = my_id

self.waiting = 0

# START_HIGHLIGHT

self.bowl = bowl

# END_HIGHLIGHT

self.DIAG = DIAG

# Register the process with the environment

env.process(self.run_the_party())

def get_hungry(self):

start_waiting = self.env.now

self.diag("requested chopstick")

rq1 = self.chopsticks[0].request()

yield rq1

self.diag("obtained chopstick")

yield self.env.timeout(self.DT)

self.diag("requested another chopstick")

rq2 = self.chopsticks[1].request()

yield rq2

self.diag("obtained another chopstick")

# START_HIGHLIGHT

if self.bowl is not None:

yield self.bowl.get(self.PORTION)

self.diag("reserved food")

# END_HIGHLIGHT

self.waiting += self.env.now - start_waiting

return rq1, rq2

def run_the_party(self):

...

Incidentally, the new class design allows me to feed different philosophers
from different bowls and even mix “true philosophers” and “pseudo-
philosophers.”

If the philosophers continue eating their rice, the bowl eventually becomes
empty, and the party ends. Another process, a chef, is solely responsible for
replenishing the bowl every T2 time units:

⬇

class Chef():

T2 = 150

def __init__(self, env, bowl):

self.env = env

self.bowl = bowl

env.process(self.replenish())

def replenish(self):

while True:

yield self.env.timeout(self.T2)

if self.bowl.level < self.bowl.capacity:

yield self.bowl.put(self.bowl.capacity - self.bowl.level)

The updated function simulate() has additional code for instantiating the bowl
and the chef and passes the bowl parameter to the philosophers:

⬇

def simulate(n, t):

"""

Simulate the system of n philosophers for up to t time units.

Return the average waiting time.

"""

env = simpy.Environment()

# START_HIGHLIGHT

rice_bowl = simpy.Container(env, init=1000, capacity=1000)

chef = Chef(env, rice_bowl)

# END_HIGHLIGHT

chopsticks = [simpy.Resource(env, capacity=1) for i in range(n)]

philosophers = [

Philosopher(env, (chopsticks[i], chopsticks[(i + 1) % n]), i,

# START_HIGHLIGHT

rice_bowl)

# END_HIGHLIGHT

for i in range(n)]

env.run(until=t)

return sum(ph.waiting for ph in philosophers) / n

Figure 4 shows the waiting time for the new realistic model. As expected,
there is almost no difference between the two models when the number of dining
philosophers is small (less than ten in this case): the chef is cooking fast
enough to supply the food. When the number of partygoers increases, the
turnaround time of the chef becomes an issue and negatively affects the
waiting time of the patrons.

Figure 4. Waiting time vs. the number of pseudo-philosophers

Waiting time vs. the number of pseudo-philosophers

##  7\. Learning to Give Up

A hungry philosopher may not be that hungry, after all. If the central bowl
has no rice and the chef is nowhere to be found, the philosopher may call it a
day: put down the chopsticks and return to thinking. Presumably, he will have
a chance to eat a supersized meal next time.

Resource requests are generally blocking. They can block the caller forever if
the resource never becomes available (this is the reason for deadlocks). SimPy
provides a mechanism for restricting the wait time by failing events and
waiting for multiple events.

Introducing the new—impatient—kind of philosophers requires changing the methods get_hungry() and run_the_party(). First, a constant MAX_WAIT is added that limits the waiting time to at most half of the time required to replenish the rice bowl. Second, when a philosopher collects both chopsticks, the model fires two events: the container request (as before) and a timeout. The generator process is unblocked when any of these events trigger, which is accomplished by combining the events with the logical “or” operator | (a vertical bar). The operator is a shortcut to the function simpy.events.AnyOf(env,events). If the modeling logic requires triggering all events before unblocking the process, use the logical “and” operator & or the function simpy.events.AllOf(env,events) instead222If you are familiar with UML, note that | corresponds to the UML merge and & corresponds to the UML join..

⬇

class Philosopher():

...

# START_HIGHLIGHT

MAX_WAIT = Chef.T2 / 2

# END_HIGHLIGHT

...

def get_hungry(self, meal_size):

start_waiting = self.env.now

self.diag("requested chopstick")

rq1 = self.chopsticks[0].request()

yield rq1

self.diag("obtained chopstick")

yield self.env.timeout(self.DT)

self.diag("requested another chopstick")

rq2 = self.chopsticks[1].request()

yield rq2

self.diag("obtained another chopstick")

if self.bowl is not None:

request = self.bowl.get(meal_size)

# START_HIGHLIGHT

yield request | self.env.timeout(self.MAX_WAIT)

if request.processed:

self.diag("reserved food")

else: # Timeout

self.diag("gave up")

self.waiting += self.env.now - start_waiting

yield simpy.Event(self.env).fail(ValueError(rq1, rq2))

# END_HIGHLIGHT

# Either no bowl or no timeout!

self.waiting += self.env.now - start_waiting

return rq1, rq2

After yielding the compound event, the program if SimPy processed the
container request by looking at the attribute request.processed. If it did,
the food was reserved, and the philosopher could start eating. Otherwise, the
wait has been interrupted by the timeout, and get_hungry() must inform the
parent process of the failure. SimPy allows a process to fail (and raise an
exception) by yielding a namesake event, as on the last highlighted line of
the previous listing. Incidentally, the raised ValueError exception takes the
chopstick request handles as the parameters: when get_hungry() fails, there is
no other way to return the handles.

The run_the_party() method is instrumented with simple meal size accounting.
Whenever a philosopher chooses not to wait for the chef, his next rice
allowance is increased by one standard portion size. (Intuitively, his overall
wait time increases because some attempts are futile). Failed attempts are
detected; the failures are mitigated by catching the exception raised by
get_hungry() and extracting the chopstick request handles from the exception.

⬇

def run_the_party(self):

# START_HIGHLIGHT

meal_size = self.PORTION

# END_HIGHLIGHT

while True:

yield self.env.timeout(random.expovariate(1 / self.T0))

get_hungry_p = self.env.process(self.get_hungry(meal_size))

# START_HIGHLIGHT

try:

rq1, rq2 = yield get_hungry_p

yield self.env.timeout(random.expovariate(1 / self.T1))

meal_size = self.PORTION

except ValueError as values: # Timeout

rq1, rq2 = values.args

meal_size += self.PORTION

# END_HIGHLIGHT

self.chopsticks[0].release(rq1)

self.chopsticks[1].release(rq2)

self.diag("released chopsticks")

The new timeout feature can be disabled by setting MAX_WAIT to a substantially
large number—say, larger than T2.

Figure 5 shows the waiting time for the impatient philosophers. The difference
between this model and the previous one is visible only when the number of
partygoers is large enough.

Figure 5. Waiting time vs. the number of impatient philosophers

Waiting time vs. the number of impatient philosophers

##  8\. Learning to Communicate

Another classical example—a customer service counter—explains how to model
processes that interact directly rather than by sharing resources. In the new
scenario, customers arrive with exponentially distributed delays and queueing
into a line. (In the example below, the first ten customers are conveniently
“generated” by a customer_generator_p() process.) The operator at the customer
service counter takes the next customer from the line and services them in
exactly SERVICE_DELAY time units. A customer is served successfully with a
probability of 0.9. If there are no customers in the line, the operator
becomes idle (“falls asleep,” if you like) but is reactivated (“woken up”) by
the next arriving customer.

SimPy provides two direct interaction mechanisms: process interruption and
event-based synchronization. To interact, SimPy processes must have references
to either each simpy.events.Process, as returned by calls to env.process(), or
to the events involved in the interaction (simpy.events.Event).

The following code illustrates both communication mechanisms. It begins with
the imports and definitions of global variables. The Python standard double-
ended queue collections.deque models the service line.

SimPy uses Python exceptions to manage process interruptions. A new exception
class, CustomerFailedException, is created to distinguish SimPy process-
related exceptions from proper Python exceptions.

⬇

import random

from collections import deque

import simpy

# Shared global variables and "constants"

SERVICE_DELAY = 10

service_line = deque()

counter_idle = False # The state of the operator at the service counter

env = simpy.Environment()

class CustomerFailedException(Exception):

pass

The model consists of three types of processes. The customer_generator()
function imitates the rest of the world. In a complete model, customers would
arrive from other model components rather than being “born” on the spot.

⬇

def customer_generator():

for _ in range(10):

env.process(customer())

yield env.timeout(random.expovariate(1 / SERVICE_DELAY))

customer_generator_p = env.process(customer_generator())

Customers in the model are simple stateless creatures. Once born, they obtain
a service ticket represented as a simpy.events.Event and enter it into the
service_line. If the service counter operator is asleep (counter_idle==True),
the customer interrupts the counter process by calling counter_p.interrupt().
This method raises a simpy.Interrupt exception in the target process and
“awakens” it. The matching lines are highlighted in the code below.

Now that the operator is not asleep anymore, the customer yields the ticket.
The event gets triggered. The customer is blocked until some other process
processes the event. Naturally, the event becomes processed, and if it is
processed successfully, the execution of the customer resumes333If you are
familiar with the C pthreads, you may recognize a striking similarity between
SimPy events and pthread_cond_t. Triggering and processing an event are loose
equivalents of pthread_cond_wait() and pthread_cond_signal().. Otherwise, a
CustomerFailedException is raised, which terminates the simulator if it is not
correctly handled.

⬇

def customer():

print("Customer arrived @{0:.1f}".format(env.now))

ticket = env.event()

service_line.append(ticket)

if counter_idle:

# START_HIGHLIGHT

counter_p.interrupt()

# END_HIGHLIGHT

try:

yield ticket

print("Customer left @{0:.1f}".format(env.now))

except CustomerFailedException:

print("Customer failed (and left) @{0:.1f}".format(env.now))

The service counter is a potentially infinite process. If there is at least
one waiting customer, the first ticket is removed from the service line, and
the corresponding customer is taken care of. Then the counter process “flips a
coin” and either fails the ticket event
(ticket.fail("CustomerFailedException")) or succeeds it (ticket.succeed()). In
the former case, a CustomerFailedException is raised on the client side, as
explained above.

If the service line is empty, the counter operator becomes idle and yields an
event. This event is triggered but never processed, putting the operator in
eternal sleep. However, as explained above, sleep can also be interrupted by
an arriving customer process.

⬇

def counter():

global counter_idle

while True:

if service_line:

ticket = service_line.popleft()

yield env.timeout(SERVICE_DELAY)

if random.randint(0,9) == 9:

ticket.fail(CustomerFailedException())

else:

ticket.succeed()

else:

counter_idle = True

print("The operator fell asleep @{0:.1f}".format(env.now))

try:

yield env.event()

# START_HIGHLIGHT

except simpy.Interrupt:

# END_HIGHLIGHT

counter_idle = False

print("The operator woke up @{0:.1f}".format(env.now))

counter_p = env.process(counter())

The simulation’s sample output follows:

⬇

env.run()

#> The operator fell asleep @0.0

#> Customer arrived @0.0

#> The operator woke up @0.0

#> Customer arrived @0.4

#> Customer left @10.0

#> The operator fell asleep @20.0

#> Customer left @20.0

#> Customer arrived @24.6

#> The operator woke up @24.6

#> The operator fell asleep @34.6

#> Customer left @34.6

#> Customer arrived @59.3

#> The operator woke up @59.3

#> Customer arrived @64.2

#> Customer arrived @65.9

#> Customer left @69.3

#> Customer arrived @75.8

#> Customer left @79.3

#> Customer arrived @83.1

#> Customer arrived @86.7

#> Customer left @89.3

#> Customer arrived @90.9

#> Customer left @99.3

#> Customer left @109.3

#> Customer left @119.3

#> The operator fell asleep @129.3

#> Customer left @129.3

You can further instrument the model with statistics-gathering tools.

##  9\. Conclusion

This article demonstrates the possibility of using Python for not-so-trivial
M&S projects. The “M” (modeling) part of the story is somewhat sketchy.
Hovewer, computer modeling, even discrete event modeling alone, is too vast to
be covered in a dozen pages without distracting the reader’s attention from
the holistic M&S experience.

On the SimPy side, the most notable features that should be sufficient for
many simple M&S projects were reviewed. Here is a summary of what was left
uncovered:

  * •

env.schedule(event, priority=1, delay=0)—A mechanism for scheduling an event
in the future rather than triggering it immediately.

  * •

event.callbacks—A list of callback functions that will be called when the
event is processed. (The default callback always resumes the blocked process
that triggered the event.)

  * •

PriorityResource, PreemptiveResource, PriorityStore, FilterStore—A collection
of specialized resources.

  * •

RealtimeEnvironment—An environment for real-time simulation that requires
synchronization between the simulation clock and the “wall clock.”

You are referred to the SimPy documentation (simpy, ) for additional
information.

## References

  * (1) Banks, J., Carson, J., Nelson, B., and Nicol, D. Discrete-Event System Simulation, 5 ed.  Pearson, 2009.
  * (2) Fritzson, P., Pop, A., Abdelhak, K., Ashgar, A., Bachmann, B., Braun, W., Bouskela, D., Braun, R., Buffoni, L., Casella, F., Castro, R., Franke, R., Fritzson, D., Gebremedhin, M., Heuermann, A., Lie, B., Mengist, A., Mikelsons, L., Moudgalya, K., Ochel, L., Palanisamy, A., Ruge, V., Schamai, W., Sjölund, M., Thiele, B., Tinnerholm, J., and Östlund, P. The OpenModelica Integrated Environment for Modeling, Simulation, and Model-Based Development.  Modeling, Identification and Control 41, 4 (2020), 241–295.
  * (3) Kirkerud, B. Object-Oriented Programming with SIMULA.  Addison-Wesley, Boston, MA, 1989.
  * (4) Ptolemaeus, C., Ed.  System Design, Modeling, and Simulation using Ptolemy II.  Ptolemy.org, 2014.  http://ptolemy.org/books/Systems.
  * (5) Silberschatz, A., Galvin, P., and Gagne, G. Operating System Concepts, 10 ed.  Wiley, 2018.  See Chapter 5 for discussion on the Dining Philosophers Problem.
  * (6) Team SimPy.  SimPy, 2023.  https://simpy.readthedocs.io.
  * (7) The MathWorks Inc. Matlab version: 9.13.0 (r2022b), 2022.  https://www.mathworks.com.


# Measure More, Question More: Experimental Studies on Transformer-based
Language Models and Complement Coercion

Yuling Gu
Allen Institute for AI, Seattle, WA
<EMAIL_ADDRESS>This is work mainly done when the author was a student at
New York University.

###### Abstract

Transformer-based language models have shown strong performance on an array of
natural language understanding tasks. However, the question of how these
models react to implicit meaning has been largely unexplored. We investigate
this using the complement coercion phenomenon, which involves sentences like
“The student finished the book about sailing” where the action “reading” is
implicit. We compare LMs’ surprisal estimates at various critical sentence
regions in sentences with and without implicit meaning. Effects associated
with recovering implicit meaning were found at a critical region other than
where sentences minimally differ. We then use follow-up experiments to factor
out potential confounds, revealing different perspectives that offer a richer
and more accurate picture.

##  1 Introduction

While transformer-based language models (LMs) have been evaluated on a range
of language understanding tasks, there is much less attention on how they deal
with implicit meaning in a phenomenon like complement coercion (Pustejovsky,
1991, 1995), where there is an interesting interplay of semantics and
commonsense knowledge. In this work, we look into complement coercion in
direct object position as a case study for exploring implicit meaning in LMs.

Sentences with complement coercion contain implicit meaning that we infer
based on the semantic meaning of the actions or objects in the sentence as
well as our commonsense knowledge about them. Consider the sentence in the
coerced condition in Figure 1, one would expect an event-selecting verb like
finished to be followed by a complement that expresses some event or activity
rather than a complement phrase like the book which has the default
interpretation of an entity. Interestingly, although there is no mention of
the initiated activity, the meaning that the student began some activity
involving the book is still available and a reader would interpret the
sentence such as in the second (preferred) condition in Figure 1, a sentence
which involves no implicit meaning.

Figure 1: Example of a set of test sentences in experiment 1 and the critical
regions for measurement.

In NLP, a fundamental question as to how LMs behave as they progress through
such sentences with implicit meaning in their processing remains unanswered,
and it is not yet clear whether they behave differently for sentences with and
without implicit meaning. This paper aims to improve our understanding of how
neural LMs behave when processing English sentences with and without implicit
meaning by using the psycholinguistic phenomenon of complement coercion
(Section 2). Using the GPT-2 (Radford et al., 2019) family models as a case
study, we use surprisal estimates at critical sentence regions to model LMs’
behavior (see Figure 1, details in Section 3). We focus on complement coercion
in the direct object position, introducing a series of three diagnostic
datasets drawn from human psycholinguistics experiments (Section 4). Given
their origin in psycholinguistics, these diagnostic datasets are controlled to
ask targeted questions regarding the processing of implicit meaning and
specifically chosen to elicit differences in human processing of coercion
versus control conditions.

This paper has three main contributions. First, to the best of our knowledge,
our work is the first of its kind to study transformer-based LMs’ behavior on
the psycholinguistic phenomenon of complement coercion in English, using
surprisal estimates. Second, while most previous works studying LMs’ behavior
compare full sentences Warstadt et al. (2020) or examine one critical region
per sentence Hu et al. (2020), for each sentence, we take measurements at
three positions important for analysis of the complement coercion phenomenon
to provide a richer analysis. Finally, the series of three carefully designed
experiments we present in the paper provides an illustrative example of how
targeted follow-up experiments could be effectively used to tease apart
confounding factors in probing experiments to give us a more accurate picture.
Our first experiment provides strong evidence that GPT-2 models consistently
distinguish sentences with implicit meaning (coerced conditions) from those
without. However, rigorous follow-up experiments reveal that this apparent
distinction between the two conditions can be completely attributed to
confounding factors.

##  2 What is coercion?

In natural language, the meaning of an expression is often thought to be
constructed from the meaning of its smaller parts Fodor and Lepore (2002);
Heim and Kratzer (1998); Montague (1970). However, there are also many cases
where the meaning of an expression may be richer than what is overtly
expressed and composed from its constituents, giving rise to various semantic
theories on non-syntactic operations that contribute extrasyntactic content to
meaning Groenendijk and Stokhof (1989); Herman (1988); Partee (1986, 1995);
Partee and Rooth (1983). One such process of semantic enrichment is through
coercion, where a linguistic element that has a default interpretation
mismatched with the semantic context is coerced to take on another meaning
that is more compatible semantically Jackendoff (1997); Pustejovsky (1995). We
focus our study on complement coercion in the direct object position which we
discuss in detail in Section 2.1.

###  2.1 Complement coercion in direct object position

Complement coercion occurs when a verb, such as begin, finish, complete, that
semantically selects (s-selects) for event-describing complements, takes on a
complement that has the default interpretation of an entity instead. One
syntactic environment that complement coercion can occur in is the direct
object position, when a verb like finished is composed directly with an
entity-denoting determiner phrase (NP) in direct object position (Traxler et
al., 2002; McElree et al., 2006b; Pylkkänen and McElree, 2007; Pylkkänen,
2008) as in the coerced condition in Figure 1.

In the coerced condition in Figure 1, finished s-selects for an event-
describing complement and is most naturally combined with a VP-complement.
However, the NP the book has the default interpretation of an entity. As such,
the composition of the verb finished with the entity-denoting NP the book
presents a “type-mismatch” (Partee, 1986). But instead of resulting in an
anomalous sentence, this sentence very naturally received the interpretation
that the student finished some contextually relevant, though not overtly
stated, activity involving the book. Therefore, the semantic composition of
the coerced condition in Figure 1 involves some element that is not present in
its syntax. Due to the enriched composition, readers typically interpret that
sentence the same way as one that explicitly expresses the meaning of the
event (like the preferred condition in Figure 1).

In the reconstruction of the covert event, semantic knowledge that finished
s-selects for an event-describing complement sets up the expectation that it
will be followed up actions like reading, writing, eating, watching, etc. We
then rely on our commonsense knowledge, that someone cannot be “eating” or
“watching” the book to rule these possibilities out. Finally, the subject, the
student, together with our world-knowledge that it is more common for a
student to read books rather than to write them, helps us to infer “reading”
as the implied action. Therefore, interpreting the implicit meaning here
involves both semantics and commonsense knowledge.

###  2.2 Coercion in cognitive sciences

The property of natural language that the meaning of certain expressions
cannot be recovered from what has been explicitly expressed has garnered much
attention both in the linguistics and psycholinguistics community. Some
variants of coercion have been studied from a linguistics perspective looking
into the theories behind this phenomenon using concepts such as type-shifting,
contributing to a rich literature including Barker (2016); Dölling (1992);
Groenendijk and Stokhof (1989); Heim and Kratzer (1998); Jackendoff (1997);
Partee and Rooth (1983). The underlying shifting operation which coerces the
entity-denoting NP from its default interpretation into an event to satisfy
the selectional properties of the verb (discussed in Section 2.1), has also
received much attention in linguistics Partee (1986, 1995); Jackendoff (1997);
Pustejovsky (1991, 1993, 1995).

In the psycholinguistics community, there has been an increasing body of work
reaffirming the significance of the phenomenon using experiments measuring
eye-movements (Traxler et al., 2002; Harris et al., 2008; McElree et al.,
2006a), reading times (Traxler et al., 2002; McElree et al., 2001; Pylkkänen,
2008; Brennan and Pylkkänen, 2008; McElree et al., 2006b) and
magnetoencephalography (MEG) studies (Harris et al., 2008; Pylkkänen and
McElree, 2007; Pylkkänen, 2008; Brennan and Pylkkänen, 2008; Pylkkänen et al.,
2009). Specifically, for the case of complement coercion in direct object
position, McElree et al. (2001); Traxler et al. (2002) and McElree et al.
(2006b) found increased cost of processing associated with coerced conditions
in terms of reading time for complements; Traxler et al. (2002, 2005) and
McElree et al. (2006a)’s results through tracking eye-movements also point in
the same direction; Pylkkänen and McElree (2007) localized the cost of such
processing by observing increased amplitudes in the anterior midline field
(AMF) in ventromedial prefrontal areas at 350-450 msec.

However, the question of how transformer-based LMs handle implicit meaning in
the phenomenon of coercion has received much less attention. To the best of
our knowledge, our work is the first to provide a systematic study on this.

##  3 Experiment

To study transformer-based LMs’ behavior, we use the conditional probability
(subsection 3.1) of auto-regressive LMs to compute surprisal estimates
(subsection 3.2), and compare surprisal estimates at critical sentence regions
(subsection 3.3). Given that GPT-2 (Radford et al., 2019) models have been
shown to better predict human neural and behavioral data during language
processing compared to other LMs (Schrimpf et al., 2021; Goldstein et al.,
2022), we focus on the family of GPT-2 models as a case study (subsection
3.4).

###  3.1 Language models’ output probabilities

Let $S$ be a sentence with tokens $w_{1},\cdots,w_{N}$, the conditional
probability at token $i$, $P(w_{i}|w_{1},\cdots,w_{i-1})$ is estimated by the
model either using statistical-based methods like $n$-gram (Chen and Goodman,
1999) or neural network-based methods like LSTM (Mikolov et al., 2010) and
transformer (Radford et al., 2019). The conditional probability from auto-
regressive LMs only depends on the words to the left of that position. That
is, when an auto-regressive LM predicts the token at position $i$, it only
relies on tokens $w_{1},\cdots,w_{i-1}$, and we use this to model human’s
processing of token $i$ in a sentence having read what came before it. To
obtain the conditional probability, we apply softmax to the logits produced by
the LMs. This probability is then used to compute the surprisal score.

###  3.2 Surprisal

Hale (2001) and Levy (2008) suggested a quantification of the cognitive effort
required to process a word in a sentence using the surprisal of the word:

$S(w_{i})=-\log_{2}P(w_{i}|w_{1},\cdots,w_{i-1}).$

In previous work, van Schijndel and Linzen (2018) used surprisal estimates
from LMs (RNN models with LSTM units) to model increased processing difficulty
associated with increased reading times in garden path sentences and Demberg
and Keller (2008) showed that surprisal from a broad-coverage unlexicalized
(i.e., only using the structural probabilities) LM strongly correlates with
reading times. Looking at the phenomenon of coercion in German, Delogu et al.
(2017) presents results from monitoring eye-tracking and event-related
potentials (ERPs) as well as estimating the surprisal of the complement noun
from a trigram language model, showing that the coercion cost could be largely
attributed to the surprisal of the complement noun.

In this work, we look at the phenomenon of coercion in English, and compute
the surprisal estimates for transformer-based LMs to analyze the LMs’ behavior
in processing sentences. Surprisal estimate for each region is calculated by
averaging the surprisal estimates of all words in that region. We follow Levy
(2008)’s original proposal of the surprisal theory in interpreting surprisal
as going beyond the traditional domain of predictability (primarily semantic
phenomenon) and that it can arise from any source.

###  3.3 Measure positions

Measured position | What it comprises |  Illustration of the position in example set
---|---|---
Differing verb |  Verb which sentences in each set minimally differ |  Coerced: The secretary began the memo about the new office policy.
Preferred: The secretary typed the memo about the new office policy.
Non-preferred: The secretary read the memo about the new office policy.
Target region |  Determiner and noun following the differing verb |  Coerced: The secretary began the memo about the new office policy.
Preferred: The secretary typed the memo about the new office policy.
Non-preferred: The secretary read the memo about the new office policy.
Post-target region |  Two words immediately following the target region |  Coerced: The secretary began the memo about the new office policy.
Preferred: The secretary typed the memo about the new office policy.
Non-preferred: The secretary read the memo about the new office policy.
Table 1: Details of critical sentence regions measured for analyzing LMs.
Illustrated examples are based on a triplet in our diagnostic Dataset 1 as
found in Traxler et al. (2002).

In a phenomenon like coercion, the effects associated with the coerced
condition are not uniform across every word in a sentence and can even vary
across regions where the effects may show up. Therefore, it is inadequate to
provide an analysis based on making one measurement per sentence or simply
averaging across each sentence. Based on insights from psycholinguistic
experiments like Traxler et al. (2002), we capitalize on making measurements
at three critical sentence regions important for analysis of the coercion
phenomenon to provide a richer analysis. These regions are summarized in Table
1 with illustrative examples. First, we pay attention to the verb region of
each sentence in which the sentence differs from one or more of the other
sentences in the set (triplet or quadruplet). Next, we examine the noun and
its determiner following the verb region as the target region. Coerced
conditions incur additional processing cost when readers realize that the
default interpretation of the entity-denoting NP does not match the
selectional restrictions of the preceding verb, and this happens when (or
shortly after) readers encounter the NP (Traxler et al., 2002), making the NP
an important region to analyze. Finally, as effects sometimes emerge on words
following the target region (Rayner and Pollatsek, 1989), we also examine the
post-target region which comprises the two words following the target region.

###  3.4 Models

Model | Layers | Hidden | Heads | Parameters
---|---|---|---|---
DistilGPT-2 | 6 | 768 | 12 | 82M
GPT-2 | 12 | 768 | 12 | 117M
GPT-2 medium | 24 | 1024 | 16 | 345M
GPT-2 large | 36 | 1280 | 20 | 774M
GPT-2 xl | 48 | 1600 | 25 | 1558M
Table 2: Details of the five GPT-2 models studied.

GPT-2 uses multi-layer transformer decoder as the architecture, and its
pretraining uses left-to-right autoregressive language modeling. We compare
results across four LMs of varying sizes as introduced in the original GPT-2
paper (Radford et al., 2019), together with DistilGPT-2, distilled from the
base GPT2 model using the same distillation method as in Sanh et al. (2019).
The architectures of the models used are summarized in Table 2. For all models
analyzed in this paper, we use the Huggingface implementation (Wolf et al.,
2019, 2020),111https://github.com/huggingface/transformers based on PyTorch
(Paszke et al., 2019). Following Ettinger (2020) and Hu et al. (2020), since
our diagnostic tests are designed to study processing of words in context
(rather than performance on a particular task), we examine the models’
behavior without task-specific fine-tuning.

##  4 Diagnostic datasets

Dataset | Original psycholinguistic experiment | Selection from original stimuli
---|---|---
1 |  “Coercion in sentence processing: Evidence from eye-movements and self-paced reading” by Traxler et al. (2002) |  36 triplets ( Coercion/Preferred/Non-preferred) from stimuli for Experiment 1
2 |  32 quadruplets (Event/Neutral verb + Event/Entity NP) from stimuli for Experiments 2 and 3
3 |  “An MEG Study of Silent Meaning” by Pylkkänen and McElree (2007) |  35 triplets (Coerced/Anomalous/Control) from the Nonembedded Stimuli
Table 3: Source of diagnostic datasets used in our experiments.

For our diagnostic datasets, we use expert-designed sentence sets from
psycholinguistic experiments as summarized in Table 3. Each triplet or
quadruplet contains one sentence with the coerced condition, and other
sentences specifically designed to tease apart different potential confounding
factors which we will elaborate on in the rest of this section.

###  4.1 Dataset 1: Coercion/Preferred/Non-preferred

####  4.1.1 Targeted design

Our first dataset is targeted at investigating if LMs show differences in
their behavior for coerced versus non-coerced conditions. Further, it is
designed to answer the potential follow-up question of whether we can rule out
that the observed differences in their behavior are due to differing levels of
preference for the conditions rather than associated with recovering the
implicit meaning in coerced condition. This dataset originates from Experiment
1 performed in Traxler et al. (2002) which compared measurements of eye-
movements, showing that humans demonstrate rapid difficulty soon after
encountering the NP complement when verbs that require an event are followed
by complements typically used as entities rather than events. The data
comprises 36 triplets, and we illustrate the format of each of those triples,
by showing an example in 4.1.1 a-c.

. . Coerced: The secretary began the memo about the new office policy. .̱
Preferred: The secretary typed the memo about the new office policy. .̧ Non-
preferred: The secretary read the memo about the new office policy.

Sentences like 4.1.1 present coerced conditions which involve implicit meaning
as readers need to reconstruct the covert event. In contrast, in sentences
4.1.1 and 4.1.1, the verb explicitly specifies the activity. The difference
between sentences 4.1.1 and 4.1.1 is that according to a preference norming
pre-test, sentences like 4.1.1 present the preferred interpretation of
sentences such as 4.1.1, whereas sentences like 4.1.1 present a plausible but
non-preferred interpretation. To derive the preferred and non-preferred
conditions, Traxler et al. (2002) collected data on how participants would
typically interpret sentences through fill-in-the-blank tasks involving type-
shifting sentences like “The editor finished  the article.”

####  4.1.2 Studying LMs with the dataset

If LMs show differences in their behavior for coerced versus non-coerced
conditions representing the extra processing cost, surprisal measured at some
measure position would be greater for condition (a) in example 4.1.1 compared
to conditions (b) and (c). To further differentiate if this difference is due
to the additional processing associated with reconstructing the covert event
rather than processing atypical non-preferred relationships, it is important
that this difference is not significant when comparing the preferred condition
(b) to the non-preferred condition (c). This would support that the coerced
condition incurs additional processing cost in reconstructing the implicit
meaning and rule out the effect of costs associated with processing non-
preferred conditions.

###  4.2 Dataset 2: Event/Neutral verb + Event/Entity NP

####  4.2.1 Targeted design

Our second dataset is targeted to investigate if we can rule out two potential
alternative explanations involving elements in the coerced conditions –
whether the potential differences in LMs’ behavior for coerced versus non-
coerced conditions are in fact due to (1) event-selecting verbs like began
being simply semantically underspecified, or (2) event interpretations of NPs
being more difficult to compute, causing different behavior associated with
general processing difficulty rather than one associated with reconstructing
the implicit meaning. This dataset originates from Experiments 2 and 3
performed in Traxler et al. (2002) which compared measurements of eye-
movements and self-paced reading respectively. Traxler et al. (2002)’s
experimental results showed that neither of the two factors affected
processing difficulty, ruling out the two alternative explanations for the
differences in processing observed in humans. This data comprises 32
quadruplets, and we illustrate the format of each of those quadruplets, by
showing an example in 4.2.1 a-d.

. . Event verb + Event NP: The boy started the fight after school today. .̱
Neutral verb + Event NP: The boy saw the fight after school today. .̧ Event
verb + Entity NP: The boy started the puzzle after school today. .̣ Neutral
verb + Entity NP: The boy saw the puzzle after school today.

Sentences like 4.2.1 present coerced conditions because with event-selecting
verbs like started, the entity-denoting NP complement the puzzle requires
type-shifting to give an event interpretation and readers need to reconstruct
the implicit action (e.g. solving). Sentences of the form in 4.2.1 and 4.2.1,
with neutral verbs like saw which can be used felicitously with both entity-
denoting or event-denoting complements, do not require such coercion
operation. Sentence 4.2.1 also presents a non-coerced condition since the
event-denoting complement the fight satisfies the selectional restrictions of
the verb.

####  4.2.2 Studying LMs with the dataset

If LMs show differences in their behavior for coerced versus non-coerced
conditions, in at least one of the measure positions, they would have the
greatest surprisal in for coerced condition (c) in example 4.2.1 compared to
the other three conditions. However, if this greater surprisal is only due to
verbs like started being (arguably) less specific than verbs like saw,
behavior within condition pairs (a) and (c), as well as (b) and (d) would be
similar, and any difference would only be across the pairs where the surprisal
for (a) and (c) would be greater than that of (b) and (d). For the second
confounding factor, if the difference is generally associated with processing
event interpretations of NPs rather than computing implicit meaning, LMs’
surprisal on conditions (a), (b) and (c) where NPs are assigned event
interpretations would each be significantly greater than condition (d) where
the NP takes on an entity interpretation. We use this diagnostic dataset to
tease apart these two possible confounding factors.

###  4.3 Dataset 3: Coerced/Anomalous/Control

####  4.3.1 Targeted design

Our final diagnostic dataset is designed to answer the follow-up question of
whether we can rule out that any observed effect of coercion the LMs show is
due to treating the coerced condition as an anomalous condition (when it is in
fact fully grammatical and plausible) rather than associated with recovering
the implicit meaning. This dataset originates from the Nonembedded Stimuli
used in Pylkkänen and McElree (2007), which compared MEG measurements, showing
that the coerced and anomalous conditions elicited distinct effects, ruling
out the possibility that the coerced condition is processed like an anomaly in
reading. The data comprises 35 triplets, and we illustrate the format of those
triplets by showing an example in 4.3.1 a-c.

. . Coerced: The journalist began the article before his coffee break. .̱
Anomalous: The journalist astonished the article before his coffee break. .̧
Control: The journalist wrote the article before his coffee break.

Sentences like 4.3.1 present coerced conditions which require the reader to
posit an activity, relate the entity-denoting NP to that activity, and
construe the activity as the complement of the event-selecting verb. In
contrast, similar to the verbs in preferred and non-preferred conditions for
Dataset 1, as well the neutral verbs in Dataset 2, the verb in 4.3.1
explicitly expresses an eventive interpretation in which readers do not need
to generate the eventive meaning that is not part of the overt linguistic
input. An anomalous condition like 4.3.1 is also introduced in each set of
sentences. The anomalous sentence is constructed with verbs like astonish
which require direct objects that denote an experiencer of the psychological
state. This requirement cannot be fulfilled with entity-denoting NPs such as
the article, resulting in an anomaly.

####  4.3.2 Studying LMs with the dataset

Similar to the case for Dataset 1, if LMs show differences in their behavior
for coerced versus non-coerced conditions representing the extra processing
cost, there will be greater surprisal at some measure position for condition
(a) compared to (c). Further, if this effect is different from anomaly
detection, the observed effects for condition (a) will be different from that
of condition (b).

##  5 Results and analysis

For the different diagnostic datasets and LMs experimented, we report
surprisal estimates across the three critical sentence regions. The surprisal
estimates from measurements on the three datasets are presented in Figures 2,
3 and 4 respectively. To test for statistical significance, we performed
Friedman test for repeated measures and post-hoc Wilcoxon matched-pairs
signed-rank test with Bonferroni correction. We report differences that are
significant under both the Friedman test and post-hoc test. An alpha level of
$0.05$ was used.

###  5.1 Experiment 1: Coerced/Preferred/Non-preferred

It is consistent across all five models that at the target region, Friedman’s
test of differences among repeated measures showed that there is a significant
difference (in all cases, $p<0.001$) among surprisal estimates measured in the
coerced, preferred and non-preferred conditions.

####  5.1.1 Greater surprisal in coerced condition

For all five models, post-hoc tests using Wilcoxon matched-pairs signed-rank
test with Bonferroni correction showed that surprisal for the target region in
the coerced condition is significantly greater than both that in the preferred
and non-preferred conditions ($p<0.001$ for all pairwise comparisons). This
result suggests that surprisal estimates in LMs reflect the difficulty
associated with recovering the covert meaning in the coerced condition
compared to other conditions.

####  5.1.2 Greater surprisal $\neq$ non-preferred

Further, based on post-hoc tests for surprisal estimates at the target region,
it is also consistent across the LMs that the difference in surprisal at the
preferred and non-preferred condition is not significant ($p>0.05$). The
observation that surprisal estimates between the preferred and non-preferred
conditions are similar provides evidence that higher surprisal estimates from
LMs are not associated with processing less typical relations in non-preferred
conditions, because if this were the case, there would have been significantly
greater surprisal for the non-preferred condition compared to the preferred
condition. This further supports that higher surprisal estimates from LMs
could reflect costs specific to processing implicit meaning which is only
present in the coerced condition out of the three conditions in Dataset 1.

Figure 2: Bar graphs showing mean surprisal estimates from Experiment 1, by
model (column), region (row) and condition (x-axis of each subplot). The error
bars represent standard error. At the target region (see corresponding row),
there is consistently greater surprisal in the coerced condition (first bar in
each subplot) than other conditions.

###  5.2 Experiment 2: Event/Neutral verb + Event/Entity NP

In this experiment, it is consistent across all five models that at the target
region, Friedman’s test of differences among repeated measures shows that
there is significant difference (in all cases, $p<0.001$) among surprisal
estimates measured in the (a) Event verb + Event NP, (b) Neutral verb + Event
NP, (c) Event verb + Entity NP, and (d) Neutral verb + Entity NP conditions.

####  5.2.1 Greater surprisal for coerced condition or Entity NP?

Consistent across all five models, post-hoc tests show that surprisal
estimates at the target region are significantly higher for condition (c),
which involves an implicit activity, compared to both conditions (a) and (b)
($p<0.01$ for all pairwise comparisons). However, in comparing condition (c)
to condition (d) which has no implicit activity, only DistilGPT-2 shows
greater surprisal for the condition with implicit activity ($p<0.05$), whereas
no significant difference in surprisal estimates between these two conditions
is observed for the four other models ($p>0.05$). These results demonstrate
that for the case of DistilGPT-2, surprisal estimates at the target region do
seem to reflect effects associated with implicit meaning in the coerced
condition, since higher surprisal estimates are specific to sentences with
implicit activities compared to other sentences. However, for the other four
models, the measurements suggest that the greater surprisal observed could
potentially be attributed to a general effect due to processing entity-
denoting NPs (conditions (c) and (d)).

Further, post-hoc tests for the larger GPT-2 models (medium, large and xl)
show a reliable effect of surprisal at the target region being greater for
condition (d) compared to conditions (a) and (b) ($p<0.05$) but conditions (c)
and (d) are not significantly different from each other. This points towards
greater surprisal reflecting processing cost associated with entity-denoting
NP in general rather than the coerced condition. As all sentences in Dataset 1
had entity-denoting NPs, no comparison was made for different NPs and this
effect was not observed from our experiments on Dataset 1. This new finding
using Dataset 2 highlights that investigation of a phenomenon based on one
setup or sentence template is not enough, and the need to have carefully
varied controls.

####  5.2.2 Greater surprisal $\neq$ less specific verbs

If greater surprisal at the target region is only due to verbs like started
being (arguably) less specific than verbs like saw, behavior within condition
pairs (a) and (c), as well as (b) and (d) would be similar, and any difference
would only be across the pairs where the surprisal for (a) and (c) would be
greater than that of (b) and (d) respectively. We rule out this possibility
because based on the post-hoc tests, surprisal for condition (c) is
consistently greater than (a) across all five models, and surprisal for (d) is
greater than (b) for the medium, large and xl models. There is also a reliable
trend that surprisal estimates for conditions (a) and (b) are not
significantly different, and surprisal for condition (c) is not significantly
greater than (d) except for the case of DistilGPT-2. These analysis show that
greater surprisal from LMs does not reflect processing cost due to verbs like
started being (arguably) less specific.

####  5.2.3 Greater surprisal $\neq$ event interpretations of NPs

If the difference in surprisal at the target region is generally associated
with processing event interpretations of NPs rather than computing implicit
meaning, LMs’ surprisal for conditions (a), (b) and (c) where NPs are assigned
event interpretations would each be significantly greater than condition (d)
where the NP takes on an entity interpretation. This explanation is ruled out
because surprisal estimates at conditions (a) and (b) are never higher than
that at (d) in all cases, and surprisal for condition (c) is not significantly
greater than (d) generally (except for the case of DistilGPT-2).

Figure 3: Bar graphs showing mean surprisal estimates from Experiment 2, by
model (column), region (row) and condition (x-axis of each subplot). The error
bars represent standard error. At the target region (see corresponding row),
there is generally significantly greater surprisal for sentences with entity-
denoting NPs (two rightmost bars in each subplot).

###  5.3 Experiment 3: Coerced/Anomalous/Control

Across all five models and the three different measure positions, Friedman’s
test show that there is a significant difference (in all cases, $p<0.001$)
among surprisal estimates measured in the coerced, anomalous and control
conditions. At the differing verb and target region, it is consistent across
all five models that post-hoc tests show that all pairwise differences are
significant ($p<0.05$). At the post-target region, it is consistent across the
five models that surprisal for the anomalous condition is significantly higher
($p<0.01$) than that of the coerced and control conditions.

####  5.3.1 Greater surprisal for coerced condition compared to control
condition

At the differing verb and target region, there is a reliable trend of
significantly higher surprisal estimates for the coerced condition compared to
the control condition. This observation for the target region aligns with the
results from measurements on Dataset 1, which also suggests that greater
surprisal could reflect additional processing cost for dealing with implicit
meaning that is present in the coerced condition.

####  5.3.2 Even greater surprisal for processing anomalous condition

However, we also found that across all five models and all three measure
positions, there is consistently higher surprisal estimates for the anomalous
condition compared to the coerced condition (and control condition). This
raises the question of whether greater surprisal demonstrated by LMs is
associated with reconstructing implicit meaning (in the coerced condition) or
that it reflects anomaly detection. Given such observation, we cannot dismiss
the possibility that LMs are processing coercion like a weaker form of
anomaly.

Figure 4: Bar graphs showing mean surprisal estimates from Experiment 3, by
model (column), region (row) and condition (x-axis of each subplot). The error
bars represent standard error. Across all five models and all three measure
positions, there is consistently greater surprisal for processing the
anomalous condition (second bar in each subplot) than the coerced condition
(and control condition).

##  6 Related work

A range of prior work has contributed towards rigorously probing and improving
LMs, including uncovering how LMs exploit common surface patterns (Jiang and
de Marneffe, 2021), introducing probing method that considers utility of
information by models (Elazar et al., 2021b), datasets that isolate specific
linguistics phenomenon (Warstadt et al., 2020), strategies for improving
performance on out-of-distribution examples (Ben-David et al., 2022), and for
avoiding spurious correlations (Tu et al., 2020). To better evaluate LMs,
Dhingra et al. (2022) use a smaller, carefully curated diagnostic dataset,
Soler and Apidianaki (2021) control for parameters highly correlated with the
target phenomenon, Ettinger (2020) and Chang and Bergen (2022) draw insights
from cognitive sciences. Our approach further contributes to such efforts. By
making measurements at different critical regions and using targeted
psycholinguistics diagnostic datasets to perform a series of connected
experiments, we demonstrate the inadequacy of drawing conclusions based on
each single probing experiment alone and the need to prioritize the depth of
analysis.

Another line of relevant works study phenomena where meaning in language is
not explicitly expressed, such as in idioms and similes (Chakrabarty et al.,
2022), metaphor processing (Djokic et al., 2020), relations between entities
(Elazar et al., 2021a), as well as implicit reasoning strategies for question-
answering (Geva et al., 2021), and sentence decontextualization (Choi et al.,
2021). On the phenomenon of coercion, Elazar et al. (2020) was the first to
crowdsource data for it, but achieved low agreement scores between annotators
(Fleiss’ Kappa, $k=0.24$) (Fleiss, 1971), and the phenomenon has been
otherwise left unexplored. Our work circumvents this issue by using
diagnostics from psycholinguistic experiments, and fills a gap in this space.

##  7 Discussion and conclusion

This work presents a systematic study examining LMs’ behavior when processing
sentences with implicit meaning, using the complement coercion phenomenon,
which has been largely left unexplored. Effects were observed at the target
region rather than the minimally differing verb region, highlighting that
differences in language can be more than what meets the eye and the importance
of looking at multiple sentence regions critical for the phenomenon. Our
series of experiments designed to ask targeted questions are effective in
identifying potential confounds which would be otherwise mistaken for coercion
effects. We arrive at a richer and more accurate analysis by making
measurements at various critical regions in sentences (measure more) and
having a series of targeted follow-up experiments (question more),
methodologies that can be potentially applied to various future works dealing
with language in general. We hope our work can inspire future efforts to
measure more and question more.

## References

  * Barker (2016) Chris Barker. 2016.  Why relational nominals make good concealed questions.  _Lingua_ , 182(2):12–29.
  * Ben-David et al. (2022) Eyal Ben-David, Nadav Oved, and Roi Reichart. 2022.  PADA: Example-based Prompt Learning for on-the-fly Adaptation to Unseen Domains.  _Transactions of the Association for Computational Linguistics_ , 10(0):414–433.
  * Brennan and Pylkkänen (2008) Jonathan Brennan and Liina Pylkkänen. 2008.  Processing events: Behavioral and neuromagnetic correlates of aspectual coercion.  _Brain and Language_ , 106(2):132–143.
  * Chakrabarty et al. (2022) Tuhin Chakrabarty, Yejin Choi, and Vered Shwartz. 2022.  It’s not Rocket Science: Interpreting Figurative Language in Narratives.  _Transactions of the Association for Computational Linguistics_ , 10(0):589–606.
  * Chang and Bergen (2022) Tyler Chang and Benjamin Bergen. 2022.  Word Acquisition in Neural Language Models.  _Transactions of the Association for Computational Linguistics_ , 10(0):1–16.
  * Chen and Goodman (1999) Stanley F. Chen and Joshua Goodman. 1999.  An empirical study of smoothing techniques for language modeling.  _Computer Speech & Language_, 13(4):359–394.
  * Choi et al. (2021) Eunsol Choi, Jennimaria Palomaki, Matthew Lamm, Tom Kwiatkowski, Dipanjan Das, and Michael Collins. 2021.  Decontextualization: Making Sentences Stand-Alone.  _Transactions of the Association for Computational Linguistics_ , 9(0):447–461.
  * Delogu et al. (2017) Francesca Delogu, Matthew W. Crocker, and Heiner Drenhaus. 2017.  Teasing apart coercion and surprisal: Evidence from eye-movements and ERPs.  _Cognition_ , 161:46 – 59.
  * Demberg and Keller (2008) V. Demberg and Frank Keller. 2008.  Data from eye-tracking corpora as evidence for theories of syntactic processing complexity.  _Cognition_ , 109:193–210.
  * Dhingra et al. (2022) Bhuwan Dhingra, Jeremy Cole, Julian Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William Cohen. 2022.  Time-Aware Language Models as Temporal Knowledge Bases.  _Transactions of the Association for Computational Linguistics_ , 10(0):257–273.
  * Djokic et al. (2020) Vesna G. Djokic, Jean Maillard, Luana Bulat, and Ekaterina Shutova. 2020.  Decoding Brain Activity Associated with Literal and Metaphoric Sentence Comprehension using Distributional Semantic Models.  _Transactions of the Association for Computational Linguistics_ , 8(0):231–246.
  * Dölling (1992) Johannes Dölling. 1992.  Polysemy and sort coercion in semantic representations. Discourse and lexical meaning.  In _Proceedings of a Workshop of the DFG Sonderforschungsbereich_ , 340, page 61–78. ed. by Peter Bosch and Peter Gerstl.
  * Elazar et al. (2021a) Yanai Elazar, Victoria Basmov, Yoav Goldberg, and Reut Tsarfaty. 2021a.  Text-based NP Enrichment.  _ArXiv_ , abs/2109.12085v2.
  * Elazar et al. (2020) Yanai Elazar, Victoria Basmov, Shauli Ravfogel, Yoav Goldberg, and Reut Tsarfaty. 2020.  The Extraordinary Failure of Complement Coercion Crowdsourcing.  In _Proceedings of the First Workshop on Insights from Negative Results in NLP_ , pages 106–116, Online. Association for Computational Linguistics.
  * Elazar et al. (2021b) Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. 2021b.  Amnesic Probing: Behavioral Explanation With Amnesic Counterfactuals.  _Transactions of the Association for Computational Linguistics_ , 9(0):160–175.
  * Ettinger (2020) Allyson Ettinger. 2020.  What BERT is Not: Lessons from a New Suite of Psycholinguistic Diagnostics for Language Models.  _Transactions of the Association for Computational Linguistics_ , 8(0):34–48.
  * Fleiss (1971) Joseph L. Fleiss. 1971.  Measuring nominal scale agreement among many raters.  _Psychological Bulletin_ , 76:378–382.
  * Fodor and Lepore (2002) Jerry A. Fodor and Ernie Lepore. 2002.  _The compositionality papers_.  Oxford: Clarendon Press.
  * Geva et al. (2021) Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021.  Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies.  _Transactions of the Association for Computational Linguistics_ , 9(0):346–361.
  * Goldstein et al. (2022) Ariel Goldstein, Zaid Zada, Eliav Buchnik, Mariano Schain, Amy Rose Price, Bobbi Aubrey, Samuel A. Nastase, Amir Feder, Dotan Emanuel, Alon Cohen, Aren Jansen, Harshvardhan Gazula, Gina Choe, Aditi Rao, Catherine Kim, Colton Casto, Lora Fanda, Werner K. Doyle, Daniel Friedman, Patricia Dugan, Lucia Melloni, Roi Reichart, Sasha Devore, Adeen Flinker, Liat Hasenfratz, Omer Levy, Avinatan Hassidim, Michael Brenner, Y. Matias, Kenneth A. Norman, Orrin Devinsky, and Uri Hasson. 2022.  Shared computational principles for language processing in humans and deep language models.  _Nature Neuroscience_ , 25:369 – 380.
  * Groenendijk and Stokhof (1989) Jeroen Groenendijk and Martin Stokhof. 1989.  Type-Shifting Rules and the Semantics of Interrogatives.  _Properties, types and meanings.In Chierchia et al. (Eds.). Semantics issues_ , 2:21–68.
  * Hale (2001) John Hale. 2001.  A Probabilistic Earley Parser as a Psycholinguistic Model.  In _Second Meeting of the North American Chapter of the Association for Computational Linguistics_.
  * Harris et al. (2008) Jesse Harris, Liina Pylkkänen, Brian McElree, and Steven Frisson. 2008.  The cost of question concealment: Eye-tracking and MEG evidence.  _Brain and Language_ , 107(1):44–61.
  * Heim and Kratzer (1998) Irene Heim and Angelika Kratzer. 1998.  _Semantics in generative grammar_.  Oxford: Blackwell Publishers.
  * Herman (1988) Hendriks Herman. 1988.  Type change in semantics: The scope of quantification and coordination.  _E. Klein & J. van Benthem (Eds.),Categories, polymorphism and unification_, page 96–119.
  * Hu et al. (2020) Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger Levy. 2020.  A Systematic Assessment of Syntactic Generalization in Neural Language Models.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1725–1744, Online. Association for Computational Linguistics.
  * Jackendoff (1997) Ray Jackendoff. 1997.  _The architecture of the language faculty_.  MIT Press.
  * Jiang and de Marneffe (2021) Nanjiang Jiang and Marie-Catherine de Marneffe. 2021.  He Thinks He Knows Better than the Doctors: BERT for Event Factuality Fails on Pragmatics.  _Transactions of the Association for Computational Linguistics_ , 9(0):1081–1097.
  * Levy (2008) Roger Levy. 2008.  Expectation-based syntactic comprehension.  _Cognition_ , 106(3):1126 – 1177.
  * McElree et al. (2006a) Brian McElree, Steven Frisson, and Martin J. Pickering. 2006a.  Deferred Interpretations: Why Starting Dickens is Taxing but Reading Dickens Isn’t.  _Cognitive Science_ , 30:113–124.
  * McElree et al. (2006b) Brian McElree, Liina Pylkkänen, Martin J. Pickering, and Matthew J. Traxler. 2006b.  A time course analysis of enriched composition.  _Psychonomic Bulletin and Review_ , 13:53–59.
  * McElree et al. (2001) Brian McElree, Matthew J. Traxler, Martin J. Pickering, Rachel E. Seely, and Ray Jackendoff. 2001.  Reading time evidence for enriched semantic composition.  _Cognition_ , 78:B15–B25.
  * Mikolov et al. (2010) Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jan Černockỳ, and Sanjeev Khudanpur. 2010.  Recurrent neural network based language model.  In _Eleventh annual conference of the international speech communication association_.
  * Montague (1970) Richard Montague. 1970.  Universal grammar.  _Theoria_ , 36:373–398.
  * Partee (1986) Barbara H. Partee. 1986.  Noun phrase interpretation and type-shifting principles.  _J. Groenendijk et al. (Eds.), Studies of discourse representation theory and the theory of generalized quantifiers_ , page 115–143.
  * Partee (1995) Barbara H. Partee. 1995.  Lexical semantics and compositionality.  _An invitation to cognitive science. In L. R. Gleitman & M. Liberman (Eds.). Language_, 1:311–360.
  * Partee and Rooth (1983) Barbara H. Partee and Mats Rooth. 1983.  Generalized conjunction and type ambiguity.  _R. Bauerle, C. Schwarze, & A. von Stechow (Eds.), Meaning, use, and interpretation of language_, page 361–383.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019.  PyTorch: An Imperative Style, High-Performance Deep Learning Library.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, _Advances in Neural Information Processing Systems 32_ , pages 8024–8035. Curran Associates, Inc.
  * Pustejovsky (1991) James Pustejovsky. 1991.  The generative lexicon.  _Computational linguistics_ , 17:409–441.
  * Pustejovsky (1993) James Pustejovsky. 1993.  Type coercion and lexical selection.  _Semantics and the lexicon_ , pages 73–94.
  * Pustejovsky (1995) James Pustejovsky. 1995.  _The generative Lexicon_.  MIT Press.
  * Pylkkänen (2008) Liina Pylkkänen. 2008.  Mismatching Meanings in Brain and Behavior.  _Language and Linguistics Compass_ , 2:712–738.
  * Pylkkänen et al. (2009) Liina Pylkkänen, Andrea E.Martin, Brian McElree, and Andrew Smart. 2009.  The Anterior Midline Field: Coercion or decision making?  _Brain and Language_ , 108(3):184–190.
  * Pylkkänen and McElree (2007) Liina Pylkkänen and Brian McElree. 2007.  An MEG study of silent meaning.  _Journal of Cognitive Neuroscience_ , 19:1905–1921.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.  _OpenAI Blog_ , 1(8).
  * Rayner and Pollatsek (1989) Keith. Rayner and Alexander. Pollatsek. 1989.  _The psychology of reading_.  Prentice Hall Englewood Cliffs, N.J.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter.  _ArXiv_ , abs/1910.01108.
  * van Schijndel and Linzen (2018) Marten van Schijndel and Tal Linzen. 2018.  Modeling garden path effects without explicit hierarchical syntax.  _Cognitive Science_.
  * Schrimpf et al. (2021) Martin Schrimpf, Idan Asher Blank, Greta Tuckute, Carina Kauf, Eghbal A. Hosseini, Nancy Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko. 2021.  The neural architecture of language: Integrative modeling converges on predictive processing.  _Proceedings of the National Academy of Sciences_ , 118(45):e2105646118.
  * Soler and Apidianaki (2021) Aina Garí Soler and Marianna Apidianaki. 2021.  Let’s Play Mono-Poly: BERT Can Reveal Words’ Polysemy Level and Partitionability into Senses.  _Transactions of the Association for Computational Linguistics_ , 9(0):825–844.
  * Traxler et al. (2005) Matthew J. Traxler, Brian McElree, Rihana S. Williams, and Martin J. Pickering. 2005\.  Context effects in coercion: Evidence from eye-movements.  _Journal of Memory and Language_ , 53:1–25.
  * Traxler et al. (2002) Matthew J. Traxler, Martin J. Pickering, and Brian McElree. 2002.  Coercion in sentence processing: Evidence from eye-movements and self-paced reading.  _Journal of Memory and Language_ , 47:530–547.
  * Tu et al. (2020) Lifu Tu, Garima Lalwani, Spandana Gella, and He He. 2020.  An Empirical Study on Robustness to Spurious Correlations using Pre-trained Language Models.  _Transactions of the Association for Computational Linguistics_ , 8(0):621–633.
  * Warstadt et al. (2020) Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Monananey, Wei Peng, Sheng-Fu Wang, and Samuel Bowman. 2020.  BLiMP: The Benchmark of Linguistic Minimal Pairs for English.  _Transactions of the Association for Computational Linguistics_ , 8(0):377–392.
  * Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. 2019.  HuggingFace’s Transformers: State-of-the-art Natural Language Processing.  _ArXiv_ , abs/1910.03771.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020.  Transformers: State-of-the-Art Natural Language Processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.


Missing 'biblatex' package
The bibliography requires the 'biblatex' package.

New York, NY, USA

Association for Computing Machinery

journaltitleSIGARCH Comput. Archit. News
titleNeurosurgeon: Collaborative Intelligence Between the Cloud and Mobile Edge

journaltitleIEEE Transactions on Mobile Computing
noteto appear
titleJointDNN: An Efficient Training and Inference Engine for Intelligent Mobile Cloud Computing Services

booktitleProc. IEEE ICIP'18
titleDeep Feature Compression for Collaborative Object Detection

edition2nd Edition
titleElements of Information Theory

titleAn Information-Theoretic View for Deep Learning

titleAnatomize Deep Learning with Information Theory

titleDeep Learning and the Information Bottleneck Principle

booktitleProc. ICLR
titleVery Deep Convolutional Networks for Large-Scale Image Recognition

booktitleProc. MICCAI
titleU-Net: Convolutional Networks for Biomedical Image Segmentation

booktitleProc. IEEE CVPR'16
titleDeep Residual Learning for Image Recognition

noteNeurIPS'19 demo
titleShared Mobile-Cloud Inference for Collaborative Intelligence

titleBatch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

titleHow Does Batch Normalization Help Optimization?

Morgan Kaufmann

edition5th Edition
titleIntroduction to Data Compression

journaltitleInternational Journal of Computer Vision (IJCV)
titleImageNet Large Scale Visual Recognition Challenge
family=Independent JPEG Group,

International Telecommunication Union

titleInformation Technology – Digital Compression and Coding of Continuous-tone Still Images – Requirements and Guidelines

Elsevier Science Publishers B. V.

journaltitleDigit. Investig.
titleUsing JPEG Quantization Tables to Identify Imagery Processed by Software
JPEG,Digital ballistics,Quantization,Calvin,Image authentication

journaltitleIEEE Transactions on Information Forensics and Security
titleDigital Image Authentication From JPEG Headers

Kluwer Academic Publishers

titleJPEG2000: Image Compression Fundamentals, Standards, and Practice
family=Image and Signal Processing Group,

Elsevier - Academic Press

edition2nd Edition
titleMultidimensional Signal, Image, and Video Processing and Coding

journaltitleIEEE Access
titleTensor Completion Methods for Collaborative Intelligence

titleTowards Collaborative Intelligence Friendly Architectures for Deep Learning

RFC Editor

RFC Editor

howpublishedInternet Requests for Comments
titleCongestion Control in IP/TCP Internetworks

booktitleProc. IEEE ICIP'19
titleMulti-Task Learning with Compressible Features for Collaborative Intelligence

booktitleProc. IEEE MMSP'18
titleNear-Lossless Deep Feature Compression for Collaborative Intelligence

booktitleProc. IEEE ICASSP'20)
titleBack-And-Forth Prediction for Deep Tensor Compression

booktitleProc. IEEE ICME'20
noteto appear
titleLightweight compression of neural network feature tensors for collaborative intelligence

journaltitleIEEE Transactions on Image Processing
titleToward Intelligent Sensing: Intermediate Deep Feature Compression