11institutetext:  Space Research & Planetary Sciences, Physics Institute,
University of Bern, Sidlerstrasse 5, CH-3012 Bern Switzerland 22institutetext:
Faculty of Aerospace Engineering, Delft University of Technology, Delft, The
Netherlands
22email<EMAIL_ADDRESS>33institutetext: Niels Bohr Institute,
University of Copenhagen, Øster Voldgade 5-7, 1350, Copenhagen K, Denmark

# Mind the trap

Non-negligible effect of volatile trapping in ice on C/O ratios in
protoplanetary disks and exoplanetary atmospheres

N.F.W. Ligterink 1122 K.A. Kipfer 11 S. Gavino 33

(Received April 16, 2024; accepted June 17, 2024)

###### Abstract

Aims. The ability of bulk ices (H2O, CO2) to trap volatiles has been well
studied in any experimental sense, but largely ignored in protoplanetary disk
and planet formation models as well as the interpretation of their
observations. We demonstrate the influence of volatile trapping on C/O ratios
in planet-forming environments.

Methods. We created a simple model of CO, CO2, and H2O snowlines in
protoplanetary disks and calculated the C/O ratio at different radii and
temperatures. We included a trapping factor, which partially inhibits the
release of volatiles (CO, CO2) at their snowline and releases them instead,
together with the bulk ice species (H2O, CO2) . Our aim has been to assess its
influence of trapping solid-state and gas phase C/O ratios throughout planet-
forming environments.

Results. Volatile trapping significantly affects C/O ratios in protoplanetary
disks. Variations in the ratio are reduced and become more homogeneous
throughout the disk when compared to models that do not include volatile
trapping. Trapping reduces the proportion of volatiles in the gas and, as
such, reduces the available carbon- and oxygen-bearing molecules for gaseous
accretion to planetary atmospheres. Volatile trapping is expected to also
affect the elemental hydrogen and nitrogen budgets.

Conclusions. Volatile trapping is an overlooked, but important effect to
consider when assessing the C/O ratios in protoplanetary disks and exoplanet
atmospheres. Due to volatile trapping, exoplanets with stellar C/O have the
possibility to be formed within the CO and CO2 snowline.

###### Key Words.:

Astrochemistry – Molecular processes – Protoplanetary disks – Planets and
satellites: atmospheres

##  1 Introduction

Water (H2O), carbon dioxide (CO2), and carbon monoxide (CO) are some of the
most abundant molecules found in the interstellar medium and planet-forming
environments, specifically on ice-coated dust grains (Boogert et al., 2015).
Due to their prominence and widespread availability, the physicochemical
processes involving these molecules have been extensively studied in the
laboratory (e.g. Hama & Watanabe, 2013; Van Dishoeck et al., 2013; Linnartz et
al., 2015). This includes their thermal desorption behaviour and the interplay
of these molecules during desorption. Their desorption temperatures differ
significantly, with the monolayer coverage peak desorption temperatures
measured in the laboratory, ranging from $\sim$30–40 K for the hypervolatile
CO, $\sim$80–90 K for CO2, and to around 155–175 K for H2O (Minissale et al.,
2022). In fact, out of all prominent ice mantle components (e.g. also
including CH4, NH3, and CH3OH), water is the least volatile. Combined with its
high abundance, H2O ice serves as an important surface onto which molecules
can adsorb and react, but also as a medium into which volatile species can be
trapped (Bar-Nun et al., 1985; Collings et al., 2004). This way of locking up
atoms and molecules is sometimes listed as mechanical trapping, entrapment, or
physical trapping, but it is always related to the ability of a bulk medium to
cover up a single or cluster of species and preserve them in the ice matrix.
Laboratory experiments have shown that large fractions of volatile molecules
can be trapped well above their peak desorption temperature. An example is
given in Fig. 1, which shows a temperature programmed desorption (TPD) trace
of a mixed H2O:CO2:CO ice adapted from Kipfer et al. (2024a). As the ice is
linearly heated, the release of these molecules is followed with a mass
spectrometer. The molecules sublimate at their characteristic pure
temperatures (1, 2, 4), but CO and CO2 also release well above their
desorption temperature. Carbon monoxide co-releases with CO2 (2), while CO and
CO2 both also release with the water sublimation event. This happens in the
form of co-desorption with H2O at $\sim$170 K (4) and in the form of a volcano
desorption event at $\sim$145 K (3), which occurs when water-ice rearranges
from an amorphous to crystalline structure and volatile species are expulsed
from the ice (Burke & Brown, 2010). Water ice can trap fractions of over tens
of percent and up to a hundred percent of added volatiles, depending on the
molecule or atom involved, ice structure, ice formation temperature, mixing
ratios, and deposition rate (Bar-Nun et al., 1985; Notesco et al., 2003;
Collings et al., 2004; Fayolle et al., 2011; Almayrac et al., 2022; Simon et
al., 2023; Kipfer et al., 2024a). Similar trapping efficiencies have been
found for CO2 ice as bulk medium (Simon et al., 2019). A more detailed
description of how desorption and trapping fractions are determined in
laboratory experiments is provided in Appendix A. The trapping of volatiles
affects the physical and chemical processes in star and planet-forming
environments, but nuanced differences are expected between trapping in a
laboratory and natural environment. For example, Ciesla et al. (2018) showed
that efficient trapping takes place when the timescale for a volatile species
to desorb is longer than the time it takes to be covered by a monolayer of
water; this is the so-called ‘burial regime’. These conditions are met at high
deposition rates, as usually used in laboratory experiments, or when surface
temperatures are low. Considering the low temperatures at which H2O formation
takes place on interstellar dust grains (Hama & Watanabe, 2013), it is
expected that experimentally determined trapping efficiencies apply to
interstellar environments. However, slow deposition in interstellar
environments and at elevated temperatures will likely result in different
trapping efficiencies.

Figure 1: Thermal desorption of a mixed H2O:CO2:CO ice film at 100:15:5 ratio
recorded during a Temperature Programmed Desorption (TPD) experiment, where
the release of each molecule is measured with a mass spectrometer. Carbon
monoxide was traced with the 13CO isotope. Four distinct release events are
visible: pure-phase CO (1), pure-phase CO2 with CO co-desorbing (2), volcano
desorption of CO and CO2 (3), and water desorption with co-desorption of CO
and CO2 (4). Desorption and trapping fractions of volatiles can be determined
by integrating the TPD trace over discrete temperature ranges. The data are
adapted from Kipfer et al. (2024a).

Planets are thought to form in protoplanetary disks by the accretion of gas
and ice-coated dust grains, pebbles, or planetesimals (e.g. Johansen et al.,
2007; Lambrechts & Johansen, 2012; Bitsch et al., 2019; Lambrechts et al.,
2019). The bulk molecular species contained in solids and in gas play a
prominent role in setting the elemental composition of a planet. Öberg et al.
(2011b) presents a model of the radially varying gas phase and solid-state
carbon-over-oxygen (C/O) ratio in a protoplanetary disk, due the step-wise
release of CO, CO2, and H2O from the ice to the gas. The model predicts that
the C/O ratio of a planetary atmosphere depends on which side of the major
volatile (CO/CO2/H2O) snowlines the object is formed. A high C/O ratio ($>$
0.8) indicates that the formation was initiated outside the snowlines prior to
inward migration (Madhusudhan et al., 2014). An exoplanet transit is typically
used to derive the atmospheric composition and a significant number of C/O
ratio values have now been measured (Hoch et al., 2023). The Öberg et al.
(2011b) model has been used to determine the formation location of exoplanets
based on their observed atmospheric C/O ratio (Mollière et al., 2020; Zhang et
al., 2021). For instance, the supersolar gas phase C/O ratio ($\gtrapprox$
0.55) outside the water snowline was used to explain the C/O $\geq$ 1 ratio
measured in the atmosphere of the exoplanet WASP-12b (Madhusudhan et al.,
2011). Conversely, the low C/O ratio measured in the atmosphere of WASP-77Ab
is interpreted as a formation scenario where the atmospheric material is
accreted inside the major snowlines (Line et al., 2021). In the last decade,
more refined models to predict protoplanetary disk snowlines and C/O ratios
have been presented in the literature, which includes solid-state and gas
phase chemical processes, grain growth, or novel physical processes (Owen,
2020; Notsu et al., 2020; Cridland et al., 2020; Zhang et al., 2020a; Eistrup
et al., 2022; Gavino et al., 2023). However, the trapping of volatiles within
the matrix of bulk H2O and/or CO2 has largely been ignored in the models
(Schneeberger et al., 2023), despite that formalisms for volatile trapping
have been included in a select number of astrochemical models (Viti et al.,
2004; Visser et al., 2009; Taquet et al., 2012; Garrod et al., 2022).

In this paper, we revisit the static model of Öberg et al. (2011b) and include
volatile trapping to show that this has a non-negligible effect on the C/O
ratio in a protoplanetary disk and (exo)planets that form there. In Section 2,
we describe the modified model and show how trapping affects solid-state and
gas phase C/O ratios in protoplanetary disks. In Section 3, these results and
their implications for protoplanetary disk and exoplanet atmosphere C/O ratios
are discussed.

##  2 Model and results

To determine the protoplanetary disk C/O ratio, we adopted the static snowline
model presented in Öberg et al. (2011b). In short, this model assumes that CO,
CO2, and H2O ices fully sublimate at distinct sublimation temperatures, that
is, their respective snowlines. At each snowline, one molecular species
desorbs, which results in the depletion of solid carbon and/or oxygen, while
simultaneously setting or modifying the gas phase elemental budget. Carbon
grains and silicates are significant elemental carriers, but sublimate at much
higher temperatures than H2O. Therefore, they remain solid in the temperature
range used in this model and only affect the solid carbon and oxygen budget.
The model parameters, namely, the sublimation temperatures and elemental
abundances per molecule, are presented in Table 1. The relative abundances
between H2O, CO2, and CO are (in that order) 0.9:0.3:1.5 and are derived from
the CBRR 2422.8-3423 protoplanetary disk (Pontoppidan, 2006). We note that
sublimation temperatures differ from the aforementioned laboratory desorption
temperatures, due to the difference in heating timescale. Snowlines are
plotted on a protoplanetary disk temperature profile following the equation
$T=T_{\rm 0}r^{-q}$, where $T_{\rm 0}$ = 200 K, $r$ is the radius in a.u., and
$q$ is the power law index of 0.62 (Andrews & Williams, 2007), where smaller
radii correspond to higher temperatures and larger radii to lower
temperatures. The resulting radial gas phase and solid-state C/O distributions
of the original Öberg et al. (2011b) model (black line) are shown in Fig. 2.
This model does not take volatile trapping into account.

Table 1:  Model parameters

Molecule | $T$ | $n_{\rm O}$ | $n_{\rm C}$
---|---|---|---
| (K) | 10-4 $\times$ $n_{\rm H}$ | 10-4 $\times$ $n_{\rm H}$
CO | 20 | 1.5 | 1.5
CO2 | 47 | 0.6 | 0.3
H2O | 135 | 0.9 | –
Carbon grains | 500 | – | 0.6
Silicate | 1500 | 1.4 | –
111Model parameters are adopted from Öberg et al. (2011b), which in turn are
based on data from Pontoppidan (2006), Draine (2003), and Whittet (2010).

Figure 2: Gas phase and solid-state C/O ratios at various radii in a
protoplanetary disk, based on the trapping and release of CO, CO2, and H2O.
The original model of Öberg et al. (2011b) without trapping is presented in
black. The updated model takes the trapping of CO and CO2 in H2O ($F_{\rm
H_{2}O}$(CO,CO2)) and CO in CO2 ($F_{CO_{2}}$(CO)) at different fractions into
account (blue, green, and red lines) and shows that the solid C/O ratio is
strongly affected. The snowlines of CO, CO2, and H2O are indicated.  Figure 3:
Trapping efficiencies of CO and CO2 in water ($F_{\rm H_{2}O}$(CO,CO2)), and
CO in CO2 ($F_{\rm CO_{2}}$(CO)) plotted against the host: volatile ice ratio
(left) and the ice layer thickness (right). The thickness is given in
monolayer (ML), where 1 ML = 1015 molecules cm-2. The grey shaded boxes show
the H2O:CO and H2O:CO2 ice ratios observed in a variety of interstellar
sources, while the dashed lines show their largest and smallest median values
for data presented in Boogert et al. (2015). The laboratory data used in this
plot are presented in Table 2.

We modified the above model to include volatile trapping by releasing a
fraction of the volatiles with water ($F_{\rm H_{2}O}$(CO,CO2)) or CO2
($F_{\rm CO_{2}}$(CO)). Volatile release associated with water sublimation
occurs in two ways: volcano desorption, namely, the rapid release of volatiles
during the amorphous to crystalline phase change of ice, and co-desorption,
namely, the release of volatiles together water desorption (Burke & Brown,
2010). Volcano desorption occurs at a lower temperature than the peak water
sublimation temperature by approximately $\sim$20% (Collings et al., 2004;
Kipfer et al., 2024a). Experimental studies show that significant volatile
release starts with (or is even dominated by) volcano desorption and continues
until all volatiles are depleted during the water co-desorption event (Martín-
Doménech et al., 2014). In the modified static model we release all volatiles
interior of the water snowline, but we note that under realistic conditions
loss of some CO and CO2 will also occur leading up to the water snowline. Co-
desorption of volatiles with CO2 occurs at the CO2 sublimation temperature
(Simon et al., 2019; Kipfer et al., 2024a) and this is adopted as such in the
modified model.

Over recent decades, many studies have determined which fractions of volatiles
are trapped by H2O and CO2 (e.g. Bar-Nun et al., 1985; Collings et al., 2004;
Fayolle et al., 2011; Ligterink et al., 2018; Simon et al., 2023; Kipfer et
al., 2024a). Figure 3 shows a compilation of the literature results and
presents trapping fractions plotted against the matrix: volatile mixing ratio
and the ice layer thickness given in monolayers (ML, 1 ML = 1015 molecules
cm-2). The data and methods to determine trapping fractions are described in
Appendix A. In general, thicker ice films trap more volatiles, while a large
volatile concentration results in a smaller fraction of volatiles being
trapped. From this overview, we can say that trapping fractions in an ice
mantle are $F_{\rm CO_{2}}$(CO) = 0.1 – 0.7, $F_{\rm H_{2}O}$(CO2) = 0.3 –
0.95, and $F_{\rm H_{2}O}$(CO) = 0.2 – 0.7 for mixing ratios $\geq$5:1. We
test the effect of various realistic trapping fractions in a series of model
runs.

Figure 2 shows three models with water trapping fractions of $F_{\rm
H_{2}O}$(CO,CO2) = 0.3, 0.5, and 0.8. Both CO and CO2 are trapped in the same
fraction. Trapping by CO2 is ignored. We find that trapping has a significant
effect on the solid-state C/O ratio, while it does not affect its gas phase
counterpart. As larger fractions of CO and CO2 are trapped in water ice, the
variations in solid-state C/O ratio are reduced and can even be considered
constant from outside the CO snowline up to the H2O snowline for the largest
fraction of trapping.

Figure 4 shows three models where CO and CO2 are trapped in different
fractions in water, at $F_{\rm H_{2}O}$(CO,CO2) = (0.3;0.5), (0.5;0.8), and
(0.3;0.8). In the solid-state, this again results in a reduction in C/O
variations. However, this time also the gas phase C/O ratio is affected, which
becomes more homogeneous.

Trapping of CO in CO2 is included in Figs. 6 and 7. In general, reduction of
variations and homogenisation of the solid-state and gas phase C/O ratios is
seen again. The release of a fraction of CO with CO2 causes a more pronounced
drop in solid-state C/O ratio around the CO2 snowline. We conclude that
volatile trapping has a non-negligible effect on both solid-state and gas
phase C/O ratios.

Figure 4: Same as Figure 2, but different trapping ratios of CO and CO2 in
H2O.

##  3 Discussion and Implications

In the following sections, we discuss the results and implications from the
perspective of interstellar ice compositions and laboratory data on trapping
efficiencies, protoplanetary disks, and exoplanet atmospheres.

###  3.1 Ice composition and trapping efficiency

The modified model demonstrates that the trapping of volatiles affects the C/O
ratio by reducing variations to this ratio. The extent of this reduction with
respect to the solid material depends on the fraction of volatile that is
trapped, while for the gas phase, it depends on the size of the difference
between trapped fractions of volatiles. Laboratory experiments show that
volatile trapping is efficient (see Fig. 3), but this efficiency decreases as
the volatile concentration increases. For example, Fayolle et al. (2011)
measured a CO trapping fraction in water of $F_{\rm H_{2}O}$(CO) = 0.04 for a
1:1 H2O:CO ice film of 17 ML. Since the ice composition used in Öberg et al.
(2011b) is 0.9:0.3:1.5 H2O:CO2:CO, that is, CO is $\sim$1.7 times more
abundant than H2O, this led to the claim that volatile trapping is negligible.
While this might be the case for this particular ice composition, this mixing
ratio is not representative of the ice generally found in the interstellar
medium or protoplanetary disks (Boogert et al., 2015). For example, the median
ice compositions presented in Öberg et al. (2011a) range from 100:38:31
H2O:CO2:CO for cloud cores to 100:13:13 for high-mass protostars. Similar
patterns are seen for protoplanetary disks (Aikawa et al., 2012; Sturm et al.,
2023), where H2O is the dominant ice component. Figure 5, shows the model
results for a realistic ice composition of H2O:CO2:CO at a 100:25:25 ratio,
both with and without trapping. The contributions of oxygen from silicates and
carbon from carbonaceous dust are omitted in this model. For mixing ratios of
4:1 H2O to CO2 or CO, $F_{\rm H_{2}O}$(CO2) $\geq$40% and $F_{\rm H_{2}O}$(CO)
$\geq$25% have bee reported (Fayolle et al., 2011; Simon et al., 2023). As can
be seen in Fig. 5, a trapping efficiency of 30% significantly affects the C/O
ratio and causes the same kind of homogenisation as observed in the fiducial
model.

Figure 5: Same as Figure 2, but only for H2O, CO2, and CO at 100:25:25 ratio.
The contributions of silicates to atomic oxygen and carbonaceous dust to
atomic carbon are omitted.

Trapping of volatiles depends on the structure of the ice. For example, when
volatile species are covered by a layer of water instead of mixed, the
volatile can be fully trapped (May et al., 2013a, b, where trapping is defined
as the release of volatiles during volcano and co-desorption with water
sublimation). Astrochemical models have demonstrated that volatiles can be
concentrated in the bottom layers of the ice mantle and subsequently be
covered by H2O as an ice mantle grows Garrod et al. (2022). Furthermore, a
recent experimental work by Potapov et al. (2023) has shown that the UV
irradiation of carbonaceous dust covered by H2O ice results in significant
formation of CO2 at the dust-ice interface. Thus, covering of volatiles on
ice-coated dust grains is plausible in planet-forming regions and results in
trapping fractions that are substantially larger than those of mixed ice.

Most experimental trapping studies make use of binary ice mixtures and show
that volatile trapping is efficient. However, this efficiency (that is, that
of the trapped fraction) can decrease as more volatiles are included in the
ice. For example, Kipfer et al. (2024a) shows that the fraction of N2 trapped
in water-ice is influenced by the amount of CO2 that is included in N2:CO2:H2O
mixtures and decreases as more CO2 is incorporated. This observation has been
linked to competition for binding sites when multiple volatile species are
present in a water-ice matrix (e.g. Simon et al., 2023; Ligterink et al.,
2024). Ice mantles of grains in planet-forming environments often consist of
more than five dominant components and understanding the trapping behaviour of
such multicomponent mixtures is relevant to assessing how trapping affects C/O
ratios. However, the number of studies that have been conducted on ice systems
that approach the complexity of interstellar ice mantles is limited. For
example, analysis of TPD data of H2O:CO:CO2:CH3OH:NH3 mixed ice systems
presented in Martín-Doménech et al. (2014) suggests that trapping fractions of
CO and CO2 are similar to those of binary and ternary systems (see Fig. 3 and
Appendix A). However, dedicated investigations of multi-component ice films
are needed to determine if this observation holds and how trapping
efficiencies are affected by various ice components and compositions.

In this study, we focus on the elemental carbon and oxygen budget derived from
H2O, CO2, and CO. However, molecules such as methane (CH4) should be
considered as well. This volatile molecule can also be efficiently trapped in
H2O and CO2 (Simon et al., 2023). Furthermore, elemental ratios involving
atomic nitrogen and hydrogen are also of interest and carriers of these atoms
have also been shown to show efficiently in terms of trapping (e.g. N2, see
Kipfer et al., 2024a).

###  3.2 Protoplanetary disk snowlines and missing volatiles

Volatile trapping potentially has two prominent observable effects on
protoplanetary disks. First, it reduces the proportion of volatiles, such as
CO, which are available in the gas phase in regions where the volatile should
not be frozen out. Simultaneous measurements of HD (hydrogen deuteride) and CO
in disks indicate that these objects are depleted in CO by a factor 5 – 100
(McClure et al., 2016; Zhang et al., 2020b). To account for this depletion,
various effects have been proposed, such as pebble growth, physical
sequestration, and chemical processes (e.g. Krijt et al., 2018; Eistrup et
al., 2018). Models that combine these processes can predict a CO depletion
factor close to 100 (Krijt et al., 2020). Volatile trapping in ice will
contribute to such depletion with trapping fractions of up to 95% (see Fig. 3)
for thin film ice. However, orders of magnitude depletion are unlikely for
thin ice films and are likely to require thicker ice mantles as found on
pebbles (Bosman et al., 2018).

Second, trapping tends to ‘inject’ volatiles at multiple locations in the disk
into the gas phase, resulting in a step-wise increase of volatile abundances
towards the protostar. For example, CO will enter the gas phase when its pure-
phase ice sublimates, when CO2 desorbs, and at the onset of H2O sublimation.
Observations of such step-wise increases might be challenging, due to limited
spatial resolution, optically thick emission, excitation temperatures, and the
usage of indirect chemical tracers (e.g. N2H+, van’t Hoff et al., 2017).
However, hints of it might be seen in some studies, such as observations
performed by Zhang et al. (2020a), where an enhanced gas phase CO level and
C/H ratio were found inside the CO snowline of the HD 163296 protoplanetary
disk. These authors assigned this observation to pebble drift, but an
alternative explanation relates to the release of carbon-bearing volatiles
during the water ice volcano desorption event (see Fig. 1). This increases the
gaseous carbon budget, while gas phase oxygen and hydrogen levels remain
comparatively low because H2O does not yet sublimate.

Other mechanisms can contribute to a step-wise increasing emission or double
emission line pattern in a disk. The presence of pure multiple snowlines of
one volatile species can be due to the presence of multiple dust surface
temperatures. Using a two-sized model, Gavino et al. (2023) predicted two pure
CO snowlines in a T Tauri disk, the first one at a few tens of au, the second
at 100-150 au from the central star. This effect does not require radial drift
and relies only on the presence of multiple dust temperatures. The presence of
multiple dust sizes and temperatures can also contribute to create a snowline
shape that is spread out radially and vertically (Gavino et al., 2021) rather
than marking an abrupt step-wise emission. Radial drift can also affect
volatile snowlines. Cleeves (2016), using a simple parameterised model, showed
that the removal of large grains in the outer disk reshapes the thermal
structure that can create multiple CO snowlines.

Trapped volatiles may be chemically processed into more complex molecules and
thus removed from the ice. As ice-coated grains near a protostar, their
temperatures and amounts of impinging radiation increase, which results in
enhanced chemical reactions (Potapov et al., 2023). Under these conditions, CO
embedded in H2O-ice is efficiently converted into CO2 (van Scheltinga et al.,
2022). In turn, CO2 thermally reacts with NH3 to form carbamic acid (NH2COOH,
Khanna & Moore, 1999; Noble et al., 2014; Marks et al., 2023). The formation
of much more complex and refractory molecules is possible as well (e.g. Qasim
et al., 2023; Kipfer et al., 2024b), which will desorb at much higher
temperatures ($\sim$200 – 400 K) and much closer to protostar (Ligterink &
Minissale, 2023).

Finally, it is worth pointing out that recent edge-on observations of HH 48 NE
protoplanetary disk with JWST revealed that CO ice features extend to large
disk heights, where the temperature is likely to be high enough that pure,
surface-bound, CO ice should have sublimated (Sturm et al., 2023). One
interpretation of this observation is that CO is trapped in other bulk ice
components. This notion is supported by the fact that the CO ice IR feature
observed in the disk is broader than that of pure CO measured in the
laboratory, which can be the result of interactions of CO with a H2O-rich
environment. Therefore, this observation highlights the importance of volatile
trapping and the effects it may have on the disk chemistry.

###  3.3 C/O ratio in exoplanet atmospheres

Exoplanets are thought to inherit their elemental abundances from the gas and
solids at the location where they form in the parent protoplanetary disk. As
such, the measured relative abundance of their chemical content can be used as
a diagnostic of their formation pathway. In particular, the Öberg et al.
(2011b) model shows that the C/O ratio in hot Jupiters atmospheres is expected
to be a signpost for their formation location relative to the main volatile
snowlines in the disk. The planets that accrete their envelope inside the
water snowline result in an oxygen-rich atmosphere (assuming a carbon-depleted
disk) and a subsolar C/O ratio (e.g. Cridland et al., 2016; Mordasini et al.,
2016), which favours in situ formation scenarios. The model also predicts that
atmospheres with superstellar C/O ratios imply the accretion of gas between
the H2O and CO snowline (and subsequent inward migration after the gas is
dissipated), substellar C/O ratios imply the accretion of solids between the
H2O and CO snowline, and a stellar C/O ratio suggests that the exoplanet
formed outside the CO snowline (e.g. Madhusudhan et al., 2011, 2014, 2017;
Mollière et al., 2020; Zhang et al., 2021; Line et al., 2021). However, when
volatile trapping is included, we find that solid C/O ratios outside and
inside the CO snowline are similar, see Fig. 2. Therefore, the observation of
a stellar C/O ratio in an exoplanet atmosphere does not automatically imply
that this object was formed outside the CO snowline. Instead, stellar C/O can
likely be acquired due to the formation of the planet between the H2O and CO
snowline. Furthermore, this can affect the ice chemical composition during
inward pebbles drift and the subsequent elemental composition of the planet
formed. In this scenario, the grains migrate and carry the frozen CO or CO2 on
the surface to their respective snowline, increasing locally the gas phase
abundances (e.g. Booth et al., 2017). Volatile trapping should reduce the
chemical enhancement at snowline locations while the drifting pebble surfaces
retain more CO and CO2 ice than predicted. Superstellar C/O ratios can still
be explained by the accretion of gases, although volatile trapping does reduce
the amount of available gaseous material.

##  4 Conclusions

In this study, we demonstrate that volatile trapping in bulk ice strongly
affects C/O ratios in protoplanetary disks. Strong variations in the C/O
ratios seen in models without trapping have been reduced. Solid elemental
ratios become constant from outside the CO snowline to inside the H2O
snowline. While gas phase C/O ratios are still enhanced, trapping reduces the
amount of carbon- and oxygen-bearing molecules available in the gas. A similar
effect is expected for volatile molecules that set the budget of other
elements, such as nitrogen and sulfur. The homogenisation of the C/O ratios
ensures that planets can acquire a stellar C/O ratio outside and inside the CO
snowline. Observations of C/O ratios in exoplanet atmospheres are therefore
difficult to link to a specific formation location in a protoplanetary disk.

###### Acknowledgements.

The authors thank E.G. Bøgelund, S.F. Wampfler, J. Drążkowska, D. Semenov, H.
Cuppen and M.N. Drozdovskaya for useful discussions, and A. Cridland for
feedback on the manuscript. N.F.W.L. and K.A.K. acknowledge support from the
Swiss National Science Foundation (SNSF) Ambizione grant 193453 and NCCR
PlanetS. S.G. acknowledge support from the Independent Research Fund Denmark
(grant No. 0135-00123B).

## References

  * Aikawa et al. (2012) Aikawa, Y., Kamuro, D., Sakon, I., et al. 2012, Astronomy & Astrophysics, 538, A57
  * Almayrac et al. (2022) Almayrac, M. G., Bekaert, D. V., Broadley, M. W., et al. 2022, The Planetary Science Journal, 3, 252
  * Andrews & Williams (2007) Andrews, S. M. & Williams, J. P. 2007, ApJ, 659, 705
  * Bar-Nun et al. (1985) Bar-Nun, A., Herman, G., Laufer, D., & Rappaport, M. 1985, Icarus, 63, 317
  * Bitsch et al. (2019) Bitsch, B., Izidoro, A., Johansen, A., et al. 2019, A&A, 623, A88
  * Boogert et al. (2015) Boogert, A. A., Gerakines, P. A., & Whittet, D. C. 2015, ARA&A, 53, 541
  * Boogert et al. (2015) Boogert, A. C. A., Gerakines, P. A., & Whittet, D. C. B. 2015, ARAA, 53, 541
  * Booth et al. (2017) Booth, R. A., Clarke, C. J., Madhusudhan, N., & Ilee, J. D. 2017, MNRAS, 469, 3994
  * Bosman et al. (2018) Bosman, A. D., Tielens, A. G., & van Dishoeck, E. F. 2018, Astronomy & Astrophysics, 611, A80
  * Burke & Brown (2010) Burke, D. J. & Brown, W. A. 2010, PCCP, 12, 5947
  * Ciesla et al. (2018) Ciesla, F. J., Krijt, S., Yokochi, R., & Sandford, S. 2018, The Astrophysical Journal, 867, 146
  * Cleeves (2016) Cleeves, L. I. 2016, ApJ, 816, L21
  * Collings et al. (2004) Collings, M. P., Anderson, M. A., Chen, R., et al. 2004, MNRAS, 354, 1133
  * Cridland et al. (2020) Cridland, A. J., Bosman, A. D., & van Dishoeck, E. F. 2020, Astronomy & Astrophysics, 635, A68
  * Cridland et al. (2016) Cridland, A. J., Pudritz, R. E., & Alessi, M. 2016, MNRAS, 461, 3274
  * Draine (2003) Draine, B. T. 2003, Annual Review of Astronomy and Astrophysics, 41, 241
  * Eistrup et al. (2022) Eistrup, C., Cleeves, L. I., & Krijt, S. 2022, Astronomy & Astrophysics, 667, A121
  * Eistrup et al. (2018) Eistrup, C., Walsh, C., & van Dishoeck, E. F. 2018, Astronomy & Astrophysics, 613, A14
  * Fayolle et al. (2011) Fayolle, E. C., Öberg, K., Cuppen, H. M., Visser, R., & Linnartz, H. 2011, Astronomy & Astrophysics, 529, A74
  * Garrod et al. (2022) Garrod, R. T., Jin, M., Matis, K. A., et al. 2022, The Astrophysical Journal Supplement Series, 259, 1
  * Gavino et al. (2021) Gavino, S., Dutrey, A., Wakelam, V., et al. 2021, A&A, 654, A65
  * Gavino et al. (2023) Gavino, S., Kobus, J., Dutrey, A., et al. 2023, Astronomy & Astrophysics, 680, A59
  * Gudipati et al. (2023) Gudipati, M. S., Fleury, B., Wagner, R., et al. 2023, Faraday Discuss., 245, 467
  * Hama & Watanabe (2013) Hama, T. & Watanabe, N. 2013, Chemical reviews, 113, 8783
  * Hoch et al. (2023) Hoch, K. K. W., Konopacky, Q. M., Theissen, C. A., et al. 2023, AJ, 166, 85
  * Johansen et al. (2007) Johansen, A., Oishi, J. S., Mac Low, M.-M., et al. 2007, Nature, 448, 1022
  * Khanna & Moore (1999) Khanna, R. & Moore, M. 1999, Spectrochimica Acta Part A: Molecular and Biomolecular Spectroscopy, 55, 961
  * Kipfer et al. (2024a) Kipfer, K., Ligterink, N., Rubin, M., et al. 2024a, Astronomy & Astrophysics, 686, A102
  * Kipfer et al. (2024b) Kipfer, K. A., Galli, A., Riedo, A., et al. 2024b, Icarus, 410, 115742
  * Kouchi & Yamamoto (1995) Kouchi, A. & Yamamoto, T. 1995, Progress in crystal growth and characterization of materials, 30, 83
  * Krijt et al. (2020) Krijt, S., Bosman, A. D., Zhang, K., et al. 2020, ApJ, 899, 134
  * Krijt et al. (2018) Krijt, S., Schwarz, K. R., Bergin, E. A., & Ciesla, F. J. 2018, The Astrophysical Journal, 864, 78
  * Lambrechts & Johansen (2012) Lambrechts, M. & Johansen, A. 2012, A&A, 544, A32
  * Lambrechts et al. (2019) Lambrechts, M., Morbidelli, A., Jacobson, S. A., et al. 2019, A&A, 627, A83
  * Ligterink et al. (2024) Ligterink, N., Kipfer, K., Rubin, M., et al. 2024, A&A
  * Ligterink & Minissale (2023) Ligterink, N. & Minissale, M. 2023, Astronomy & Astrophysics, 676, A80
  * Ligterink et al. (2018) Ligterink, N., Walsh, C., Bhuin, R., et al. 2018, A&A, 612, A88
  * Line et al. (2021) Line, M. R., Brogi, M., Bean, J. L., et al. 2021, Nature, 598, 580
  * Linnartz et al. (2015) Linnartz, H., Ioppolo, S., & Fedoseev, G. 2015, International Reviews in Physical Chemistry, 34, 205
  * Madhusudhan et al. (2014) Madhusudhan, N., Amin, M. A., & Kennedy, G. M. 2014, ApJ, 794, L12
  * Madhusudhan et al. (2017) Madhusudhan, N., Bitsch, B., Johansen, A., & Eriksson, L. 2017, MNRAS, 469, 4102
  * Madhusudhan et al. (2011) Madhusudhan, N., Harrington, J., Stevenson, K. B., et al. 2011, Nature, 469, 64
  * Marks et al. (2023) Marks, J. H., Wang, J., Sun, B.-J., et al. 2023, ACS Central Science, 9, 2241
  * Martín-Doménech et al. (2014) Martín-Doménech, R., Caro, G., Bueno, J., & Goesmann, F. 2014, Astronomy & Astrophysics, 564
  * May et al. (2013a) May, R. A., Scott Smith, R., & Kay, B. D. 2013a, The Journal of Chemical Physics, 138
  * May et al. (2013b) May, R. A., Smith, R. S., & Kay, B. D. 2013b, The Journal of Chemical Physics, 138
  * McClure et al. (2016) McClure, M., Bergin, E., Cleeves, L., et al. 2016, The Astrophysical Journal, 831, 167
  * Minissale et al. (2022) Minissale, M., Aikawa, Y., Bergin, E., et al. 2022, ACS Earth Space Chem., 6, 597
  * Mollière et al. (2020) Mollière, P., Stolker, T., Lacour, S., et al. 2020, Astronomy & Astrophysics, 640, A131
  * Mordasini et al. (2016) Mordasini, C., van Boekel, R., Mollière, P., Henning, T., & Benneke, B. 2016, ApJ, 832, 41
  * Noble et al. (2014) Noble, J., Theule, P., Duvernay, F., et al. 2014, PCCP, 16, 23604
  * Notesco et al. (2003) Notesco, G., Bar-Nun, A., & Owen, T. 2003, Icarus, 162, 183
  * Notsu et al. (2020) Notsu, S., Eistrup, C., Walsh, C., & Nomura, H. 2020, Monthly Notices of the Royal Astronomical Society, 499, 2229
  * Öberg et al. (2011a) Öberg, K. I., Boogert, A. A., Pontoppidan, K. M., et al. 2011a, The Astrophysical Journal, 740, 109
  * Öberg et al. (2011b) Öberg, K. I., Murray-Clay, R., & Bergin, E. A. 2011b, The Astrophysical Journal Letters, 743, L16
  * Owen (2020) Owen, J. E. 2020, Monthly Notices of the Royal Astronomical Society, 495, 3160
  * Pontoppidan (2006) Pontoppidan, K. 2006, Astronomy & Astrophysics, 453, L47
  * Potapov et al. (2023) Potapov, A., Semenov, D., Jäger, C., & Henning, T. 2023, The Astrophysical Journal, 954, 167
  * Qasim et al. (2023) Qasim, D., McLain, H. L., Aponte, J. C., et al. 2023, ACS Earth Space Chem.
  * Rubin et al. (2023) Rubin, M., Altwegg, K., Berthelier, J.-J., et al. 2023, Monthly Notices of the Royal Astronomical Society, 526, 4209
  * Schneeberger et al. (2023) Schneeberger, A., Mousis, O., Aguichine, A., & Lunine, J. I. 2023, Astronomy & Astrophysics, 670, A28
  * Simon et al. (2019) Simon, A., Öberg, K. I., Rajappan, M., & Maksiutenko, P. 2019, The Astrophysical Journal, 883, 21
  * Simon et al. (2023) Simon, A., Rajappan, M., & Öberg, K. I. 2023, The Astrophysical Journal, 955, 5
  * Sturm et al. (2023) Sturm, J., McClure, M., Beck, T., et al. 2023, Astronomy & Astrophysics, 679, A138
  * Taquet et al. (2012) Taquet, V., Ceccarelli, C., & Kahane, C. 2012, Astronomy & Astrophysics, 538, A42
  * Van Dishoeck et al. (2013) Van Dishoeck, E. F., Herbst, E., & Neufeld, D. A. 2013, Chemical Reviews, 113, 9043
  * van Scheltinga et al. (2022) van Scheltinga, J. T., Ligterink, N., Bosman, A., Hogerheijde, M., & Linnartz, H. 2022, A&A, 666, A35
  * van’t Hoff et al. (2017) van’t Hoff, M. L., Walsh, C., Kama, M., Facchini, S., & van Dishoeck, E. F. 2017, Astronomy & Astrophysics, 599, A101
  * Visser et al. (2009) Visser, R., van Dishoeck, E. F., Doty, S. D., & Dullemond, C. P. 2009, A&A, 495, 881
  * Viti et al. (2004) Viti, S., Collings, M. P., Dever, J. W., McCoustra, M. R., & Williams, D. A. 2004, Monthly Notices of the Royal Astronomical Society, 354, 1141
  * Whittet (2010) Whittet, D. 2010, The Astrophysical Journal, 710, 1009
  * Zhang et al. (2020a) Zhang, K., Bosman, A. D., & Bergin, E. A. 2020a, The Astrophysical Journal Letters, 891, L16
  * Zhang et al. (2020b) Zhang, K., Schwarz, K. R., & Bergin, E. A. 2020b, The Astrophysical Journal Letters, 891, L17
  * Zhang et al. (2021) Zhang, Y., Snellen, I. A., Bohn, A. J., et al. 2021, Nature, 595, 370

##  Appendix A Trapping fractions

Table 2 shows the trapped fractions of volatiles obtained from various
studies. Trapping fractions are usually determined in the following way. In a
laboratory setup, an ice film consisting of two or more components is prepared
by dosing gas mixture on an ultra-cold (5 – 80 K) surface. Next, the ice is
linearly heated in a process called Temperature Programmed Desorption (TPD)
with heating rates that are generally a few K min-1. A mass spectrometer
records the molecules desorbing from the ice film at their characteristic
mass-over-charge ($m/z$), for example, $m/z$=18 for H2O+, $m/z$=28 for CO+,
and $m/z$=44 for CO${}_{2}^{+}$, see Fig. 1. From the TPD trace, desorption
and trapping fractions are determined by integrating the signal over a
specified temperature regime that corresponds to a certain desorption event.
For example, Kipfer et al. (2024a) determines CO desorption fractions by
integrating between 15–60 K for pure CO, 60–115 K for CO co-desorption with
CO2, and 115–200 K for CO release associated with water (this combines the so-
called volcano desorption event and the water co-desorption into one). Between
different studies, integration ranges can differ slightly, but this does not
significantly affect determined fractions due to the intensity of desorption
events (note the logarithmic scale in Fig. 1).

Various studies provide relevant TPD data, but without determining desorption
or trapping fractions. In these cases, fractions can be retrieved by analysing
digitised TPD traces, see for example Rubin et al. (2023), where fractional
desorption is determined for TPD data from Kouchi & Yamamoto (1995) and
Gudipati et al. (2023). For this study, we additionally analyse TPD data from
Collings et al. (2004) and Martín-Doménech et al. (2014). Integration range
are set from $T_{\rm start}$ – 60 K for pure CO desorption, 60 – 130 K for
pure CO2 desorption and CO co-desorption, and 130 – 200 K for CO and CO2
release associated with H2O sublimation, where $T_{\rm start}$ is the lowest
temperature presented in the TPD trace.

Some nuances are important to be aware of when interpreting desorption data of
ice films consisting of three or more components. The desorption behaviour of
binary ice mixtures is fairly easy to understand. Usually, a volatile species
(e.g. CO) is embedded in a more abundant and less volatile host medium (e.g.
CO2). In these cases, the release of the volatile occurs at its pure-phase
sublimation temperature and when the host medium sublimates (for example, see
Simon et al. 2019, for experiments of CO trapping in CO2). Trends in trapping
fractions are easy to assess based on mixing ratios and ice film thicknesses.
However, when a third component is added such as H2O, the trapping behaviour
of CO in CO2 is changed, because CO2 itself interacts with and is trapped in
H2O (see e.g. Kipfer et al. 2024a). This complicates plotting trends of the CO
co-release with CO2, because the pure-phase CO2 thickness and mixing ratios
are difficult to determine and affected by the additional components in the
ice. Therefore, the total ice thickness and mixing ratios are used to plot
trends in trapping behaviour in ice mixtures consisting of three or more
components, but this is not an accurate representation of the ice composition.

Table 2: Volatile trapping experimental results

Molecule |  $F_{\rm H_{2}O}$(X) | Thickness (ML) | H2O:X | Reference
---|---|---|---|---
CO | [0.41 – 0.64] | 1000 | [21 – 125] | (Kipfer et al. 2024a)
CO | 0.23 | 1000 | 6 |  Gudipati et al. (2023), Rubin et al. (2023)
CO | [0.27 – 0.61] | [14 – 69] | [3 – 10] | Simon et al. (2023)
CO | [0.36 – 0.61] | [309 – 1396] | [6.7–11.5] |  Martín-Doménech et al. (2014)a
CO | [0.04 – 0.96] | [14 – 30] | [1 – 20] | Fayolle et al. (2011)
CO | 0.21 | 105 | 20 |  Collings et al. (2004)a
CO | 0.07 |  $\geq$1000 | 4.3 |  Kouchi & Yamamoto (1995), Rubin et al. (2023)
CO2 | 0.55 | 1000 | 3 |  Gudipati et al. (2023), Rubin et al. (2023)
CO2 | [0.76 – 0.94] | [750 – 1396] | [5 – 10] |  Martín-Doménech et al. (2014)a
CO2 | [0.32 – 0.92] | [14 – 30] | [3 – 20] | Fayolle et al. (2011)
CO2 | 0.34 | 105 | 20 |  Collings et al. (2004)a
CO2 | 0.20 |  $\geq$1000 | 6.5 |  Kouchi & Yamamoto (1995), Rubin et al. (2023)
Molecule |  $F_{\rm CO_{2}}$(CO) | Thickness (ML) | CO2:CO | Reference
CO | [0.11 – 0.34] | 1000 | [2 – 28] | Kipfer et al. (2024a)
CO | 0.07 |  $\geq$1000 | 0.67 |  Gudipati et al. (2023), Rubin et al. (2023)
CO | [0.33 – 0.67] | [13 – 72] | [3 – 13] | Simon et al. (2023)
CO | [0.14 – 0.60] | [6 – 53] | [1 – 9] | Simon et al. (2019)
CO | 0.17 | 1000 | 2 |  Kouchi & Yamamoto (1995), Rubin et al. (2023)

222$F_{\rm H_{2}O}$(x) indicates the fraction of CO or CO2 trapped in water.
$F_{\rm CO_{2}}$(CO) indicates the fraction of CO trapped in CO2. The
thickness is indicated in monolayers (ML), where 1 ML = 1015 molecules cm-2.
aTrapping fractions determined in this work.

##  Appendix B Additional plots

Additional model results where not only volatile trapping in H2O is included,
but also CO trapping in CO2 are shown in Figs. 6 and 7. A reduction in C/O
variations is again observed, but the release of a fraction of the CO with CO2
causes a more pronounced drop in solid-state C/O ratio around the CO2
snowline.

Figure 6: Same as Figure 2, but including CO trapping in CO2. We note that for
the model $F_{\rm H_{2}O}$(CO,CO2) = 0.8, $F_{CO_{2}}$(CO) = 0.2 there is no
gas phase C/O ratios outside the CO2 snowline because CO does not release
before the CO2 co-desorption event.  Figure 7: Same as Figure 2, but different
trapping ratios of CO and CO2 in H2O and CO trapping in CO2.


#  Evaluating NLG systems
A brief introduction

Emiel van Miltenburg
Tilburg University

###### Summary

This year the International Conference on Natural Language Generation (INLG)
will feature an award for the paper with the best evaluation. The purpose of
this award is to provide an incentive for NLG researchers to pay more
attention to the way they assess the output of their systems. This essay
provides a short introduction to evaluation in NLG, explaining key terms and
distinctions.

## How can I evaluate my system?

It is hard to say in general how you should evaluate your NLG system. Much
depends on the kind of system that you are developing, and the context in
which it is being used. A first step is to get used to commonly used
terminology in the field, so that you know what possibilities are out there.

Intrinsic evaluation

refers to the assessment of system output in isolation. For example,
grammaticality is a property that you can assess using intrinsic evaluation.
You could use either _human evaluation_ (e.g. grammaticality judgments) or
_automatic metrics_ (e.g. a precision grammar or a grammar checker) to
determine whether the output of an NLG system is grammatical. (For an overview
of different properties that you could evaluate, see Belz et al. 2020.)
Finally, you could carry out an _error analysis_ to determine where the system
still falls short.

Extrinsic evaluation

refers to the impact that a system may have on downstream processes. If you
have a newly developed NLG system, you could for example see whether employees
become more efficient or more productive after the system has been deployed.

Sometimes people also use the term “extrinsic evaluation” to assess the impact
of a specific module in an NLG pipeline. An _intrinsic_ evaluation of an NLG
pipeline module would just look at the quality of its output, rather than the
final text that is produced by the full pipeline. You could carry out an
_extrinsic_ evaluation by determining the extent to which the final output
improves when you replace an existing module (e.g. for rule-based referring
expression generation) with a newly developed module.

If you are developing multiple new modules, you could carry out _ablation
tests_ (systematically leaving out or replacing different modules) to see how
much each new module contributes to the system’s overall performance.

## Human versus automatic evaluation

Human evaluation is generally seen as the gold standard in NLG research,
because in the end it is essential that human readers appreciate the output of
your system. Having that said, there is great value in reliable automatic
metrics, since they are cheaper and not as labor-intensive. Reliability is the
keyword here: no matter how you measure different properties of the generated
text, we have to be able to trust the conclusions that you draw from your
observations.

## What are current best practices in the field?

It is always risky to talk about best practices, because evaluation is so
context-dependent. As the saying goes: “a foolish consistency is the hobgoblin
of little minds” (Emerson, 1982). For some projects, it may be better to
deviate from existing standards. Having said that, here are ten steps you may
find useful in planning your evaluation. Some of these steps raise fundamental
questions about your research project, so it is important to start thinking
about evaluation at the start of your project, and to not consider evaluation
as an afterthought. Moreover, good evaluations take time, which means you need
to schedule enough time to carry out a reliable evaluation.

  1. 1.

_Determine the target audience._ Who are you developing your NLG system for?
Ideally you will evaluate the performance of your system with a group of
participants that matches the demographic properties of your target audience.
Even if you don’t carry out a human evaluation, it is still important to
understand the application context because of the next point.

  2. 2.

_Get to know user needs_. If you have an applied NLG project, you should
ideally start from an understanding of the stakeholders’ needs. That means
talking to the people who will use your system, getting to know what they want
to do with your system and what properties are important for them. Then you
can develop an evaluation protocol that is in line with user needs.

  3. 3.

_Identify relevant work._ Search for relevant literature, identifying models
you want to compare to and metrics that are commonly used. Determine if those
metrics make sense given your project. Consider reproducing relevant results,
to be able to carry out any comparisons yourself.

  4. 4.

_Determine your goals and expectations_. Based on the earlier steps, formulate
a relevant research question. Think about possible outcomes of your project,
which outcome is more likely, and why you expect this to be the case. Also
consider how different outcomes of your evaluation should be interpreted. Try
to be as precise as possible. Does it make sense to form hypotheses about your
experiments, and to motivate them on the basis of earlier literature? This
helps with theory building in NLG. As a part of this process, you should:

     * •

_Identify key independent variables._ There are many factors that could
influence the characteristics of the output text. Different sets of inputs
lead to different kinds of outputs, and different system properties affect the
way the output text looks. Your job is to identify the main variables of
interest.

     * •

_Determine key dependent variables_. There are many different properties of
NLG output that you could assess, for example: grammaticality, fluency,
completeness, naturalness, appropriateness, and the list goes on. You don’t
have space to cover them all (although you could provide an additional
extensive evaluation in the appendix), and some properties are probably more
relevant than others for the purpose of your project. Clearly define the
constructs of interest before you start thinking about how to operationalise
those constructs. Use those definitions to critically assess the metrics you
are planning to use.

  5. 5.

_Check the validity of your set-up._ Having identified all relevant variables,
you can operationalise the different dependent variable through different
kinds of metrics. Here it is important to ensure the validity of your metrics.
In other words: do your metrics measure what they are supposed to measure? One
way to ensure the validity of a metric is to study the correlation between
that metric and a trusted reference, such as human judgments. (Ehud Reiter
(2018) investigated this in detail for the BLEU metric.) Alternatively you
could think of theoretical arguments to motivate why your metric provides a
good approximation of the variable of interest.

  6. 6.

_Select a sensible subset for evaluation._ If you cannot evaluate all the
output of your model (and possibly the models that you are comparing your work
to), think about the way you are sampling the outputs-to-be-evaluated. The
sampling procedure heavily impacts the validity of your evaluation and the
generalisability of your results.

  7. 7.

_Get IRB approval (if appropriate for your study)._ When you know what the
evaluation will look like, you can apply for approval with your local
institutional review board (IRB, also known as ‘ethics committee’) to
determine that your study follows current ethics guidelines. If you do this at
the onset of your project, you are less likely to run into any procedural
delays. With a research proposal for the IRB in hand, you may also decide to
turn the proposal into a full preregistration of your study (see van
Miltenburg et al. 2021b).

  8. 8.

_Keep a log._ Carry out your study and note any deviations from your original
plans, including the reasons why you changed your mind. These insights are
essential to provide the rationale behind your study design. If you do not
write this information down, you will forget it. For example, you could create
a private GitHub repository to hold all your code, data, and notes. (Once your
project is done, you can publish the repository alongside your paper. There
are also services to create a link to an anonymised version of your
repository, that you can include in your submitted paper.) Or you could manage
your project through the Open Science Foundation (OSF).

  9. 9.

_Be explicit about your materials and methods._ Report all relevant
information about how you carried out your evaluation, so that others could
reproduce your work using only your paper. If you carry out a human
evaluation, the Human Evaluation DataSheet (Shimorina and Belz, 2022) provides
an overview of important details to record.

  10. 10.

_Describe all relevant results_. If you are reporting overall scores, consider
providing a table with disaggregated results for different subsets of the
input data. Next to overview tables, you may also want to create insightful
visualisations of the results. (Though choose wisely; don’t just duplicate
your results in another modality.) Go beyond the “higher is better” narrative
and explain what the results mean for your system and the NLG literature in
general. Be open about the limitations of your evaluation and the challenges
that still lie ahead.

Bonus tip: _archive all data associated with your study._ This includes
including all outputs for the validation and test sets, crowdsourcing
templates, aggregated and non-aggregated human ratings, outputs of statistical
analysis software). Small files might be included in your GitHub repository,
but otherwise data can be hosted through other services, e.g. OSF, Zenodo,
Figshare, organisational repository or national science hosting provider. It
is possible that not all data can be made available at submission time (though
it is often possible to share data anonymously), but at least try to be as
exhaustive as possible for your camera-ready version.

## More to explore

For more in-depth reading, here are some useful references:

  * •

Celikyilmaz et al. (2020) provide a survey of NLG evaluation methods.

  * •

Gehrmann et al. (2022) provide general recommendations regarding both human
and automatic evaluations.

  * •

van der Lee et al. (2021) provide recommendations for human evaluation
studies.

  * •

van Miltenburg et al. (2021a) provide guidelines for error analysis.

  * •

Sai et al. (2022) provide an overview of automatic metrics used for NLG
evaluation.

  * •

Ehud Reiter’s blog has more recommendations for evaluating NLG systems.

If you have any questions, the community discord channel is a great place to
ask questions. Contact Dave Howcroft<EMAIL_ADDRESS>for an
invitation.

## Acknowledgments

_Thanks to Simone Balloccu, Ondřej Dušek, Dave Howcroft, Maria Keet, Ember
Manning, Maja Popović, Craig Thomson, and Sina Zarrieß for feedback on an
earlier draft of this essay._

## References

  * Belz et al. (2020) Anya Belz, Simon Mille, and David M. Howcroft. 2020.  Disentangling the properties of human evaluation methods: A classification system to support comparability, meta-evaluation and reproducibility testing.  In _Proceedings of the 13th International Conference on Natural Language Generation_ , pages 183–194, Dublin, Ireland. Association for Computational Linguistics.
  * Celikyilmaz et al. (2020) Asli Celikyilmaz, Elizabeth Clark, and Jianfeng Gao. 2020.  Evaluation of text generation: A survey.  _CoRR_ , abs/2006.14799.
  * Emerson (1982) Ralph Waldo Emerson. 1982.  _Selected Essays_.  American Library. Penguin Books, Harlow, England.
  * Gehrmann et al. (2022) Sebastian Gehrmann, Elizabeth Clark, and Thibault Sellam. 2022.  Repairing the cracked foundation: A survey of obstacles in evaluation practices for generated text.
  * van der Lee et al. (2021) Chris van der Lee, Albert Gatt, Emiel van Miltenburg, and Emiel Krahmer. 2021\.  Human evaluation of automatically generated text: Current trends and best practice guidelines.  _Computer Speech & Language_, 67:101151.
  * van Miltenburg et al. (2021a) Emiel van Miltenburg, Miruna Clinciu, Ondřej Dušek, Dimitra Gkatzia, Stephanie Inglis, Leo Leppänen, Saad Mahamood, Emma Manning, Stephanie Schoch, Craig Thomson, and Luou Wen. 2021a.  Underreporting of errors in NLG output, and what to do about it.  In _Proceedings of the 14th International Conference on Natural Language Generation_ , pages 140–153, Aberdeen, Scotland, UK. Association for Computational Linguistics.
  * van Miltenburg et al. (2021b) Emiel van Miltenburg, Chris van der Lee, and Emiel Krahmer. 2021b.  Preregistering NLP research.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 613–623, Online. Association for Computational Linguistics.
  * Reiter (2018) Ehud Reiter. 2018.  A structured review of the validity of BLEU.  _Computational Linguistics_ , 44(3):393–401.
  * Sai et al. (2022) Ananya B. Sai, Akash Kumar Mohankumar, and Mitesh M. Khapra. 2022.  A survey of evaluation metrics used for nlg systems.  _ACM Comput. Surv._ , 55(2).
  * Shimorina and Belz (2022) Anastasia Shimorina and Anya Belz. 2022.  The human evaluation datasheet: A template for recording details of human evaluation experiments in NLP.  In _Proceedings of the 2nd Workshop on Human Evaluation of NLP Systems (HumEval)_ , pages 54–75, Dublin, Ireland. Association for Computational Linguistics.


# A Light Sliding-Window Part-of-Speech Tagger for the Apertium Free/Open-
Source Machine Translation Platform

###### Abstract

This paper describes a free/open-source implementation of the light sliding-
window (LSW) part-of-speech tagger for the Apertium free/open-source machine
translation platform. Firstly, the mechanism and training process of the
tagger are reviewed, and a new method for incorporating linguistic rules is
proposed. Secondly, experiments are conducted to compare the performances of
the tagger under different window settings, with or without Apertium-style
“forbid” rules, with or without Constraint Grammar, and also with respect to
the traditional HMM tagger in Apertium.
Keywords: part-of-speech tagging, light sliding-window, machine translation,
free/open-source

A Light Sliding-Window Part-of-Speech Tagger for the Apertium Free/Open-Source
Machine Translation Platform

Gang Chen, Mikel L. Forcada
---
Key Laboratory of Computational Linguistics (Peking University), Ministry of
Education, China
Departament de Llenguatges i Sistemes Informàtics, Universitat d’Alacant,
Spain
<EMAIL_ADDRESS><EMAIL_ADDRESS>

Abstract content

##  1\. Introduction

Apertium111The Apertium machine translation engine, linguistic data for
various language pairs, and documentation can be downloaded from
http://www.apertium.org. is a shallow-transfer rule-based free/open-source
machine translation platform. This paper reports a free/open-source
implementation of the light sliding window (LSW) PoS tagger [Sánchez-Villamil
et al., 2005], and compares its performance with that of the original first-
order HMM tagger in Apertium [Tyers et al., 2010, Sheikh and Sánchez-Martínez,
2009, Cutting et al., 1992]. Section 2 reviews the mechanism of the LSW tagger
and proposes a method to improve its tagging accuracy by incorporating
linguistic rules, Section 3 shows the experimental results and discusses them,
and finally, in Section 4, the paper ends with some conclusions and future
plans.

##  2\. Methods

The main difference between the LSW and HMM PoS taggers is that the LSW PoS
tagger makes local decisions about the PoS tag of each word which are based on
the ambiguity class (set of PoS tags) of words in a fixed-length context
around the problem word, while HMM makes this decision by efficiently
considering all possible disambiguations of all words in the sentence, by
using a probabilistic model based on a multiplicative chain of transition and
emission probabilities. In terms of model complexity, LSW is simpler than HMM,
while, on the other hand, the number of parameters of LSW could be larger than
that of HMM, which may have a crucial influence on the tagging performance as
training material may not be sufficient to estimate them adequately.

The LSW tagger is an improved version of the sliding window (SW) PoS tagger
[Sánchez-Villamil et al., 2004], and the main goal of the LSW tagger is to
reduce the parameters of a SW tagger, by using approximations for the
parameter estimation, without a significant loss in accuracy. Therefore, we
briefly describe the SW tagger first, and then the LSW tagger.

###  2.1. The SW tagger

####  2.1.1. Overview

Let $\Gamma=\\{\gamma_{1},\gamma_{2},\dots,\gamma_{|\Gamma|}\\}$ be the tag
set, and $W=\\{w_{1},w_{2},\dots\\}$ be the words to be tagged. A partition of
$W$ is established so that $w_{i}\equiv w_{j}$ if and only if both are
assigned the same subset of tags, where each class of the partition is called
an ambiguity class. Let
$\Sigma=\\{\sigma_{1},\sigma_{2},\dots,\sigma_{|\Sigma|}\\}$ be the collection
of ambiguity classes, where each $\sigma_{i}$ is an ambiguity class. Let
$T:\Sigma\rightarrow 2^{\Gamma}$ be the function returning the collection
$T(\sigma)$ of PoS tags for an ambiguity class $\sigma$.

The PoS tagging problem may be formulated as follows: given a text
$w[1]w[2]\dots w[L]\in W^{+}$, each word $w[t]$ is assigned (using a lexicon
and a morphological analyzer) an ambiguity class $\sigma[t]\in\Sigma$ to
obtain the ambiguously tagged text
$\sigma[1]\sigma[2]\dots\sigma[t]\in\Sigma^{+}$; the task of a PoS tagger is
to obtain a tag sequence $\gamma[1]\gamma[2]\dots\gamma[t]\in\Gamma^{+}$ as
correct as possible, that is, the one that maximizes the probability of that
tag sequence given the word sequence:

$\gamma^{*}[1]\dots\gamma^{*}[L]=\operatornamewithlimits{argmax}_{\gamma[t]\in
T(\gamma[t])}P(\gamma[1]\dots\gamma[L]|\sigma[1]\dots\sigma[L])$ (1)

The core idea of SW PoS tagging is to use the ambiguity classes of neighboring
words to approximate the dependencies locally:

$P(\gamma[1]\dots\gamma[L]|\sigma[1]\dots\sigma[L])=\prod_{t=1}^{t=L}p(\gamma[t]|C_{(-)}\sigma[t]C_{(+)})$
(2)

where $t=1\dots L$, $C_{(-)}$ is the left context of length $N_{(-)}$ (e.g. if
$N_{(-)}=1$, then $C_{(-)}=\gamma[t-1])$, and $C_{(+)}$ is the left context of
length $N_{(+)}$.

####  2.1.2. Unsupervised parameter estimation

Let $p(\gamma|C_{(-)}\sigma C_{(+)})$ be the probability of a tag $\gamma$
appearing between the context $C_{(-)}$ and $C_{(+)}$. The most probable tag
$\gamma^{*}[t]$ is selected as the one with the highest probability by the
formula:

$\gamma^{*}[t]=\operatornamewithlimits{argmax}_{\gamma\in
T(\sigma[t])}p(\gamma|C_{(-)}\sigma C_{+)})$ (3)

Estimating the parameters from a tagged corpus would be straightforward, but
estimating from an untagged corpus requires an iterative process. Let
${\tilde{n}}_{C_{(-)}\gamma C_{(+)}}$ (a simpler and interchangeable
representation for $p(\gamma|C_{(-)}\sigma C_{(+)})$ ) be the effective number
of times (count) that $\gamma$ appears between the context $C_{(-)}$ and
$C_{(+)}$. Following the steps in [Sánchez-Villamil et al., 2004], we can
estimate ${\tilde{n}}_{C_{(-)}\gamma C_{(+)}}$ iteratively by:

$\begin{split}&{\tilde{n}}_{C_{(-)}\gamma C_{(+)}}^{[k]}=\\\
&{\tilde{n}}_{C_{(-)}\gamma C_{(+)}}^{[k-1]}\sum_{\sigma:\gamma\in
T(\sigma)}n_{C_{(-)}\sigma C_{(+)}}{\left(\sum_{\gamma^{\prime}\in
T(\sigma)}{\tilde{n}}_{C_{(-)}\gamma^{\prime}C_{(+)}}^{[k-1]}\right)}^{-1}\end{split}$
(4)

A recommended initial value could be obtained by assuming that all the tags
$\gamma$ in $\sigma$ are equally probable.

###  2.2. The LSW tagger

####  2.2.1. Overview

The SW tagger tags a word by looking at the ambiguity classes of neighboring
words, and has therefore a number of parameters in
$O({|\Sigma|}^{N_{(-)}+N_{(+)}}|\Gamma|)$. The LSW tagger [Sánchez-Villamil et
al., 2005] tags a word by looking at the possible tags of neighboring words,
and therefore it has a number of parameters in
$O({|\Gamma|}^{N_{(-)}+N_{(+)}+1})$. Usually the tag set size $|\Gamma|$ is
significantly smaller than the combinational ambiguity class size $|\Sigma|$.
In this way, the number parameters is effectively reduced.

The LSW approximates the best tag as follows:

$\begin{split}&\gamma^{*}=\operatornamewithlimits{argmax}_{\gamma\in
T(\sigma[t])}\\\ &\sum_{E_{(-)}\in T^{\prime}(C_{(-)}[t])\atop E_{(+)}\in
T^{\prime}(C_{(+)}[t])}p(E_{(-)}\gamma
E_{(+)}|C_{(-)}[t]\gamma[t]C_{(+)}[t])\end{split}$ (5)

where $T^{\prime}:\Sigma^{*}\rightarrow 2^{\Gamma^{*}}$, an extension of $T$,
returns the set of tag sequences for an ambiguity sequence; $E_{(-)}$ and
$E_{(+)}$ are the left and right tag sequence respectively.

####  2.2.2. Unsupervised parameter estimation

Following a procedure similar to that for the SW tagger, we can derive an
iterative process to train the LSW tagger.

$\begin{split}&{\tilde{n}}_{E_{(-)}\gamma
E_{(+)}}^{[k]}={\tilde{n}}_{E_{(-)}\gamma
E_{(+)}}^{[k-1]}\sum_{\sigma:\gamma\in T(\sigma)\atop{C_{(-)}:E_{(-)}\in
T^{\prime}(C_{(-)})\atop C_{(+)}:E_{(+)}\in T^{\prime}(C_{(+)})}}\\\
&n_{E_{(-)}\sigma E_{(+)}}{\left(\sum_{\gamma^{\prime}\in
T(\sigma)\atop{C_{(-)}:E_{(-)}\in T^{\prime}(C_{(-)})\atop C_{(+)}:E_{(+)}\in
T^{\prime}(C_{(+)})}}{\tilde{n}}_{E_{(-)}\gamma^{\prime}E_{(+)}}^{[k-1]}\right)}^{-1}\end{split}$
(6)

where ${\tilde{n}}_{E_{(-)}\gamma E_{(+)}}$ is the effective number of times
(count) that $\gamma$ appears between the context of tags $E_{(-)}$ and
$E_{(+)}$.

Similarly to the initialization step in the SW tagger, a recommended initial
value can be obtained by assuming that all the tag sequences $E_{(-)}\gamma
E_{(+)}$ in the window $C_{(-)}\sigma C_{(+)}$ are equally probable.

###  2.3. LSW with forbid and enforce rules

There are forbid and enforce rules for sequences of two PoS tags in the
current implementation of the Apertium PoS tagger. They were successfully
applied in the original HMM tagger in Apertium, with a significant improvement
in accuracy [Sheikh and Sánchez-Martínez, 2009], simply by making the
corresponding transition probabilities equal to zero. The SW tagger could not
make use of forbid and enforce rules because of the fact that it works with
ambiguity classes, while on the other hand, the LSW tagger can easily
incorporate them as it works directly with PoS tags

The rules can be introduced right after the initialization step. For a tag
sequence in the parameter space, if any consecutive two tags match a forbid
rule or fail to match an enforce rule, the underlying parameter
${\tilde{n}}_{E_{(-)}\gamma E_{(+)}}$ will be given a starting value of zero.

In this way, for an LSW tagger with rules, the initial value could be given as
follows,

${\tilde{n}}_{E_{(-)}\gamma E_{(+)}}^{[0]}=\begin{cases}0&\text{if
$E_{(-)}\gamma E_{(+)}$ is not valid,}\\\ \Lambda&\text{otherwise}\end{cases}$
(7)

where

$\Lambda=\sum_{\sigma:\gamma\in T(\sigma)\atop{C_{(-)}:E_{(-)}\in
T^{\prime}(C_{(-)})\atop C_{(+)}:E_{(+)}\in
T^{\prime}(C_{(+)})}}n_{E_{(-)}\sigma
E_{(+)}}\frac{1}{|V^{\prime}(C_{(-)}\sigma C_{(+)})|}$ (8)

where, the validity of $E_{(-)}\gamma E_{(+)}$ is determined by forbid and
enforce rules, and the function $V^{\prime}$ returns the collection of valid
(enforced or not forbidden) tag sequences contained in the ambiguity class
sequence $C_{(-)}\sigma C_{(+)}$.

##  3\. Experiments

###  3.1. Training data and test set

The experiments are conducted on three languages: Spanish (apertium-en-
es-0.8.0), Catalan (apertium-es-ca-1.1.0), and English (apertium-en-es-0.8.0).
We obtain the training data for Spanish and English by sampling text from the
Europarl corpus [Koehn, 2005], and for Catalan by sampling text from the
Catalan Wikipedia. The statistics on the training data and test data are shown
in Table 1. Test data for Catalan and Spanish come from apertium-es-ca-1.1.0.
It is worth noting that the English test set has been built by mapping the
results form the TnT [Brants, 2000] tagger as an approximation.

Items | Spanish | Catalan | English
---|---|---|---
Words (train) | 3 million | 4 million | 3 million
Amb. classes (train) | 106 | 92 | 68
Words (test) | 25, 000 | 25, 000 | 30, 000
Amb. rate (test) | 22.81% | 31.13% | 29.97%
Forbid rules | 545 | 272 | 117
Enforce rules | 15 | 25 | 41
Table 1: Major statistics for the training and test data.

###  3.2. The LSW tagger vs. the SW tagger

We firstly study whether there is a difference between the LSW tagger and the
SW tagger, keeping all other settings the same. Then we study whether rules
can help improve the accuracy for the LSW tagger. “Accuracy” in the graph
refers to the tagging precision of a tagger on the hand-tagged test set.
Figure 1 shows that rules help significantly for improving accuracy, and that
the SW tagger behaves similarly to the LSW tagger without rules, which is
consistent with the conclusion in [Sánchez-Villamil et al., 2005].

Figure 1: Performance evaluation for (1) the LSW(-1, +1) tagger, (2) the
LSW(-1, +1) tagger without rules, denoted as LSW(-1, +1)-No-Rules, and (3) the
SW(-1, +1) tagger, all on Spanish, Catalan, and English.

###  3.3. Different window settings for the LSW tagger

We study the performances of the LSW tagger with different window settings,
and of the HMM tagger, on the three languages, as shown in Figure 2. We can
see that the HMM tagger performs best among all the taggers, especially when
there is enough training data. However, when training data is limited, the LSW
taggers learn faster (need less words to learn) and more stably than the HMM
tagger.

Among all the LSW taggers, the LSW(-1, +1), i.e. left context 1 and right
context 1, performs best. When there are enough training data, the
performances of the HMM tagger and the LSW(-1, +1) tagger are quite close.

Note that under some window settings, the performances of the LSW taggers even
decrease as more training lines were added, e.g. LSW(-1) and LSW(-2, -1) for
Spanish and Catalan. This is an unexpected phenomenon, and the reason for it
would require further investigation.

Figure 2: Different window settings and their performance, tested on Spanish,
Catalan, and English.

###  3.4. Using Constraint Grammar rules to support the HMM and LSW

We also tested whether the use of Constraint Grammar (CG) rules helps to
improve the accuracy obtained by both HMM and LSW taggers, along the lines
suggested in [Hulden and Francom, 2012]. For that, we used the CG rules
already present in Apertium packages apertium-eo-es-0.8.2 for Spanish and
apertium-eo-ca-0.8.2 for Catalan respectively (a CG module is integrated in
many Apertium language pairs). Figure 3 shows that CG helps almost in all
settings. It is also shown that CG rules help the two taggers in different
situations: for the HMM tagger, the positive contribution of CG rules is
larger when training data is limited than when training data is relatively
enough; while for the LSW tagger, the trend is almost the opposite, that CG
rules contribute even more when training data is relatively enough. Note that
the logical approach would be to use CG rules both for reducing ambiguity for
the training corpus (denoted as cgTrain in Figure 3) and for reducing
ambiguity right after morphological analyzer and before the PoS tagger
(denoted as cgTag in Figure 3); the results are however almost
indistinguishable from those obtained applying CG in either step.

Figure 3: Performance evaluation for HMM and LSW with and without CG.

##  4\. Discussion and future work

We reviewed the mechanism and unsupervised parameter estimation methods for
both the SW and LSW taggers. Compared with previous work [Sánchez-Villamil et
al., 2004, Sánchez-Villamil et al., 2005], firstly, we proposed a method for
incorporating the forbid and enforce rules already used for HMM taggers in
Apertium into the LSW tagger; and secondly, the implementation is the first
time that the LSW tagger is integrated into a real machine translation system
(Apertium), and at the same time, its code is free/open-source.

We also conducted experiments to compare the performances of the LSW tagger
with different settings, and with respect to the original HMM tagger. Firstly,
the HMM tagger performs slightly better than the LSW(-1, +1) tagger when there
is enough training data, while the LSW(-1, +1) tagger learns faster and is
more stable when training data is limited. Secondly, the LSW(-1, +1) tagger
performs best among all the other window settings, and better than the SW(-1,
+1) tagger, which behaves similarly with LSW(-1, +1)-No-Rules. Thirdly, we
have found that the use of CG rule sets already existing in some Apertium
taggers helps significantly to improve accuracy based both on the HMM and LSW
taggers, and that for the HMM tagger CG rules help more when training data is
limited, while for the LSW tagger CG rules help more when training data is
relatively enough.

The reason why the performance of the LSW tagger under some window settings
worsens as more training lines are added also requires more efforts to study.
Source code is available through the Apertium Subversion
repository222https://svn.code.sf.net/p/apertium/svn/branches/apertium-swpost
under a free/open-source license.

##### Acknowledgements:

Support from Google Summer of Code (summer scholarship for Gang Chen) and from
the Spanish Ministry of Economy and Competitiveness through grant
TIN2012-32615 are gratefully acknowledged. The authors also thank Francis M.
Tyers and Jim O’Regan for useful comments.

## References

  * Brants, 2000 T. Brants.  2000\.  TnT: a statistical part-of-speech tagger.  In Proceedings of the sixth conference on Applied natural language processing, pages 224–231.
  * Cutting et al., 1992 D. Cutting, J. Kupiec, J. Pedersen, and P. Sibun.  1992\.  A practical part-of-speech tagger.  In Proceedings of the third conference on Applied natural language processing, pages 133–140.
  * Hulden and Francom, 2012 M. Hulden and J. Francom.  2012\.  Boosting statistical tagger accuracy with simple rule-based grammars.  In N. Calzolari, K. Choukri, T. Declerck, M. Ugur Dogan, B. Maegaard, J. Mariani, J. Odijk, and S. Piperidis, editors, LREC, pages 2114–2117. European Language Resources Association (ELRA).
  * Koehn, 2005 P. Koehn.  2005\.  Europarl: A Parallel Corpus for Statistical Machine Translation.  In Conference Proceedings: the tenth Machine Translation Summit, pages 79–86, Phuket, Thailand. AAMT, AAMT.
  * Sánchez-Villamil et al., 2004 E. Sánchez-Villamil, M. L. Forcada, and R. C. Carrasco.  2004\.  Unsupervised training of a finite-state sliding-window part-of-speech tagger.  In Advances in Natural Language Processing, pages 454–463. Springer.
  * Sánchez-Villamil et al., 2005 E. Sánchez-Villamil, M. L. Forcada, and R. C. Carrasco.  2005\.  Parameter reduction in unsupervisedly trained sliding-window part-of-speech taggers.  In Proceedings of Recent Advances in Natural Language Processing, Borovets, Bulgaria, September, 2005.
  * Sheikh and Sánchez-Martínez, 2009 Z. M. A. W. Sheikh and F. Sánchez-Martínez.  2009\.  Parameter reduction in unsupervisedly trained sliding-window part-of-speech taggers.  In Proceedings of the First International Workshop on Free/Open-Source Rule-Based Machine Translation, pages 67–74. Universidad de Alicante. Departamento de Lenguajes y Sistemas Informáticos.
  * Tyers et al., 2010 F. M. Tyers, F. Sánchez-Martínez, S. Ortiz-Rojas, and M. L. Forcada.  2010\.  Free/open-source resources in the Apertium platform for machine translation research and development.  The Prague Bulletin of Mathematical Linguistics, 93:67–76.