HeidelTime uses TIMEX3 tags, which are based on TimeML (Pustejovsky et al. 2010), the most widely used annotation standard for temporal expressions. In the following, we briefly explain the *value* attribute of TIMEX3 annotations of Duration and Set expressions, as we do consider their normalized values for a deeper analysis of the occurring temporal expressions. The value attribute of Duration and Set expressions contains information about the length of the duration that is mentioned, starting with P (or PT in case of time level durations) followed by a number and an abbreviation of the granularity (e.g., years: Y, month: M, week: W, days: D; hours: H, minutes: M). In addition, fuzzy expressions are referred to by X instead of precise numbers, e.g., *several weeks* is normalized to PXW, *monthly* is normalized to XXXX-XX and *annually* to XXXX.

#### **4.3 Extraction Quality**

For meaningful analysis and substantiated conclusions of temporal expressions in our diachronic corpus, the extraction (and normalization) quality of the temporal tagger should be reliable. Although HeidelTime has been extensively evaluated before on a variety of corpora2, our corpus is quite different from standard temporal tagging corpora as it contains scientific documents from multiple scientific fields published across several centuries. Creation of a proper gold standard with manual annotations covering all scientific fields across all time periods would not be feasible in an appropriate time frame. Instead, for a valuable statement of

2 E.g., on news articles as in the TempEval competitions (Verhagen et al. 2010; UzZaman et al. 2013) and on Wikipedia articles contained in the WikiWars corpus (Mazur and Dale 2010).

temporal tagging quality on our corpus, determining the correctness of expressions tagged by the temporal tagger would be meaningful.

For this, we use precision, i.e., we randomly sample 250 instances for each time period, and manually validate whether the automatically annotated temporal expressions are correctly extracted.3 Here, we consider correctly extracted instances (right) and wrongly extracted instances (wrong). The latter are either cases of ambiguity (e.g., *spring* as 'season' or 'water spring' or *current* meaning 'now' or 'electric current') or wrongly assigned temporal expressions to numbers occurring in the text. On top we differentiate correctly assigned but not relevant instances (other) due to noise in the data itself. These are, e.g., temporal expressions assigned to reference sections (especially in the 1950–2000 periods) or used within tables (mostly in the earlier time periods).

Table 2 presents precision information and the number of instances per assigned category of right, other, and wrong. We consider the other instances to be correct in terms of precision of extraction. Across periods, precision achieves 0.89 to 0.96.

**Table 2.** Precision across time periods.

#### **5 Typicality of Temporal Expressions**

To obtain temporal expressions typical of a time period, we use relative entropy, also known as Kullback-Leibler Divergence (KLD) (Kullback and Leibler 1951) – a well-known measure of (dis)similarity between probability distributions used in NLP, speech processing, and information retrieval. In comparison to relative frequency, i.e., the unconditioned probability of, e.g., a word over all words in a corpus, relative entropy is based on conditioned probability.

In information-theoretic parlance, relative entropy measures the average number of additional bits per feature (here: temporal expressions) needed to

3 We chose to use an amount of instances per period rather than an amount of documents, due to possible sparsity of temporal expressions within documents. This also allows us to validate the same amount of instances across time periods, rather than varying amounts of instances across time periods.

encode a feature of a distribution *A* (e.g., the 1650 time period) by using an encoding optimized for a distribution *B* (e.g., the 1700 time period). The more additional bits needed, the more distant *A* and *B* are. This is formalized as:

$$D(A||B) = \sum\_{i} p(feature\_i|A) \log\_2 \frac{p(feature\_i|A)}{p(feature\_i|B)} \tag{1}$$

where *p*(*featurei|A*) is the probability of a feature (i.e., a temporal expression) in a time period *A*, and *p*(*featurei|B*) the probability of that feature in a time period *B*. The *log*2 *p*(*f eaturei*|*A*) *p*(*f eaturei*|*B*) relates to the difference between both probability distributions (*log*2*p*(*featurei|A*) *− log*2*p*(*featurei|B*)), giving the number of additional bits. These are then weighted with the probability of *p*(*featurei|A*) so that the sum over all *featurei* gives the *average* number of additional bits per feature, i.e., the *relative* entropy.

In terms of typicality, the more bits are used to encode a feature, the more typical that feature is for a given time period vs. another time period. Thus, in a comparison of two time periods (e.g., 1650 vs. 1700), the higher the KLD value of a feature for one time period (e.g., 1650), the more typical that feature is for that given time period. In addition, we test for significance of a feature by an unpaired Welch's t-test. Thus, features considered typical are distinctive according to KLD and show a p-value below a given threshold (e.g., 0.05).

To compare typical features across several time periods, the most high ranking features of each comparison are considered. For example, for 1650 we obtain six feature sets typical of 1650 as we have six comparisons of 1650 with each of the other six time periods (i.e., a feature set for features typical of 1650 vs. 1700, of 1650 vs. 1750, etc.). If features are shared across feature sets and are high ranking (e.g., in the top 5), these features are considered to be typical of 1650. In other words, these are features ranking high in terms of KLD, significant in terms of p-value, and typical of a time period across all/most comparisons with other time periods. As in our case we consider seven time periods, features are considered typical which rank high for one time period in 6 to 4 feature sets (i.e., typical in more than half of the comparisons).

#### **6 Analysis**

In the following, we analyze diachronic tendencies of temporal expressions from the period of 1650 to 2000 in terms of (1) relative frequency (i.e., unconditioned probabilities), and (2) typicality (i.e., conditioned probabilities of expressions in one vs. the other time periods as described in Sect. 5).

We show how the notion of typicality based on relative entropy leads to valuable insights on the change of temporal expressions in scientific writing w.r.t. more and less frequent expressions.

#### **6.1 Frequency-Based Diachronic Tendencies**

Comparing temporal types across fifty years time periods in terms of frequency (see Fig. 1 showing log of frequency per million (pM)), Date is the most

**Fig. 1.** Diachronic tendencies of temporal expression types in scientific writing.

frequent type, followed by Duration. Set and Time expressions are less frequent. In addition, while Date remains relatively stable over time, expressions of Duration, Set and Time drop quite a bit from 1850 onwards, getting relatively rare.

#### **6.2 Diachronic Tendencies of 'Typical' Temporal Expressions**

Inspecting diachronic change through the lens of relative entropy (as described in Sect. 5) allows us to consider temporal expressions typical of one time period when compared to the other time periods. We study each type of temporal expression and carefully select the base of comparison.

Date Considering Date expressions, instead of comparing single dates (which mostly occur only once in the corpus, such as *June, 3, 1769* ), we take a level of abstraction and consider part-of-speech (POS) sequences of annotated Date expressions to better inspect the types of changes that might have occurred over time. For each Date expression, we extract POS sequences and use relative entropy to detect typical POS sequences of temporal expressions for each time period.

Table 3 shows POS sequences typical of one time period vs. 6-4 other time periods (see column comp.)4. For example, for 1650 the POS sequences Determiner-Noun (DT-NN) and Proper Noun (NP) are quite typical, which are all temporal expression referring to seasons in terms of lexical realizations (see Example 1). Both POS sequences are typical of 1650 vs. 1750 to 2000 (i.e., 5 comparisons). If we consider the POS sequences that are typical across time periods and their lexical realizations, there seems to be a development in terms of specificity and interval (see Fig. 2).

4 Note that the examples in Table 3 show most frequent realizations for relatively generic expressions such as seasons (e.g., *in the Spring*) or examples taken randomly from the corpus for specific dates (e.g., *June 3, 1769* ) as they occur only once.

**Table 3.** Typical POS sequences of Date.

CD: cardinal number, DT: determiner, IN: preposition, JJ: adjective, NN: sing. common noun, NNS: pl. common noun, NP: sing. proper noun, RB: adverb

**Fig. 2.** Specificity (black) and interval (gray) of typical Date expressions.

To capture the notion of specificity, we consider how many pieces of temporal information are given by a POS sequence to make a temporal expression most specific, with a scale from 1 to 4, where 1 is least specific (e.g., NP denoting seasons such as *Winter* as we do not know of which year etc.) and 4 is most specific (e.g., NP CD, CD such as *June 3, 1769* which gives us an exact date)5. For comparison across time periods, Fig. 2 shows the average of the specificity count over all typical POS sequences of a time period (black line). For the interval of typical Date expressions, the amount of days6 the expressions refer to is used (shown in log in Fig. 2, gray line). The more specific an expression becomes, the smaller the interval it refers to and vice versa.

Figure 2 also shows how temporal expressions move from relatively unspecific (e.g., *in the Spring* in 1650) to very specific (*June 18, 1784* in 1800) and back to unspecific expressions (e.g., *the last decades* in 2000). The interval moves instead from a wider to a smaller span and back to a wider span in 2000. Investigating the contexts, in which these expressions arise, gives further insights. While in the early time periods, season mentioning is typical, from 1800 to 1850, temporal expressions are typical with exact date, year or month expressions. These expressions are used to present exact dates of observations made by a researcher at several points in time, especially in the field of astronomy (see Example 1). From the 1950 onwards, typical Date expressions become less explicit, relating to broader (e.g., *the 1970s* in Example 3) and less specific (e.g., *current literature* in Example 3) temporal reference. These expressions are used, e.g., in the context of previous work descriptions in introduction sections of research papers.

#### *Example 1*

*In Winter it will need longer infusion, than in the Spring or Autumn.* (1650) *The difference between these two plants is this; the papaver corniculatum dies to the root in the winter, and sprouts again from its root in the spring;* (1750)

#### *Example 2*

*March 4, 1783. With a 7-feet reflector, I viewed the nebula near the 5th Serpentis, discovered by Mr. MESSIER, in 1764.* (1750)

#### *Example 3*

*In the 1970s, Rabin [38] and Solovay and Strassen [44] developed fast probabilistic algorithms for testing primality and other problems.* (2000) *There is a significant confusion in the current literature on "cellular" or "tessellation arrays" concerning the concept of a "Garden-of-Eden configuration".* (1950)

Time To investigate typical Time expressions, similarly to Date expressions, we consider their POS sequences (see Table 4).

5 On this scale, 1 denotes the mere occurrence of a temporal expression (e.g., NP such as *Winter*), 4 denotes the mere expression plus the inclusion of day, month and year (e.g., NP CD, CD such as *June 3, 1769*, which is a temporal expression + day + month + year resulting in 4 points in total), and 2 and 3 denote an occurrence of a temporal expression plus either two or three combinations of day, month and year (e.g., CD such as *1858* for 2, i.e., temporal expression + year, and NP CD such as *March 8* for 3, i.e., temporal expression + month + day).

6 We use 'day' because Date expressions refer to a point in time of the granularity 'day' or coarser, i.e., 'day' is the smallest unit.

**Table 4.** Typical POS sequences of Time.

CD: cardinal number, DT: determiner, IN: preposition, JJ: adjective, NN: sing. common noun, NP: sing. proper noun

It can be seen that only the intermediate time periods show typical expressions (1750 to 1850). In terms of granularity, in the period of 1750, expressions are less granular pointing to broader sections of a day (e.g., *morning*, *evening*) mostly used to describe observations made (see Example 4). In the 1850 period, expressions point to specific hours of a day (e.g., *9 A.M.*) mostly in descriptions of experiments.

#### *Example 4*

*Monday morning she appeared well, her pulse was calm, and she had no particular pain.* (1750)

*There being usually but one assistant, it was impossible to observe during the whole twenty-four hours; the hours of observation selected were therefore from 3 A.M. to 9 P.M. inclusive.* (1850)

Duration For Duration we consider their TIMEX3 value, as it directly encodes normalized information on the duration length and granularity of temporal expressions. Figure 3 shows typical TIMEX3 values (e.g., P1D for expressions such as *one day*) of specific time periods7. The y-axis shows the duration length in seconds on a log scale. In general, duration length gets lower from 1750 to 1850 (with expressions of seconds and hours, which are more granular) and higher in 1950 and 2000 (with expressions of decades, which are less granular).

We then again consider the contextual environments of these typical expressions. In the earlier time periods (1650 and 1700), day and year expressions are typical, mostly relating to observations or experiment descriptions (see Example 5).

7 Note also that typical expressions can either be relatively explicit (e.g., P1D for *24 h*) or fuzzy (indicated by an X in the TIMEX3 value, e.g., *few hours* for the value PTXH).

**Fig. 3.** Diachronic tendencies of typical Duration expressions.

#### *Example 5*

*After the eleven Months, the Owner having a mind to try, how the Animal would do upon Italian Earth, it died three days after it had changed the Earth.* (1650) *[...] the Opium, being cut into very thin slices, [...] is to be put into, and well mixed with, the liquor, (first made luke-warm) and fermented with a moderate Heat for eight or ten Days, [...].* (1650)

From the period of 1750 to 1950, duration length is relatively low with expressions of seconds, minutes and hours being typical of these time periods. These expressions are mainly related to observations in the 1750 period and experiment descriptions from 1800 to 1950 (see Example 6).

#### *Example 6*

*June 4, the weather continued much the same, and about 9h 30 in the evening, we had a shock of an earthquake, which lasted about four seconds, and alarmed all the inhabitants of the island.* (1750)

*[...] the glass produced by this fusion was in about twelve hours dissolved, by boiling it in a proper quantity of muriatic acid.* (1800)

*In a few hours a mass of fawn-coloured crystals was deposited;* (1850)

*The patient is then switched to the re-breathing system containing 133 Xenon at 5 mCi/1 for a period of one minute, and then returned to room air for a period of ten minutes.* (1950)

In the 1950, besides weeks and minutes, related to experiment descriptions (see Example 7), expressions of decades are typical. The latter is also true for the 2000. In both periods, expressions relating to decades refer to previous work (see Example 7).

**Fig. 4.** Diachronic tendencies of typical Set expressions in scientific writing.

Thus, Duration shifts from being used for purposes of observational to experimental science and finally to previous work references in the latest time periods.

#### *Example 7*

*For each speaker, performance was observed across numerous repetitions of the vocabulary set within a single session, as well as across a 2-week time period.* (1950)

*It constitutes the usual drift-diffusion transport equation that has been successfully used in device modeling for the last two decades.* (1950)

*Provably correct and efficient algorithms for learning DNF from random examples would be a powerful tool for the design of learning systems, and over the past two decades many researchers have sought such algorithms.* (2000)

Set For Set expressions again their TIMEX3 value is considered. Figure 4 shows typical expressions with the times per year of a Set expression on the y-axis in log8, mirroring also less granular (*annually*) and more granular (*every day*) expressions. As we have seen from Fig. 1, Set expressions are relatively rare in scientific writing and strongly decrease over time (see Sect. 6.1). This is also reflected in the few temporal expressions typical of each time period in Fig. 4. In terms of granularity, there is a shift from day to month expressions (see Example 8). Interestingly, for the latter, there has been a move from a noun phrase expression (*every/each month*) to an adverb expression (*monthly*). While, in the intermediate periods (1800 and 1850) both expressions are typical, in 1950 only *monthly* is typical. In 1750 to 1850, *every/each month* expressions relate to observations done on a monthly basis of which the mean or average is drawn and the same applies for *monthly* used with *mean* as a term (see Example 8). In 1950, instead, *monthly* is solely used as an adverb. Thus, there is a replacement of longer noun phrase expressions (*every/each month*) by the shorter adverb expression *monthly*.

*Example 8*

*Besides this, you may there see, that every day the Sun sensibly passes one degree from West to East, [...].* (1650)

8 For example, *every day* corresponds to 365 times a year, while *annually* to once a year.

*In order to determine the annual variations of the barometer, I have taken the mean of the observations in each month, [...].* (1800)

*Example 9*

*The mean was then taken in every month of every lunar hour (attending to the signs), and the monthly means were collected into yearly means.* (1850) *A disk resident file of all current recipient numbers is created monthly from the eligibility tape file supplied by Medical Services Administration.* (1950)

#### **7 Discussion and Conclusion**

We have presented an approach to investigate diachronic change in the usage of temporal expressions. First, we use temporal tagging to obtain a more comprehensive coverage of possible temporal expressions, rather than investigating specific expressions only, as was the case in previous work. Evaluation of the tagging results showed high precision (approx. 90%) across time periods.

Second, we use relative entropy to detect typical temporal expressions of specific time periods. A clear advantage to frequency-based accounts is that with relative entropy frequent as well as rare phenomena can be investigated in terms of their 'typicality' according to a variable (here: temporal expressions typical of specific time periods). Apart from gaining knowledge on diachronic changes specific to different types of temporal expressions, we also capture more abstract and more fine-grained shifts. On a more abstract level, while our findings confirm the paradigm shift from the more observational to the more experimental character of scientific writing (cf. Fankhauser et al. (2016); Gleick (2010)) for Date and Duration expressions, we also show the tendency towards previous work descriptions for these two temporal types in contemporary scientific writing. On a more fine-grained level, for Set (a rarely used temporal type especially towards more contemporary time periods), there is a linguistic shift from longer noun-phrase to shorter adverb expressions.

These findings are not only interesting in historical linguistic terms, but are also relevant to improve adaptation of temporal taggers to different time periods. Especially for recall, gold-standard annotations are needed. Since this is a quite resource and time consuming task, our approach can help in gaining insights on the use of typical temporal expressions in specific contexts across periods. These contexts can then be further exploited in terms of possible temporal expression occurrences to achieve better recall. In addition, temporal expressions might change in terms of linguistic realization as with the Set type in our case. Accounting for shifts in linguistic realization will also improve recall. While this is true for diachronic variation, the approach also generalizes to domain-specific variation. In future work, we plan to work in this direction, further elaborating our methodology for diachronic and domain variation.

**Acknowledgments.** This work is partially funded by *Deutsche Forschungsgemeinschaft* (DFG) under grant SFB 1102: *Information Density and Linguistic Encoding* (www.sfb1102.uni-saarland.de). We are also indebted to Stefan Fischer for his contributions to corpus processing and Elke Teich for valuable improvement suggestions. Also, we thank the anonymous reviewers for their valuable comments.

#### **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **A Case Study on the Relevance of the Competence Assumption for Implicature Calculation in Dialogue Systems**

Judith Victoria Fischer(B)

Sprachwissenschaftliches Institut, Ruhr-Universit¨at Bochum, Bochum, Germany<EMAIL_ADDRESS>
**Abstract.** The *competence assumption* (CA) concerns the estimation of a user that an implicature, derived from an utterance generated in a dialogue or recommender system, reflects the epistemic state of the system about the validity of alternative expressions. The CA can be assigned globally or locally. In this paper, we present an experimental study on the effects of locally and globally assigned competence in a sales scenario. The results of this study suggest that dialogue systems should include means for modelling global competence and that assigning local competence does not improve the pragmatic competence of a dialogue system.

#### **1 Introduction**

In order to enhance user acceptance, systems that generate utterances in dialogical scenarios as, e. g., question-answering systems or recommender systems with natural language interfaces, should show some degree of pragmatic behavior. In particular, dialogue systems face the challenge of generating utterances with sufficient scope for the user's possible calculation of implicatures. In the following example, a virtual sales assistant S uses the scalar expression *good*, instead of the semantically stronger *excellent*.

S: *The HP Laptop has a good AMD Radeon graphics coprocessor.*

According to the standard view of quantity implicature calculation, a user, reading this statement, would reason as follows:

The system generated some ψ (*good*) instead of a semantically stronger alternative φ (*excellent*). There must be reasons for not generating φ: Either the system does not believe φ to be true or it believes φ not to be true:

¬Bel*S*(φ) ∨ Bel*S*(¬φ)

Which interpretation holds, relies on the *competence* or *experthood assumption*: Only if it is shared knowledge of system and user that the system is competent in a way that allows them to predict the truth value of the semantically stronger φ, the user will eventually infer the stronger reading (Potts 2015). The inference can be outlined as follows:

Two different inferences are involved in this overall picture. The first inference concerns the first epistemic assignment (i. e., 2) that just states that the speaker does not know whether or not φ holds. This so-called *weak implicature* was derived from the utterance ψ by following the Gricean quantity maxim. The *strong* or *secondary implicature* Bel*S*(¬φ) strengthens the weak implicature. It can be derived by considering the speaker's competence: Either the speaker believes that φ holds or that it doesn't hold. For further details on the relation between implicatures and the competence assumption, see Geurts (2010) and Sauerland (2004).

The competence assumption thus is a crucial component of implicature calculation, because it prevents scalar expressions to be inferred with the stronger implicature reading by default. For the competence assumption to work, certain requirements must be met. Consider the following example:

S: *I've heard a lecture at ComputerCon about that generation of graphics coprocessors. The HP Laptop has a good AMD Radeon graphics coprocessor.*

In this example, the statement that the agent had heard a lecture on graphics coprocessors suggests that S is at least competent with respect to these processors. Hence, the user will infer without doubt that the graphics coprocessor in question is not excellent, which might affect his purchase decision more strongly than the weak implicature that the system just doesn't know whether the coprocessor is excellent or not.

Thus, assuming competence for implicature calculation might have severe effects on the course of the conversation and its outcome. But does the additional information really change the user's assumptions about the system's reliability? Does the sentence *trigger* a competence assumption for the user and thus a stronger implicature reading?

The aim of this paper is to explore this issue in the context of sales and recommender systems: We are interested in positive, neutral, and negative competence triggers and their influence on the competence assumption. Furthermore, we want to know whether the CA should be triggered locally (i. e., for single assertions or themes) or globally (i. e., for the overall dialogue).

#### **2 Related Work**

Although there are a number of dialogue systems that deal with various aspects of implicature calculation, the role of the CA in legitimating implicature calculation has not been accounted for. Artificial agents have been utilized before, for either examining various pragmatic reasoning phenomena or maximizing dialogue efficiency, or both. For example, Vogel et al. (2013) use artificial agents to show that they behave in a Gricean manner to maximize their joint utility when faced with reference games or other interactional scenarios. The agent's reasoning about the opponent's belief states, modeled as a variant of the partially observable Markov decision process (POMDPs), to maximize joint utility, results in implicature-rich readings, but the weak–strong distinction has not been accounted for.

Stevens et al. (2016) show that sales dialogue efficiency can be enhanced with pragmatic question answering with indirect answers and consideration of user's requirements by using a game-theoretic model of query answering for their agent: However, the implicature triggered by the indirect answer is, due to the probabilistic model of user types, a weak one only.

Schl¨oder and Fern´andez (2015) develop a model for pragmatic rejection by means of implicature(s). Efstathiou and Lemon (2014) consider an account on non-cooperative dialogue in automated conversational systems and teach their agents to behave in this manner. It was shown that in a trading game, noncooperative behaviour such as deception could increase the agents performance in comparison to a cooperative agent. The CA does not play a role in these models as well.

Insights on the CA originate from linguistic and philosophical analyses (e. g., Geurts (2010)), but these works do not consider requirements for developing computational systems with a generation component in a dialogue setting.

## **3 Testing the Competence Assumption Locally and Globally**

Analogous to the situation in dialogue system research, the CA has not yet been a topic of empirical studies. The pragmatic approach in linguistics assumes that pragmatic reasoning is necessarily global (Sauerland 2004, p. 40), with which it refers to entire speech acts, not embedded sentences. In the context of this paper, *global* refers to an entire conversation, whereas *local* refers to a single speech act, primarily an assertion. It is not yet known on which level the CA is determined or whether it takes into account both levels of interpretation.

In our study, we confine ourselves to the question how the CA may be triggered and whether it will be established globally or locally. We consider the surface forms of the following aspects:

#### **3.1 Participants**

51 participants were consulted via http://clickworker.com/ and the experiment was distributed by https://www.testable.org/, an open platform for web experiments. Participation was limited to users from the US and UK. 15 participants were excluded due to failed attention checks, additional 9 participants were excluded, because they had response times below 2500 ms for more than one item. The threshold was set to 2500 ms as a result of a small pretest with three students, where response time minimum was 2900 ms. We assumed here that *clickworkers* are more familiar with the task at hand and that this would justify a lower response time minimum for them.

Thus included in the results are 24 participants.

#### **3.2 Materials**

The used items were priorly assigned to one of the categories "positive", "neutral" or "negative".

The examples were considered to be successful competence triggers if they achieved significant ratings within the spectrum of positive (100–70%), neutral (60–40%) or negative (30–0%) competence and in accordance to the priorly assigned categories. The examples were obtained through introspection, prior sales dialogue experiences – online and offline – and media research.

27 statements equally distributed within categories were tested. Of 27 statements, 9 were globally constructed competence triggers like short personal introductions and 18 statements were locally constructed competence triggers like mid-conversational sequences. For all statements, see the appendix; specific statements will hereafter be referred to with their item number (Table 1).

**Table 1.** Examples for global and local competence triggers

The statements and two attention checks were randomized for each participant, which were then asked to "Rate the competence of the sales person on a scale from 0 (not competent) to 100 (very competent)." Answer was given for each statement with a slider from 0 to 100. Other specifics of the sliders grid were hidden from the participants.

#### **3.3 Discussion and Results**

First of all, we compared the mean ratings of all items from both groups (local and global) with their priorly assigned categories. As shown in Fig. 1, mean values of positive and neutral items met their categories prerequisites, whereas negative items did not.

We then proceeded to compare the participants ratings for global versus local items with their assigned categories:

**Fig. 2.** Comparison of local and global triggers

As shown, global competence triggers are within their priorly assigned categories, whereas local competence triggers of the categories *positive* and *negative* are not. The results suggest that there might be a significant difference between positive-local and positive-global, negative-local and negative-global items.

The factorial ANOVA allows us to apprehend the effect of groups local and global and categories positive, neutral and negative on the competence ratings simultaneously.

The results of the ANOVA (Table 2) suggest that there is a significant difference within items and in the interaction between items and groups. No significant difference occurs within the groups local and global.

**Table 2.** Factorial ANOVA results

This is also supported by the post-hoc Tukey analysis, which gives the analysis of significant differences of different groups and categories compared to each other (e. g., positive-local versus positive-global).

The analysis shows that the following categories are significantly different (p < .05):

Furthermore, the following groups and categories are not significantly different:

This confirms what we have seen before in Fig. 2. Ignoring the groups local and global the categories show a significant difference towards each other. When comparing the interaction between groups and categories, it seems that above all positive-local items and neutral items struggled to differentiate themselves. Also, the findings confirm that negative-local and negative-global items as well as postive-local and positive-global items are significantly different.

#### **4 Implications for Future Work**

The results from this study on the role of the CA in dialogue systems suggest that the global competence assumption can be established fairly well. This holds especially true for the competence triggers of the category *positive* and *negative*, which had the most extreme competence ratings. In these categories, polite behaviour and indication of competence with mention of a certain position received the best mean ratings. The negative competence trigger with the lowest mean item number (8) worked with both attributes as well, whereas items number (7) and (9) – direct admission of incompetence and lack of time – didn't compete. Future work thus should pay attention to the factor of polite behaviour and indication of competence. In terms of negative competence triggers, competence should be indicated in an indirect manner, as direct admission of incompetence did not score as well in the mean values.

Local competence triggers of postive-local and neutral-local, as well as neutral-global items failed to distinguish themselves. It should be considered whether neutral competence triggers are beneficial in the first place, or if they can more easily be computed by a lack of competence triggers. Furthermore, local competence triggers did not score as well as their global counterparts. Hence, for computing local competence a more blunt use of polite behaviours or competence indicators should be reviewed.

The global items of the categories *positive* and *negative* were the most promising and therefore suggest that it would be beneficial to integrate a model of CA into the information states of a dialogue system. These first results of our ongoing study on competence triggers will be used for subsequent studies on the role of the CA for deriving weak and strong implicatures. On the whole, they will provide the empirical grounding for a probabilistic model of content determination in a question-answering system in a sales scenario.

# **Appendix**

global-positive: (1) My name is Mr. Miller, I am the assistant Head of the Computer Department at this store. (2) My name is Mr. Miller, I am the designated laptop expert of this store. (3) My name is Mr. Miller and I am here to help you with whatever question you might have.

global-neutral: (4) My name is Mr. Miller. (5) My name is Mrs. Miller. (6) My name is Mrs. Kowalsky.

global-negative: (7) My name is Mr. Miller. Unfortunately I have to inform you, that this is not my department, I am covering this shift for a colleague. (8) My name is Mr. Miller. This is my first day at this store. So please; cut me some slack. (9) My name is Mr. Miller. My shift ends in 5 min, but I'm sure we'll find something for you.

local-positive: (10) Oh, that is easy: (11) I heard a lecture about this at ComputerCon. (12) I'm glad you ask. (13) When I was upgrading my home PC, I came across some information about this: (14) I recently read an article about that. (15) Ah, that is a fascinating piece of hardware.

local-negative: (16) Let me look that up for a second. (17) I have to admit I don't know a lot about that. (18) Sadly, this is not my field of expertise. (19) Well, I don't know much about those to be honest, but what I can tell you is that: (20) I don't think this makes much of a difference, but: (21) I'll have to check in with my colleague about that.

local-neutral: (22) Interesting question. (23) Of course. (24) Well, in regards to this question, I'd have to say: (25) The specifics to that are: (26) As far as I am concerned: (27) On behalf of that, it turns out that.

# **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Supporting Sustainable Process Documentation**

Markus G¨artner1(B), Uli Hahn2, and Sibylle Hermann3

1 Institute for Natural Language Processing, University of Stuttgart, Stuttgart, Germany<EMAIL_ADDRESS>
2 Communication and Information Centre, Ulm University, Ulm, Germany<EMAIL_ADDRESS>
3 University Library, University of Stuttgart, Stuttgart, Germany<EMAIL_ADDRESS>
**Abstract.** In this paper we introduce a software design to greatly simplify the elicitation and management of process metadata for researchers. Detailed documentation of a research process not only aids in achieving reproducibility, but also increases usefulness of the documented work for others as a cornerstone of good scientific practice. However, in reality, time pressure together with the lack of simple documentation methods makes documenting workflows an arduous and often neglected task. Our method for a clean process documentation combines benefits of version control with integration into existing institutional infrastructure and a novel schema for describing process metadata.

#### **1 Introduction**

Lately the term "reproducibility crisis" has frequently been brought up in the context of criticism on research quality in general. However, the German Research Foundation (DFG) correctly emphasizes1 that reproducibility2 is only one of many quality measures for good research. Furthermore, it asks for increased attention to questions raised in the close-by area of research data management (RDM). The topic is also in line with recent survey results3 which stress the insecurity of many researchers when it comes to RDM and the related issues of research data sustainability.

Following the idea of Claerbout and Karrenbach (1992) that articles are only advertisement for the actual scholarship, due diligence should also be exercised in compiling and publishing documentations of research processes alongside the results. This helps especially for tasks that naturally can be reproducible if documented thoroughly, such as computations with strictly deterministic outcomes.

1 Press release by the German Research Foundation (DFG) on "Replicability of Research Results", April 25th 2017.

2 We do not intend to delve into the deeper semantic discussion of "reproducibility" vs. "replicability" and treat both terms as synonyms here.

3 By project bwFDCommunities http://bwfdm.scc.kit.edu or Humboldt-University Berlin http://nbn-resolving.org/urn:nbn:de:kobv:11-100213001.

c The Author(s) 2018

G. Rehm and T. Declerck (Eds.): GSCL 2017, LNAI 10713, pp. 284–291, 2018. https://doi.org/10.1007/978-3-319-73706-5\_24

But even if a process conceptually defies reproduction (e.g., results of strictly qualitative analysis) a good documentation can increase usability.

Today's research practice however is dominated by competitive and time pressures. Together with the focus on producing publishable end results combined with lack of gratification for detailed process documentation this leads to a serious neglect of documentation efforts. Our project aims at filling that gap with the software tool described in this paper. The goal is to support scientists in creating process documentation already during the workflow with minimal effort. This concept is being exemplified with use cases from researchers in computational linguistics and digital humanities where a plethora of multifaceted workflows exist.

We contextualize our work in Sect. 2 and introduce our schema for process metadata in Sect. 3. Section 4 provides an overview of our software design and shows possibilities of interconnections to related infrastructure. In Sect. 5 we summarize the findings and hint at possibilities of how this concept can be integrated within a broader context.

#### **2 Related Work**

In the context of documenting a research process, systems for two slightly distinct fields are relevant: Workflow management and workflow tracking.

A *Workflow management system* (WMS) is software that helps in setting up workflows as collections of interdependent (executable) steps. The list of (commercial) WMSs for general-purpose or enterprise use is extensive, but their usability for specialized research workflows is limited. For certain research fields dedicated WMS instances have emerged. *GenePattern*4 (Reich et al. 2006) for instance allows the creation, management and execution of analysis pipelines for genome research.

The task of *workflow tracking* is more reactive and involves documenting steps during or after their execution. Tools similar to *YesWorkflow* (McPhillips et al. 2015) offer the ability to annotate data flows and operations on the code level and to receive a graph visualization of the implicit workflow. They are however primarily aimed at script-heavy workflows. More related to software development, version control systems like Git5, Apache Subversion (SVN)6 or others provide sufficient functionality for tracking and documenting complex collaborative development workflows.

In stark contrast to the elaborate solutions listed so far, in practice it is not uncommon for researchers to document their workflow simply by means of a local Word or Excel file. Reasons for this given in surveys include the complexity of many workflow management or tracking systems, the barrier of entry for technically unskilled users and available systems essentially being not specialized enough for a given workflow. In addition many solutions focus on

4 http://www.genepattern.org.

5 https://git-scm.com.

6 https://subversion.apache.org.

creation and maintenance of executable analysis or processing pipelines, making them unsuited for workflows that involve manual steps, such as annotation or curation.

By splitting process documentation into metadata for *resources* (objects) used in it and the *actions* performed, the need to interface with existing metadata infrastructure becomes evident. Initiatives like CLARIN (Hinrichs and Krauwer 2014) already provide wide coverage of metadata repositories for communities in computational linguistics and digital humanities. While having rich object metadata available is by no means a sufficient replacement of proper process documentation, it does provide a valuable foundation to build on.

# **3 Process Metadata**

We follow the more common convention of modeling workflows as directed acyclic graphs (DAG) (Deelman et al. 2009) where each node represents a single step. As pointed out in Sect. 2, there are already established initiatives and systems for the provision or archiving of object metadata. That is metadata associated with an individual resource, usually created and archived after reaching a mature state in the resource's development or creation lifecycle. Rich in detail, such metadata records are also typically following schemas specific to the field of research they originated from. For example in the context of CLARIN the Component MetaData Infrastructure (CMDI)7 is used for entries in the CLARIN Virtual Language Observatory (VLO)8. Due to such infrastructures also providing means for persistent identification of individual records9, the metadata schema in this section is focused solely on the process itself. To this end our schema defines the following fields for a single workflow step (with the field's multiplicity in round brackets if it is optional):

**Title.** User-defined short label for the workflow step. This is meant to be very compact and can also be reduced to simply state the type of the task performed, such as "Annotation".

**Description.** This more detailed description of the workflow step is meant to be a human readable explanation that should contain enough information for another researcher to understand what was done. It is also the place where basically arbitrary additional notes can be placed. It will help to find and keep track of decisions or expectations and to raise the reusability for others.

**Input (0..n).** Resources used to perform the action. This includes an extremely diverse spectrum of resources which are in turn highly task-specific. They can range from local resources of arbitrary type (corpora, model files, configuration files, pictures, audio, etc.) to online resources (e.g., public annotation guidelines) to "pointers" at real objects such as books that don't exist in digitized form.

7 http://www.clarin.eu/cmdi.

8 https://vlo.clarin.eu.

9 E.g., metadata for the TIGER Corpus: http://hdl.handle.net/11022/1007-0000- 0000-8E2D-F.

**Output (0..n).** Resources generated by the action. Unlike input resources, these usually represent exclusively local files, since the workflow is assumed to take place on a local system.

**Tool (0..1).** The executable resource used for processing. This is either a local program or a web-service and in both cases command line parameters or settings can and should be recorded.

**Person (0..n).** Human subjects involved in the workflow step. Similar to Input, the content of this field is quite diverse, including, but not limited to, annotators, curators or experiment participants.

**Custom properties (0..n).** Arbitrary classic textual key-value metadata entries that can be added, primarily to provide machine readable storage options for metadata created by future plugins or if the user wishes to include structured information beyond the free-text in the Description.

Complex fields (Input, Output, Person, Tool) get assigned one or more typed identifiers. These identifiers take the form of *<*type, id*>* where id is the actual identification string such as a name or web address and type defines how that id is to be interpreted. These identifiers can also be used to link resources or persons to public repositories or databases (e.g., VLO-Handles for CLARIN resources or an ORCID10 for registered researchers).

The standard serialization format for the process metadata is JSON11. Our tool uses it to store process metadata locally as described in Sect. 4.1 and for exporting (parts of a) workflow documentation. Since JSON is a very simple and also widely used format, it is easily possible to convert the output, or process it with other systems.

### **4 Architecture**

In this section we provide an overview of our architecture, especially the design of the client software which is being developed in our project. The core component is a Java application which bundles the user interactions required for tracking and documenting the workflow. Behind the scenes this client uses a local Git repository for tracking changes to file resources in *workspace* folders designated by the user. The client is also meant to provide a wide range of additional functionality, among others the ability to interact with existing data and/or metadata repositories to store or retrieve resources or metadata. We describe some of these features in the following Sects. 4.1 through 4.3.

#### **4.1 Git and Process Documentation**

The Git system allows version control of a local folder's content, that is, it tracks changes to files inside this folder so that individual states or "versions" can be recorded and referenced.

10 https://orcid.org.

11 http://www.json.org.

**Storage of Process Metadata.** The recording of a workspace's state is triggered by so called Git "commit" operations. In our workflow model every workflow step corresponds to a single Git commit. Each commit is also accompanied by its respective "commit message". Those messages commonly are humanreadable descriptions made by the user to explain the nature or reason of performed modifications. However, our application design completely hides the direct Git interface from the end users as to not overwhelm them with unneeded complexity or functions. This way we can use commit messages to store process metadata (cf. Sect. 3) and thereby directly associate physical changes to the data with matching (formal) documentation.

Unfortunately Git can only automatically detect those local files that represent the "output" of a workflow step (since those have been modified or are new). This means that the completeness of a resulting workflow documentation ultimately still relies on the user. We plan assistive functions in the client that try to suggest involved input resources when recording a workflow step to reduce the effort required by the user. Their implementation is at this point however still an open issue and the correctness and usability will have to be evaluated together with the user community at a later stage.

**Increasing Documentation Consistency.** Manually documenting a workflow is prone to common mistakes, such as forgetting to include changes to a resource in the documentation or introducing inconsistencies in the descriptions. As stated above, Git cannot guarantee completeness when recording all the input resources used in a workflow step. It does on the other hand track reliably all the changes made to files that are under version control. As a result it makes it impossible to miss modification on tracked files and reminds the user to document all of them. Having the entire workflow history available when recording a new workflow step, also enables the client to detect inconsistent documentation, for instance when the user tries to assign a different description to a resource which has been previously used in another step.

**Trying Alternatives.** Only very few research workflows ever result in a strictly linear concatenation of performed steps. Usually there are situations during a workflow where an assumption was found to be untrue or an evaluation yielded unsatisfactory results. In those cases the researcher typically "goes back" and pursues an alternative approach by performing a different workflow step on the same previous workspace state (e.g., testing new settings for an analysis tool or training systems on different data, etc.). A workflow graph displays this behavior as branches originating from the same node to concurrent new child nodes. Git offers a similarly named functionality where a workflow can split into independent branches and the folder under version control can be changed to reflect any previously recorded state at will.

**Backup and Cooperation.** Git represents a decentralized or distributed version control system. While every local repository itself allows full version control, one can also import or export changes from and to a remote repository. In the context of our client this allows the "local workflow" to be connected to a remote Git repository (e.g., an institute or university wide GitLab12 installation). Benefits of this include for example an additional layer of backup for valuable resources used in the workflow or the ability to cooperatively work on the same "shared" workspace from different locations.

While building upon the Git system offers many advantages, there are also limitations and issues to address when using it for workflow documentation, especially in very resource-heavy computational projects. Since Git basically has to duplicate every file that is kept under version control, this leads to a very high storage consumption when used for already big resources, such as web corpora. As a solution the user can exclude files from being tracked, so that they won't affect performance.

### **4.2 Client Customizability**

With the process metadata outlined in Sect. 3 we can model a very broad spectrum of diverse workflows and therefore make the client usable for researchers in many fields. Different research fields and also universities or institutes often already have individual infrastructures for archiving or management of resources and metadata in place. To not create redundancies the following principles are taken into account for the client design:

**Independence.** In the most basic version our client requires absolutely no external infrastructure or third-party software to work with, besides the libraries it is shipped with. It will provide the full functionality of workflow documentation and also enable the user to create and store object metadata locally in a simple (customizable) schema following Dublin Core (Powell et al. 2005). In this configuration the user can work completely network-independent and also is not reliant on other infrastructure, making the client very light-weight.

**Extensibility.** To be able to incorporate the client into existing institutional infrastructure we use a plugin-architecture. This allows for example customized implementations for interfacing with additional repositories to be added.

#### **4.3 External Repositories**

In addition to workflow documentation in private domains (Treloar et al. 2007), the client also gives the possibility to collaborate in the shared (but not publicly open) domain, and to publish partial or final results in the public domain. To meet both requirements there are two systems with their respective interfaces that will be supported: For publishing within the shared domain, ResourceSpace13 is being used. The repository software DSpace14 (Smith 2002) is used

12 https://gitlab.com.

13 https://www.resourcespace.com.

14 http://www.dspace.org.

for publishing data with a permanent identifier (DOI) in the public domain. DSpace is a popular software for institutional publication repositories. We plan to interface with ResourceSpace for the shared domain, as it offers a better rights management as well as the possibility to share data within defined communities.

# **5 Outlook**

In this paper we introduced our design of a software supporting process documentation. We have shown the essential benefits of using version control software such as Git as a foundation for workflow tracking with a main focus on computational linguistics and digital humanities. In addition, we propose a simple yet very expressive metadata schema to describe individual steps in a research workflow. Keeping those principles separated – namely the distinction between metadata describing *objects* used in a workflow and the *actions* performed – enables the software to be very flexible. As a result it will be fairly easy to adopt the tool to specific needs (of other disciplines) and also to integrate it into the diverse landscape of existing infrastructures.

**Acknowledgments.** This work was funded by the Ministry for Science, Research and the Arts in Baden-W¨urttemberg (MWK) via the E-Science project "RePlay-DH".

# **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Optimizing Visual Representations in Semantic Multi-modal Models with Dimensionality Reduction, Denoising and Contextual Information**

Maximilian K¨oper(B), Kim-Anh Nguyen, and Sabine Schulte im Walde

Institut f¨ur Maschinelle Sprachverarbeitung, Universit¨at Stuttgart, Pfaffenwaldring 5B, 70563 Stuttgart, Germany<EMAIL_ADDRESS>
**Abstract.** This paper improves visual representations for multi-modal semantic models, by (i) applying standard dimensionality reduction and denoising techniques, and by (ii) proposing a novel technique *ContextVision* that takes corpus-based textual information into account when enhancing visual embeddings. We explore our contribution in a visual and a multi-modal setup and evaluate on benchmark word similarity and relatedness tasks. Our findings show that NMF, denoising as well as *ContextVision* perform significantly better than the original vectors or SVD-modified vectors.

#### **1 Introduction**

Computational models across tasks potentially profit from combining corpusbased, textual information with perceptional information, because word meanings are grounded in the external environment and sensorimotor experience, so they cannot be learned only based on linguistic symbols, cf. the grounding problem (Harnad 1990). Accordingly, various approaches on determining semantic relatedness have been shown to improve by using multi-modal models that enrich textual linguistic representations with information from visual, auditory, or cognitive modalities (Feng and Lapata 2010, Silberer and Lapata 2012, Roller and im Walde 2013, Bruni et al. 2014, Kiela et al. 2014, Kiela and Clark 2015, Lazaridou et al. 2015).

While multi-modal models may be realized as either count or predict approaches, increasing attention is being devoted to the development, improvement and properties of low-dimensional continuous word representations (socalled *embeddings*), following the success of *word2vec* (Mikolov et al. 2013). Similarly, recent advances in computer vision and particularly in the field of deep learning have led to the development of better visual representations. Here, features are extracted from convolutional neural networks (CNNs) (LeCun et al. 1998), that were previously trained on object recognition tasks. For example, Kiela and Bottou (2014) showed that CNN-based image representations perform superior in semantic relatedness prediction than other visual representations, such as an aggregation of SIFT features (Lowe 1999) into a bag of visual words (Sivic and Zisserman 2003).

Insight into the typically high-dimensional CNN-based representations is sparse, however. It is known that dimension reduction techniques, such as Singular Value Decomposition (SVD), improve performance on word similarity tasks when applied to word representations (Deerwester et al. 1990). In particular, Bullinaria and Levy (2012) observed highly significant improvements after applying SVD to standard corpus vectors. In addition, Nguyen et al. (2016) proposed a method to remove noisy information from word embeddings, resulting in superior performance on a variety of word similarity and relatedness benchmark tests.

In this paper, we provide an in-depth exploration of improving visual representations within a semantic model that predicts semantic similarity and relatedness, by applying dimensionality reduction and denoising. Furthermore, we introduce a novel approach that modifies visual representations in relation to corpus-based textual information. Following the methodology from Kiela et al. (2016), evaluations are carried out across three different CNN architectures, three different image sources and two different evaluation datasets. We assess the performance of the visual modality by itself, and we zoom into a multimodal setup where the visual representations are combined with textual representations. Our findings show that all methods but SVD improve the visual representations. This improvement is especially large on the word relatedness task.

## **2 Methods**

In this section we introduce two dimensionality reduction techniques (Sect. 2.1), a denoising approach (Sect. 2.2) and our new approach *ContextVision* (Sect. 2.3).

#### **2.1 Dimensionality Reduction**

*Singular Value Decomposition* (SVD) (Golub and Van Loan 1996.) is a matrix algebra operation that can be used to reduce matrix dimensionality yielding a new high-dimensional space. SVD is a commonly used technique, also refered to as Latent Semantic Analysis (LSA) when applied to word similarity. *Non-negative matrix factorization* (NMF) (Lee and Seung 1999) is a matrix factorisation approach where the reduced matrix contains only non-negative real numbers (Lin 2007). NMF has a wide range of applications, including topic modeling, (soft) clustering and image feature representation (Lee and Seung 1999).

#### **2.2 Denoising**

Nguyen et al. (2016) proposed a denoising method **(DEN)** that uses a non-linear, parameterized, feed-forward neural network as a filter on word embeddings to reduce noise. The method aims to strengthen salient context dimensions and to weaken unnecessary contexts. While Nguyen et al. (2016) increase the dimensionality, we apply the same technique to reduce dimensionality.

#### **2.3 Context-Based Visual Representations**

Our novel model *ContextVision* (CV) strengthens visual vector representations by taking into account corpus-based contextual information. Inspired by Lazaridou et al. (2015), our model jointly learns the *linguistic* and *visual* vector representations by combining two modalities (i.e., the linguistic modality and the visual modality). Differently to the multi-modal Skip-gram model by Lazaridou et al. (2015), we focus on improving the visual representation, while Lazaridou et al. aim to improve the linguistic representation, without performing updates on the visual representation, which are fixed in advance.

The linguistic modality uses contextual information and word negative contexts, and in the visual modality the visual vector representations are strengthened by taking the corresponding word vector representations, the contextual information, and the visual negative contexts into account.

We start out with describing the Skip-gram with negative sampling (SGNS) (Levy and Goldberg 2014) which is a variant of the Skip-gram model (Mikolov et al. 2013). Given a plain text corpus, SGNS aims to learn word vector representations in which words that appear in similar contexts are encoded by similar vector representations. Mathematically, SGNS model optimizes the following objective function:

$$\mathbf{J}\_{SGNS} = \sum\_{w \in V\_W} \sum\_{c \in V\_C} \mathbf{J}\_{ling}(w, c) \tag{1}$$

$$\begin{split} \mathbf{J}\_{ling}(w, c) &= \#(w, c) \log \sigma(w, c) \\ &\quad + \; k\_l \cdot \mathbb{E}\_{c \sim P\_D} [\log \sigma(-w, c\_N)] \end{split} \tag{2}$$

where J*ling*(w, c) is trained on a plain-text corpus of words w ∈ V*W* and their contexts c ∈ V*C* , with V*W* and V*C* the word and context vocabularies, respectively. The collection of observed words and context pairs is denoted as D; the term #(w, c) refers to the number of times the pair (w, c) appeared in D; the term σ(x) is the sigmoid function; the term k*l* is the number of linguistic negative samples and the term c*N* is the linguistic sampled context, drawn according to the empirical unigram distribution P. In our model, SGNS is applied to learn the linguistic modality.

In the visual modality, we improve the visual representations through contextual information; therefore the dimensionality of visual representations and linguistic representations needs to be equal in size. We rely on the denoising approach (Nguyen et al. 2016) to reduce the dimensionality of visual representations. The visual vector representations are then enforced by (i) directly increasing the similarity between the visual and the corresponding linguistic vector representations, and by (ii) encouraging the contextual information which co-occurs with the linguistic information. More specifically, we formulate the objective function of the visual modality, J*vision*(v*w*, c), as follows:

$$\begin{aligned} \mathbf{J}\_{vision}(v\_w, c) &= \#(v\_w, c)(\cos(w, v\_w) \\ &+ \min\{0, \theta - \cos(v\_w, c) + \cos(w, c)\}) \\ &+ k\_v \cdot \mathbb{E}\_{c\_V \sim P\_V}[\log \sigma(-v\_w, c\_V)] \end{aligned} \tag{3}$$

where J*vision*(v*w*, c) is trained simultaneously with J*ling*(w, c) on the plain-text corpus of words w and their contexts c. v*w* represents the visual information corresponding to the word w; and term θ is the margin; cos(x, y) refers to the cosine similarity between x and y. The terms k*v*, E*cV* , and P*V* are similarly defined as the linguistic modality. Note that if a word w is not associated with the corresponding visual information v*w*, then J*vision*(v*w*, c) is set to 0.

In the final step, the objective function which is used to improve the visual vector representations combines Eqs. 1, 2, and 3 by the objective function in Eq. 4:

$$J = \sum\_{w \in V\_W} \sum\_{c \in V\_C} \left( \mathcal{J}\_{ling}(w, c) + \mathcal{J}\_{vision}(v\_w, c) \right) \tag{4}$$

#### **3 Experiments**

#### **3.1 Experimental Settings**

We use an English Wikipedia dump1 from June 2016 as the corpus resource for training the *ContextVision*, containing approximately 1.9B tokens. We train our model with 300 dimensions, a window size of 5, 15 linguistic negative samples, 1 visual negative sample, and 0.025 as the learning rate. The threshold θ is set to 0.3. For the other methods dimensionality reduction is set to 3002 dimensions. For the resources of image data, we rely on the publically available visual embeddings taken from Kiela et al. (2016)3. The data was obtained from three different image sources, namely Google, Bing, and Flickr. For each image source three state-of-the-art convolutional network architectures for image recognition were applied: AlexNet (Krizhevsky et al. 2012), GoogLeNet (Szegedy et al. 2015) and VGGNet (Simonyan and Zisserman 2014). In each source–CNN combination, the visual representation of a word is simply the centroid of the vectors of all images labeled with the word (mean aggregation). This centroid has 1024 dimensions for GoogLeNet and 4096 dimensions for the remaining two architectures. The size of the visual vocabulary for Google, Bing, and Flickr after computing the centroids is 1578, 1578, and 1582 respectively. For evaluation we relied on two human-annotated datasets, namely the 3000 pairs from MEN (Bruni et al. 2014) and the 999 pairs from SimLex (Hill et al. 2015). MEN focuses on relatedness, and SimLex focuses on similarity.

1 https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles.xml.bz2.

2 We conducted also experiments with 100 and 200 dimensions and obtained similar findings.

3 http://www.cl.cam.ac.uk/∼dk427/cnnexpts.html.

#### **3.2 Visual Representation Setup**

Table 1 shows the results for each of the previously introduced methods, as well as the unmodified image representation (Default). It can be seen that NMF, DEN and CV increase performance on all settings except for the combination Google & AlexNet. The performance of SVD is always remarkably similar to its original representations.

Furthermore we computed the average difference for each method across all settings, as shown in Table 2. The performance increased especially on the MEN relatedness task. Here NMF obtains on average a rho correlation of ≈.10 higher than its original representations. Also DEN and CV show a clear improvement, with the latter being most useful for the SimLex task.

To ensure significance we conducted *Steiger* 's test (Steiger 1980) of the difference between two correlations. We compared each ouf the methods against its Default performance.

**Table 1.** Comparing dimensionality reduction techniques, showing Spearman's ρ on SimLex-999 and MEN. \* marks significance over the Default.

**Table 2.** Average gain/loss in ρ across sources and architectures, in comparison to Default.

Out of the 19 settings, NMF obtained significant improvements with \*=p < 0.001 in 11 cases. Despite having a lower average gain (Table 2), DEN and CV obtained more significant improvements.

In total we observed most significant improvements on images taken from bing and with the CNN GoogLeNet.

#### **3.3 Multi-modal Setup**

In the previous section we explored the performance of the visual representations alone.

We now investigate their performance in a multi-modal setup, combining them with a textual representation. Using the same parameters as in Sect. 3.1 we created word representations relying on an SGNS model (Mikolov et al. 2013). We combined the representations by scoring level fusion (or late fusion). Following Bruni et al. (2014) and Kiela and Clark (2015) we investigate the impact of both modalities by varying a weight threshold (α). Similarity is computed as follows:

$$
\sin(x, y) = \alpha \cdot \lim\_{} (x, y) + (1 - \alpha) \cdot \operatorname{vis}(x, y) \tag{5}
$$

Here ling(x, y) is cosine similarity based on the textual representation only and vis(x, y) for using the visual space.

For the following experiment we focus on AlexNet, varying the image resource between bing for the SimLex task and flickr for the MEN task. The results are shown in Fig. 1a for SimLex, and in Fig. 1b for MEN.

It can be seen that all representations obtain superior performance on the text-only representation (black dashed line, SimLex ρ = .384, MEN ρ = .741). The highest correlation can be obtained using the DEN or VC representations for SimLex. Interestingly these two methods obtain best performance when given equal weight to both modalities (α = 0.5) while the remaining methods as well as

**Fig. 1.** (a) Comparing multi-modal results on SimLex-999. Image representation from bing using AlexNet. Y-Axis shows Spearman's ρ. X-axis changes impact of each modality, from only image to the far left to only textual representation. (b) Multimodal results on MEN. Image representation from flickr using AlexNet.

the unmodified default representations obtain a peak in performance when given more weight to the textual representation. A similar picture emerges regarding the results on MEN, where also NMF obtains superior results (.748).

# **4 Conclusion**

We successfully applied dimensionality reduction as well as denoising techniques, plus a newly proposed method *ContextVision* to enhance visual representations within semantic vector space models. Except for SVD, all investigated methods showed significant improvements in single - and multi-modal setups on the task of predicting similarity and relatedness.

**Acknowledgments.** The research was supported by the DFG Collaborative Research Centre SFB 732 (Maximilian K¨oper, Kim-Anh Nguyen), the DFG Heisenberg Fellowship SCHU-2580/1 (Sabine Schulte im Walde) and the Ministry of Education and Training of the Socialist Republic of Vietnam (Scholarship 977/QD-BGDDT; Kim-Anh Nguyen). We would like to thank the four anonymous reviewers for their comments and suggestions.

# **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Using Argumentative Structure to Grade Persuasive Essays**

Andreas Stiegelmayr and Margot Mieskes(B)

University of Applied Sciences, Darmstadt, Germany<EMAIL_ADDRESS>
**Abstract.** In this work we analyse a set of persuasive essays, which were marked and graded with respect to their overall quality. Additionally, we performed a small-scale machine learning experiment incorporating features from the argumentative analysis in order to automatically classify good and bad essays on a four-point scale. Our results indicate that bad essays suffer from more than just incomplete argument structures, which is already visible using simple surface features. We show that good essays distinguish themselves in terms of the amount of argumentative elements (such as major claims, premises, etc.) they use. The results, which have been obtained using a small corpus of essays in German, indicate that information about the argumentative structure of a text is helpful in distinguishing good and bad essays.

### **1 Introduction**

Writing essays is an essential part of every-day-life of pupils and students. In persuasive essays there is an additional challenge in getting argumentative structures right. Research in automated essay scoring has been looking at a wide variety of features such as text structure, vocabulary, spelling, etc. All of which are important, but considering current research in argument mining, there is a lack of research into the relationship between argument structure and essay quality. In this work, we address how various aspects of arguments (i. e., major claims, premises, etc.) relate to the quality of an essay. Additionally, we use features based on arguments in a classification task using machine learning methods. Our results indicate that persuasive essays can be reliably classified using argument-based features. This work contributes in two ways to research in the area of argument mining and essay scoring: First, we show that the argumentative structure can be used to distinguish good and bad essays in an essay scoring task. Second, to our knowledge, this is the first work to bring these two topics together based on German data.

### **2 Related Work**

As this work is at the intersection of *argument mining* and *essay scoring* we look at relevant previous work in both areas. Reviewing the available literature in detail is beyond the scope of this paper.

#### **2.1 Argument Mining**

Although the topic of argument mining is fairly new, it goes back to acient greece. Habernal and Gurevych (2017) provide a current, extensive overview on the area. We specifically looked at the guidelines presented by Stab and Gurevych (2014): The authors analysed three components for argument structures: Major Claim, Claim and Premise. The basis of an argument is the claim, which relates to one or more premises. This relation has two attributes: support and attack. The Major Claim is the basis for the whole essay and can be found either in the introduction or in the conclusion. In the introduction it serves as a statement, which is related to the topic of the essay. In its conclusion it summarizes the arguments of the author.

Wachsmuth et al. (2016) also based their work on Stab and Gurevych (2014), but they consider Argumentative Discourse Units (ADU). ADUs can be complete sentences or partial sentences, especially in cases where two sentences are connected via "and". The authors defined a set of features, such as n-grams, part-of-speech n-grams, etc., and analysed the flow of ADUs based on graphs.

Work on German data is (compared to English data) rare. One example is by Peldszus and Stede (2013), where artificially constructed short texts were used to determine inter-annotator aggreement on argument annotation. Kluge (2014) used web documents from the educational domain, and Houy et al. (2013) used legal cases. All authors analysed the argumentative structure of their documents.

Work on essays has been carried out for example by Faulkner (2014), but with the aim of identifying the stance of an author towards a specific claim and in the domain of summarization. Stab and Gurevych (2014) also used essays in their study, but focused on the identification of arguments.

#### **2.2 Essay Scoring**

Dong and Zhang (2016) present an overview on essay scoring, including commercial tools available. They analysed a range of features for essay scoring and used them in a deep learning approach. The authors used surface features such as the length of characters, words, etc., and linguistic features such as Part-of-Speech (POS) tags and POS-n-grams. They used words and their syonyms based on the prompts for each essay and their appearance in the resulting texts. Additionally, they used uni- and bigrams and corrected for spelling errors. They considered the task as a binary classification task, with good essays defined as "essays with a score greater than or equal to the average score and the remainder are considered as bad scoring essays". The authors report a κ-based evaluation, which achieves results "close to that of human raters".

Using arguments for essay scoring has been done by Ghosh et al. (2016), based on TOEFEL-data. Their results, based on number and length of argument chaines, indicate that essays containing many claims, connected to few or no premises score lower. They also found that length is highly correlated with the scores.

#### **3 Data Set**

We collected a corpus containing 38 essays, which are available on the internet1. We also tried to get real essays by contacting various schools and teachers. These would also have teachers markings. Unfortunately, this is not a viable path to follow, due to various reasons: Firstly, these essays are subject to a very strict data protection law, which puts a range of obstacles on obtaining such data. Secondly, very few schools use electronic methods and tools for writing essays. So all schools we got in touch with and which would have been willing to grant us access to their essays and markings, provided we agree to the data protection regulations, only had essays which were hand-written on paper. Digitizing them, including proof-reading, would have been beyond the scope of this work. Therefore, we took data that was available on the internet in a machinereadable format. The corpus was manually annotated using the guidelines by Stab and Gurevych (2014) using WebAnno2. Figure 1 shows an example structure of the resulting argument tree. The whole data set contains approximately 120,000 words, and slightly over 4,000 sentences. In total, we analysed over 1,000 argument units containing over 1,000 premises and almost 300 claims. 50% of the argument units had more than 15 words. Details can be found in Table 1.

In addition to the argument annotation, we also annotated the quality of the essays, using the German school marking system, which is based on numbers 1 to 4, where 1 represents a very good result and 4 represents a very poor result. We decided to use a reduced version of the German marking systems due to the following reasons: At universities only marks from 1 to 4 are given, with marks >4 being a *fail* 3. Due to the data set size, using a more fine-grained marking scale would have given us very few data points for each class to train a machine learning system on, especially with respect to the already small data set size.

**Fig. 1.** Example for an argument tree as found in our data.

1 The list of sources can be found at https://b2drop.eudat.eu/s/tR5spZeyRcW20VB.

2 https://webanno.github.io/webanno/.

3 One annotator marked one essays with a fail.

**Table 1.** Statistical information on the corpus.

We assume, that the quality of the essay corpus is not representative of regular school essays, but rather represent the quality available on the interent. We observe, that the quality of the essays is mediocre, with many authors not explicitely stating their point of view. In some extreme cases the major claim was not detectable. This results in difficulties in deciding whether a sentence contains an argument unit or not. The distribution of the marks is therefore very skewed, with approximately 23.1% of the essays achieving good (mark 2) or very good (mark 1) marks and 77% of the essays achieving poor (mark 3) or very poor (mark 4) results. An additional problem – especially for the later automatic analysis – is the usage of metaphors, which we did not look into in this work.

About one third of the essays (13 out of 38) were graded by two persons. The percentage agreement between the two grades was 0.53. Considering a measure that is specifically designed to evaluate annotations by two coders and correcting for chance agreement (which percentage agreement does not do), we achieve a value of S = 0.42, which according to Landis and Koch. (1977) shows a moderate agreement. All values were calculated using DKPro Statistics4 (Meyer et al. 2014).

#### **4 Experimental Setup**

We use DKPro Components5, such as DKPro Core, DKPro TC and Uby for our experiments.

We defined a range of features, based on the argumentation annotation and previous work. We distinguish between *baseline* features, which have already been used in previous work and *argument* features, which are based on the argumentation annotation. The baseline features contain easy to determine features,

4 https://dkpro.github.io/dkpro-statistics/.

5 https://github.com/dkpro/.

such as number of tokens, number of sentences, etc. Additionally, we took into account POS-based features, which include nouns, verbs, adjectives, etc.

Based on earlier work, we included information about whether the author used overly long words or short words. We also checked for spelling errors using the LanguageTool6. Wachsmuth et al. (2016) observed that questions are not arguments. Therefore, we also extracted the number of questions with and without arguments. According to Stab and Gurevych (2014) one paragraph should only contain one claim. Therefore, we also counted the number of claims and the number of paragraphs in our documents. Additionally, we looked at the number of sentences with and without arguments. Finally, we examined the n-grams found in the annotated arguments. Based on Ghosh et al. (2016) we looked at the graph created by the argument structure over a document. An example can be found in Fig. 1. Tree size and grade show a strong, negative correlation (Pearsons r = *−*0.577), meaning, the larger the tree, the higher the grade. Additionally, we use the argument graph to determine whether it starts with a major claim or not and which arguments are not linked to the major claim. Finally, we determined whether a person consistently uses the correct tense. The full set of features can be found in the respective .arff-files8.

#### **5 Results and Discussion**

We experimented with various machine learning algorithms, using WEKA9. As we wanted to gain a qualitative insight into the results obtained through the machine learning methods, we specifically looked into decision trees (J48).

**Table 2.** Classification result for individual marks using the whole feature set.

We observed, that the main features contributing to the results in Table 2 were NrOfMajorClaims, NrOfPremises and RatioSentenceNonQuestion. This supports earlier work, that the number of major claims and premises allows for detecting good essays.

6 https://languagetool.org/de/.

7 This correlation is significant on α = 0.01.

8 https://b2drop.eudat.eu/s/mTyUTJrCNyO4I3e.

9 http://www.cs.waikato.ac.nz/ml/weka/.

**Table 3.** Classification result for individual marks using baseline features only.

Using only the baseline features we observed that the lower marks (3 and 4) were still classified fairly reliably, but the better marks (1 and 2) performed very poorly. Looking into the results in detail revealed that essays marked as "2" were mostly confused with essays marked as "1", which indicates, that not so good essays suffer from more than just a lack of good argumentative structure, which is already visible with the surface features. This becomes very prominent looking at the resulting tree, where the most important features for 3 and 4 were a combination of fewer characters and a high amount of spelling errors. In order to reduce the importance of the spelling errors, we artificially introduced spelling errors to the good essays (marked 1 and 2). We tried to achieve a similar ratio as for the bad essays (marked 3 and 4). Thereby, we managed to reduce the importance of the spelling feature in the feature ranking. But the overall results (including the observations concerning the usage of major claims and premises in connection to the resulting grade) were similar to those presented in Table 2 (p = 0.86; r = 0.85 and f1=0.85) and the discussion above (Table 3).

**Table 4.** Classification result for individual marks using custom features only.

The argumentative features allow us to clearly identify and distinguish between the various essays. A closer look at the resulting tree indicates, that good essays use premises cautiously and also keep the major claims low, which is in line with observations from previous work. Bad essays have a higher number of major claims, but also a high number of disconnected arguments (Table 4).

Overall, our results indicate that poor essays suffer from more than just poor argumentation and authors should address issues such as spelling, usage of tense, number of conjunctions and length of words. Once these issues are considerably improved, the argumentative elements of the essays should be considered, such as a high number of major claims. For authors who already achieve good results, the focus can be put on argumentative elements, such as the number of premises, which is higher than in very good essays.

### **6 Conclusion and Future Work**

We presented work on using argumentative structures and elements in identifying the quality of persuasive essays. We found that argumentative elements support the identification of good essays. Bad essays can be classified reliably using traditional features, indicating that these authors need to address issues such as spelling errors before improving on argumentative elements in their writing.

The next step would be to increase the data set size in order to solidify our findings. More data would also allow us to use more sophisticated machine learning methods. Additionally, we would like to incorporate a range of features previously used in the area of essays scoring, such as latent semantic analysis. Finally, we would like to have a closer look at the issue of metaphors in argumentative essays and their contribution to arguments and essay quality.

**Acknowledgements.** We would like to thank the German Institute for Educational Research and Educational Information (DIPF), where this work was carried out. Additionally, we would like to thank the reviewers for their helpful comments.

#### **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# Author Index

Ackermann, Markus 130 Batinić, Dolores 3 Benikova, Darina 171 Berger, Armin 232 Bourgonje, Peter 48, 180, 232 Budde, Klemens 146 Burkhardt, Felix 11 Claeser, Daniel 192 Dakota, Daniel 23 Degaetano-Ortlieb, Stefania 259 Deng, Jun 11 Eyben, Florian 11 Felske, Dennis 192 Fischer, Judith Victoria 276 Fraser, Alexander 81 Fricke, Rolf 232 Gärtner, Markus 284 Geiß, Johanna 115 Gerth, Jens 232 Gertz, Michael 115 Gründer-Fahrer, Sabine 199 Hahn, Uli 284 Haldenwang, Nils 208 Halleck, Fabian 146 He, Jing 232 Hellmann, Sebastian 130 Hemsen, Holmer 155 Hennig, Leonhard 155, 248 Hermann, Sibylle 284 Hübner, Marc 146

Ihler, Katrin 208 im Walde, Sabine Schulte 292 Kent, Samantha 192 Kirschenbaum, Amit 130 Klimek, Bettina 130 Kniephoff, Julian 208 König, Luca 232 Köper, Maximilian 292 Mieskes, Margot 301 Moreno-Schneider, Julián 180, 232 Nguyen, Kim-Anh 292 Quantz, Joachim 232 Räuchle, Sören 232 Rehm, Georg 48, 180, 216, 232 Rethmeier, Nils 146 Roller, Roland 146 Ruppenhofer, Josef 95 Schlaf, Antje 199 Schmidt, Danilo 146 Schmidt, Thomas 3 Schuller, Björn 11 Schwarzenberg, Robert 155 Sohl, Jessica 40 Spitz, Andreas 115 Srivastava, Ankit 48, 180, 232 Staeck, Oliver 146 Stiegelmayr, Andreas 301 Strötgen, Jannik 259 Sugisaki, Kyoko 62 Thater, Stefan 72 Thomas, Philippe 146, 248 Thomsen, Jan 232 Uszkoreit, Hans 146 Vornberger, Oliver 208

Wabnitz, David 232 Weber, Sabine 48 Weiss, Benjamin 11 Weissweiler, Leonie 81 Wiegand, Michael 95 Wojatzki, Michael 171

Wolf, Maximilian 95 Wustmann, Sebastian 199

Zesch, Torsten 171 Zinsmeister, Heike 40

Open Access This book is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this book are included in the book's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the book's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.


**Transition of Urban Morphology in the Mountainous Areas Since Early-Modern Times from the Perspective of Urban Historic Landscape—A GIS Tools and Historical Map Translation Approach**

**Danjie Shen \* and Shujing Dong**

School of Architecture and Urban Planning, Chongqing University, Chongqing 400030, China **\*** Correspondence<EMAIL_ADDRESS>
**Abstract:** Regenerating cities must blend modernization and heritage. Both urban morphology and Historic Urban Landscape (HUL) highlight historical processes and may assist in regeneration. Using Chongqing as the study example may further understand mountain cities, which are prevalent worldwide but seldom examined in morphology research. This study explores and organizes the historical modernization of Chongqing's parent city from early-modern times to the present day using a universal approach established in this research developed by the HUL perspective and research framework, Geographic information system (GIS), Depthmap tool, and historical map translation method. Large-scale modernization occurred prior to the 1980s, followed by more modest rehabilitation projects. The whole procedure is described by the phrase "Construction first, then planning, then transformation," which entails a "free growth" block structure at the outset, along with planning control. The study contributes the following: (1) Establishing a theoretical framework and research technique for the universal city based on historical sources and modern instruments; (2) Chongqing's future sustainable development and historical preservation depend in large part on figuring out the city's complicated modernization history; (3) The study of mountain cities may benefit from understanding the geographical development and spatial dynamic layering of Chongqing. (4) This study bridges the gap in time by going beyond the early modern period covered by the previous ones and into the post-statehood era (1949–2022).

**Keywords:** urban morphology; spatial dynamic layering; GIS; historical map translation; sustainable development; historical preservation; mountain city


*3.6. High-Resolution Mass Spectrometry (HR-MS) Experiments*

HR-MS and HR-MS/MS were performed on a Waters Xevo G2 QTOF mass spectrometer (Waters, Manchester, UK) with ESI in positive mode (ESI+) and in few cases in negative mode (ESI−). For both modes, the instrument parameters were as follows: for MS analysis, cone voltage 30 or 50 V, source temperature 110 ◦C, desolvatation temperature 300 ◦C, cone gas flow rate 20 L h−1, scan range *m*/*z* 50–1200; for MS/MS analysis, three collision energies were used 15, 25 and 35 V with the cone voltage maintained at 30 V or 50 V and the spectra were acquired with a mass precursor ion selection of 3 Da. All analyses were performed using the lockspray, which ensured accuracy and reproducibility. Leucine enkephalin (1 ng μL−1) introduced by a lockspray at 3 μL min−1 was used as the lockmass generating reference ions at *m*/*z* 556.2771 or 554.2615 in positive or negative mode respectively. The MassLynx software was used to calculate the most probable chemical formula and the theoretical exact mass of the molecular and fragment ions by comparison with their measured accurate ionic masses.


**6. Summary**

HIT represents a rare but potentially life threatening complication of heparin therapy. Its pathogenesis is based on an immunological reaction specifically the formation of antibodies directed against a complex of heparin and PF-4. Secondary exposure to heparin triggers a reaction leading to activation of platelets, their aggregation and adhesion to endothelium leading to formation of clots (80% in the venous system and 20% in arteries) and subsequent thrombotic events.

The definitive diagnosis of HIT requires sophisticated laboratory tests that may take a while to get results from. Therefore treatment should be initiated before final results are obtained based on high clinical suspicion. The most important principles of therapy include: discontinuation of any form of heparin therapy, treatment with indirect inhibitors of factor Xa or direct thrombin inhibitors and reversal of vitamin K antagonists if they were used previously. Additionally, transfusion of platelets should be avoided since it will only precipitate their activation and aggravate the clinical symptoms.