14 http://neo4j.com/docs/stable/configuration-io-examples.html

**UP 033 odijk odijk\_printer 2017/12/15 15:57 Page 236 #256**

**Figure 19.2:** The WhiteLab 2.0 data model for the Neo4j backend.

search and do allow for regular expression queries, but the total size of two nodes, a relationship and one property (128 bytes) is larger than a single node with one property. Nevertheless, if more elements have the same attribute and their nodes connect to the same property node, the total size quickly becomes less than when storing each element node with its own property (Figure 19.2, blue, full line). In practice this means that attribute values with a frequency lower than 3 are most e ciently stored as properties of the element node they describe, whereas higher frequency attribute values should be placed in their own node, which is then connected to the appropriate element nodes.

Compared to WhiteLab, the set of annotations in WhiteLab 2.0 has been extended to include phonetics. This includes the addition of token attributes regarding the token's position in the audio, as well as identication of the speaker at sentence level. Also, the Part-of-Speech attributes (head and features) are separated from the complete tag, allowing for more ne-grained analysis of these annotations.

In order to retain support for BlackLab, a new index tool has been added to BlackLab15 that enables indexing of multiple corpora, and a set of new BlackLab indexers has been developed specically for use with WhiteLab 2.0. With the current indexers and Importer, the size of the Neo4j database is approximately twice that of the equivalent BlackLab index.

### **19.3.4 Administration**

A new feature in WhiteLab 2.0 is the Admin interface. It allows the application manager to inspect and manage the metadata and Part-of-Speech tags across corpora. This functionality was added in light of known dierences in the tagsets used for SoNaR-500 and CGN, which can now be easily inspected. The Admin interface also allows control over the interface language and info page content (Section 19.4).

The types of corpora that WhiteLab 2.0 is designed to make accessible are mostly of a static nature, certainly so at the document level. Therefore, result sets for queries are not expected to change aer deployment. We have taken advantage of this fact by including an SQL database in the web application for user and query logging. The query logging is set up in such a way that no

15 https://github.com/Taalmonsters/BlackLab

duplicate queries are sent to the Neo4j database. For instance, if two users enter the same query within a short timeframe, the query is sent to the database only upon rst request. The second request will simply wait for the rst to nish and then access its results. Another request for the same data at a later time will also quickly return the previously stored result to the user. To keep a handle on the resources used, the web application includes some easy-to-set-up scheduled tasks, so-called Cron jobs, that run daily to remove queries that have timed out or have not been accessed in a while. A further advantage are the insights that the application manager may gather from the query statistics.

**UP 033 odijk odijk\_printer 2017/12/15 15:57 Page 237 #257**

### **19.3.5 Performance**

We test the performance of the WhiteLab 2.0 plugin for Neo4j compared to that of BlackLab-server 1.3 on the same dataset. Due to limitations of available hardware, the tests are performed on a subset of the complete OpenSoNaR+ data, namely, the entire CGN, plus the following SoNaR-500 collections: WR-P-E-E Newsletters, WR-P-E-F Press releases, WR-P-E-H Teletext pages, WR-P-E-J Wikipedia, WR-P-E-K Blogs, WR-P-P-B Books, WR-P-P-D Newsletters, WR-P-P-I Policy documents, WR-P-P-K Reports, and WR-U-E-A Chats16. The total size is around 83 million tokens. Tests are performed in a single dedicated server setup on a 12-core system with 64 Gb of RAM.

We test the response time of both backends to ve queries with increasing absolute hit counts ranging from approximately 7,500 to 250,000 hits. Each query is sent to the server 51 times over the command line. By bypassing the GUI, we disable the WhiteLab query caching for these tests. The rst call is discarded for both backends, as this warms up the index and takes considerably more time to complete. Figure 19.3 shows the average response time over the remaining 50 calls for each query. As is shown, BlackLab's performance is unhindered by increasing hit counts, where WhiteLab 2.0's response time increases almost linearly to the hit count. When we inspect the logs, we see that Neo4j spends most time on collecting the nodes that match the rst token in the CQL query, and on grouping results where necessary. Similar tests on queries of increasing complexity

**Figure 19.3:** Performance of WhiteLab 2.0 compared to BlackLab 1.3. Five queries with increasing result counts (in absolute hits) are each performed 51 times. The rst call is discarded, as this warms up the indexes. We report the average response time over the remaining 50 calls.

16 The SoNaR User Manual (available from http://ticclops.uvt.nl/SoNaR end-user documentation v.1.0.4.pdf) explains these codes.

as measured in n-gram size conrm that the initial node selection is the crux; the n-gram size has little to no eect on the response time. The delay in the grouping is particularly detrimental to queries for grouped documents, as these require two groupings: rst from hits to documents, and then into groups of documents. Overall, the tests show that there is still a lot to be done in terms of optimization of the WhiteLab 2.0 Neo4j plugin.

**UP 033 odijk odijk\_printer 2017/12/15 15:57 Page 238 #258**

# **19.4 OpenSoNaR+: User Functionality17**

We describe the WhiteLab 2.0 web interface as it is designed for OpenSoNaR+. It largely resembles the WhiteLab interface for OpenSoNaR, with added support for audio. A major advantage over the previous version is the addition of easy-to-congure interface translations. By default, the application comes with Dutch and English translations, but these may be extended by the application manager through the Admin interface. This interface also provides functionality to streamline metadata over dierent corpora. The metadata labels and values are listed including their coverage of indexed corpora, which provides a quick overview of possible similarities and discrepancies between corpus metadata. Dierent labels that refer to the same type of information can be grouped together under the same label. The translation functionality used for the interface components is also applied to the metadata labels.

The WhiteLab user by default lands on the **Search** page of OpenSoNaR+ when logging in. Next to this page we have the **Explore** page and the **Info** page. The Admin interface is hidden behind a login page and thus not available to regular users.

### **19.4.1 Info Page**

The Info page provides information about the system. It provides a rst-user manual which gives an overview of the main functionalities that OpenSoNaR+ oers. It also provides the user manuals of the SoNaR and CGN corpora, which oer in-depth information on the composition of both the contemporary written Dutch corpus and the spoken Dutch corpus. The system also provides a guided tour to its users, which gives the user a quick introduction to each page's uses and possibilities. Access to the guided tour is through the question mark button to the le in the top bar of the interface.

### **19.4.2 Explore**

The Explore page gives statistical and visual information about the corpus contents. It provides insight into the distribution of the texts available per genre and according to their provenance, which is basically whether they were collected in the Netherlands or in Flanders or are of unidentied or unidentiable provenance. The latter is the case for example with text materials obtained from the European Union or Wikipedia.

On the basis of metadata selections under the 'statistics' tab, the user can obtain custom frequency lists for particular subselections of the corpora. These are further discussed under Subsection 19.4.5. This page also aords access to n-gram (where n is 1 to 5) frequency lists derived from subcorpora for word forms, lemmata, POS tags and phonetic transcriptions.

Finally, the page aords direct access to a particular document in the incorporated collections on the basis of its le name. This should be a useful feature for possible research verication or replication when the particular document has been referred to in a research paper.

17 This section is an adaptation and extension of the initial description in Reynaert et al. 2014.

### **19.4.3 Search**

**UP 033 odijk odijk\_printer 2017/12/15 15:57 Page 239 #259**

The Search environment is to date the most elaborate. It provides four levels of access to the contents: Simple, Extended, Advanced and Expert.

The **Simple search** option provides Google-style, single query box access. Entering a search term here will instantiate a search over the full contents of the corpus. The search is for word forms, which may be phrases (n-grams), in which case insensitive matches are sought that respect the actual sequence of words. This latter functionality is also provided by the Extended and Advanced search environments.

The **Extended search** environment allows one to impose selection lters on the search eected. These lters are of two kinds. First, there are lters on the metadata. Second, there are lters on the lexical level, allowing one to search for either word forms, lemmata, POS tags and phonetic transcriptions for the spoken Dutch data.

The metadata lters are at rst hidden behind a bar visible above the actual lexical query elds. When the user wants to impose metadata lters the bar is expanded by a simple mouse click and the user is presented with a row consisting of three drop-down boxes. The middle box has just two options: 'is' or 'is not'. The le box gives access to all the metadata elds available in the corpus CMDI metadata les. The right box, upon selection of a particular metadata eld in the le box, dynamically expands with the list of available metadata contents, where applicable. Metadata lters can be stacked. Through a 'plus' button to the right of the query row, one may obtain further rows in each of which further restrictions on the query may be imposed. The metadata view shows the proportional and absolute (i.e. number of tokens) size of the dataset matching the currently selected lters. When a metadata lter is selected or updated, these numbers are automatically updated, allowing the user to quickly inspect subcorpus size prior to searching.

The metadata selection interface additionally provides the option of grouping the query results obtained by a range of features. For example, if one here selects the option of having the results presented by country of origin of the hit texts, one is not presented directly with the Key Words in Context (KWIC) list of results, but rather with a bar representation of the number of hits per country. One may then click on one of these bars and be presented with the KWIC list. This then gives the user the possibility to select one of these subsets and to further work on these as a new, independent query.

The lexical lters allow one to optionally perform case-sensitive searches for word forms, lemmata and/or phonetic transcriptions. POS tags can of course be searched too. When the search is for lemmata, all the word forms sharing the same lemma will be retrieved. For POS tag searches the user is presented with a drop-down list which presents a layperson's translation in plain language for the actual POS tags involved. Combinations of, for instance, word forms and POS searches are possible to direct the search for the word 'drink' (ibidem in English) towards the rst person singular of the present tense verb form, rather than its use as a noun.

For the **Advanced search** option we fully acknowledge to have emulated the elegant interface to CQL-query building as provided by the Swedish Sprakbanken ˚ 18 . Users are rst presented with a single box containing three query elds. By horizontally or vertically adding further boxes they may build quite complex queries without the need to know the query language behind them. Vertical boxes may be stacked with 'and' or 'or' conditions. These boxes give access not only to queries on full word forms (word 'is' or word 'is not') but also to words beginning with or containing or ending with a specic character string.  Users get to see the query they have built and have the option of further extending it, manually.

18 See 'Korp' at https://spraakbanken.gu.se/korp/#?lang=en&search tab=1

The **Expert search** requires knowledge of the query language incorporated in the system. It is CQL, the Corpus Query Language19. In its essence, this search option's limitations are dened mainly by the user's CQL prociency. However, to support the educational requirements of White-Lab 2.0, queries can be entered in one interface (e.g. Simple search) and viewed in another, more complex interface (e.g. Expert search) without rst having to execute the query. Using this functionality, students and laypeople can directly see the CQL query generated from their string query and actually increase their familiarity with the Corpus Query Language.

**UP 033 odijk odijk\_printer 2017/12/15 15:57 Page 240 #260**

### **19.4.4 Presentation of the Results**

Regardless of the search option one has chosen, by default, eventually a KWIC list of results is presented. A red button for each of the text snippets gives direct access to the full-text view of the document. There, moving the cursor over any of the words in the text, one gets to see a small window with the word form's unique ID, lemma and POS tag. Documents retrieved from CGN have a button for the whole text and buttons per sentence for calling up the appropriate sound recording. New tabs give access to the particular document's full metadata, to document specic statistics on size in terms of word tokens and types and derived measures. Finally, the user is presented with a visualisation of the token to POS tag distribution and the vocabulary growth curve.

A feature of the Extended and Advanced search options we have not seen in other corpus exploration environments is that multiple queries can be performed in one operation. This is facilitated by the fact that by clicking on the 'list' button to the right of the query boxes the user may eortlessly upload a pre-prepared list of query terms. Aer uploading, these query terms are converted by the system into actual, separate CQL queries which are visible in the query history. The user then has the option of having the output presented separately, per query, or mixed. If in the Advanced search environment a user uploads more than one query list, the system makes a combination of all the query terms in the lists. Given x terms in list A and y terms in list B, this results in x times y queries. If this is not what the user intended, then the user has the option of uploading a list of, for instance, word bigrams to be searched for in the Extended search environment.

### **19.4.5 Export of the Results**

Both the Explore and Search pages allow the users to export the results of their queries. This would be the frequency list built on the basis of the selections made, whether metadata-based, lexical, or indeed both. Or else, one may export the list of documents that were selected. What WhiteLab by design does not provide, is export of the full documents. This facility exceeds for the best part the IPR-agreements that were achieved with the text providers. However, the full corpora containing the full texts are freely obtainable for research purposes from the INT.

The query results are exported in various formats, including comma-separated lists suitable for loading in a spreadsheet. The format should be easily convertible to the specic formats required by statistical packages such as R20 or SPSS21 .

### **19.4.6 Query History**

An important new feature of the updated WhiteLab is that a user's query history is stored and is accessible to the user through an unobtrusive sea-green button in the lower le corner of the window.

19 A nice tutorial is at: http://cwb.sourceforge.net/files/CQP Tutorial/.

20 https://www.r-project.org/

21 https://www.ibm.com/marketplace/cloud/statistical-analysis-and-reporting/

The results of one's export actions are to be found here as part of the summary of each query one has undertaken.

**UP 033 odijk odijk\_printer 2017/12/15 15:57 Page 241 #261**

### **19.5 Performance and Availability**

### **19.5.1 Performance**

As far as technologies go, Neo4j is relatively young and still in active development. Since the start of the OpenSoNaR-CGN project, many new versions have already been released, including updates that will likely increase performance for WhiteLab 2.0 once implemented. This trend is expected to continue over the coming years. WhiteLab 2.0 is already set up to reap the benets of these advancements, while also being able to function with established technologies through BlackLab.

Currently, the query caching of the interface is resolved using an SQL database. We recognize that this can also be solved using a key-value store such as Redis. At the time of development of the current version of WhiteLab, use of Redis still imposed a lot of security risks and was not advised in production systems. However, recent developments have greatly improved Redis's security22, making it a feasible alternative that we will denitely consider. Moreover, BlackLab-server provides its own query caching.

Independent from external developments, we see a number of possibilities for improving performance of the Neo4j backend. Certainly, the application itself can be further optimized and streamlined. But most benet would likely be gained from decreasing the size of the Neo4j database, either through simplication of the data model, or separation of structure and content. The latter could be achieved, for instance, through a dual-database setup, where one database holds the document structures and the other the linguistic network. Another possibility we intend to investigate is storage of the annotations in an optimized string index such as the one used by word2vec (Mikolov et al., 2013), which reaches great speeds on huge collections of strings.

### **19.5.2 Availability**

All WhiteLab 2.0 components are released under the GNU Aero General Public License and are currently available at https://github.com/Taalmonsters/WhiteLab2.0. An installation manual for use with either the BlackLab or the Neo4j backend is provided.

### **19.6 Conclusion**

The distribution version of CGN requires 115GB in archived form and SoNaR-500 takes up 62.6GB. Unwieldy at best, and to all intents and purposes practically inaccessible to the average researcher. Though freely available for research, some unwary researchers were nastily surprised when trying to unpack SoNaR on common laptops running everyday soware. Reports of these mishaps prompted the original OpenSoNaR project proposal to be written.

Results of the rst project being well-received, OpenSoNaR-CGN followed suit. In relatively little time and on a modest budget with a small, but dedicated, team, we have managed to put OpenSoNaR+ – both corpora, text and sound – at everyone's ngertips.

We hope WhiteLab may serve researchers well. We denitely hope it will nd favour with new and existing corpus endeavours in the Low Countries and far beyond.

22 https://www.reddit.com/r/redis/comments/3zv85m/new security feature redis protected mode/

# **Acknowledgements**

**UP 033 odijk odijk\_printer 2017/12/15 15:57 Page 242 #262**

We gratefully acknowledge the feedback we received from our user group and the funding provided by CLARIN-NL under grant numbers CLARIN-NL-12-013 and CLARIN-NL-15-005. Martin Reynaert further acknowledges being funded by the new Dutch national CLARIN project CLARIAH and by NWO in project Nederlab. Finally we would like to thank the two anonymous reviewers of the prenal version of this chapter for their constructive feedback.

# **References**

Demonstrations, pages 124–128, Dublin, Ireland. Dublin City University and Association for Computational Linguistics.

**UP 033 odijk odijk\_printer 2017/12/15 15:57 Page 243 #263**

**UP 033 odijk odijk\_printer 2017/12/15 15:57 Page 244 #264**

# CHAPTER 20

**UP 033 odijk odijk\_printer 2017/12/15 15:57 Page 245 #265**

# **Creating Research Environments with BlackLab**

#### J. de Doesa , J. Niestadtb and K. Depuydtc

a INT, Leiden, Netherlands<EMAIL_ADDRESS>b INT, Leiden, Netherlands<EMAIL_ADDRESS>c INT, Leiden, Netherlands<EMAIL_ADDRESS>
### **ABSTRACT**

The BlackLab search engine for linguistically annotated corpora is a recurring element in several CLARIN and other recent search and retrieval projects. Besides the core search library, we have developed the BlackLab Server REST web service which makes it easy for computational linguists and programmers to write anything from quick analysis scripts to full-edged search interfaces, and the AutoSearch application which allows nontechnical linguistic researchers to index and search their own data.

This chapter describes the motivation for developing the BlackLab platform, how it has been used in actual research, and sketches future developments which will make it a more powerful tool for the creation of research environments.

### **20.1 Introduction: Why BlackLab and BlackLab Server?**

There are several excellent linguistic corpus search engines that support the creation of corpus retrieval systems: the Sketch Engine (Kilgarri et al., 2004) is a superb product, the Corpus Workbench (Evert and Hardie, 2011) is widely used to create search interfaces for corpora, cf. for instance (Borin et al., 2012; Nygaard et al., 2008), and there are more recent alternatives like Corpuscle (Meurer, 2012), and Poliqarp (Janus and Przepiorkowski, 2007). ´

Nevertheless, there were reasons to look for alternatives. In the context of the CLARIN and CLARIAH research infrastuctures, we need a versatile platform that supports the creation of research environments that can be either generic or tailored to specic needs. Of course, the search engine at the heart of such a platform should still be powerful, scalable, ecient and feature-rich, but other requirements are just as important: the core components should be easy to maintain and

#### **How to cite this book chapter:**

de Does, J, Niestadt, J and Depuydt, K. 2017. Creating Research Environments with BlackLab. In: Odijk, J and van Hessen, A. (eds.) CLARIN in the Low Countries, Pp. 245–257. London: Ubiquity Press. DOI: https://doi.org/10.5334/bbi.20. License: CC-BY 4.0

extend because of clear APIs and modular design, and, because of the simplicity of both the library and the server API, it should be easy to develop custom front ends and extensions.

**UP 033 odijk odijk\_printer 2017/12/15 15:57 Page 246 #266**

Our choice to develop a new corpus retrieval platform that uses Lucene as the underlying search engine has the advantage that we can prot not only from the active development of the Lucene core, but also from Lucene-based products like Solr and Elasticsearch to implement new features.

# **20.2 BlackLab, BlackLab Server and AutoSearch**

### **20.2.1 The Design of BlackLab**

We had the following objectives in mind while designing BlackLab:

## Extending the Basic Lucene Indexing and Retrieval Model

Lucene is at the heart of BlackLab. Each indexed document becomes a Lucene document, and metadata elds such as title and author become Lucene elds. The document content is indexed in a more sophisticated way: token and character positions are stored. This enables highlighting of search results in the original content.

BlackLab extends this basic mechanism in several ways:

1 http://proycon.github.io/folia/

2 The Corpus Workbench site has a great introduction to CQL: http://cwb.sourceforge.net/files/CQP Tutorial/ Note that BlackLab supports the most important features, but not yet all of the features.

**Forward index** For quickly displaying keyword-in-context (KWIC) views, sorting and grouping hits on context, and counting occurrences of terms in whole documents or in the vicinity of a set of hits, a specialized data structure called a forward index has been implemented. The forward index is really the complement to Lucene's reverse index: whereas Lucene answers questions of the form 'where in my documents does the word X occur?', the forward index is optimized to answer questions of the form 'what word occurs in document Y at position Z?'

**UP 033 odijk odijk\_printer 2017/12/15 15:57 Page 247 #267**

## **20.2.2 Features of BlackLab Server**

BlackLab Server was developed for two reasons: to provide a clean back end for the corpus front end, a large part of which is written in JavaScript, and to make it as easy as possible to carry out quick corpus analyses from any scripting language without compromising the speed of the Black-Lab Java code. BlackLab Server is a REST web service: it responds to URL requests in either JSON or XML format. It is implemented as a Java servlet that will run, for instance, in Apache Tomcat. It provides several dierent search modes, such as: search for occurrences of a word, search for documents containing a word, show a snippet of text from a document, or retrieve the full original document. In addition to sorting results, it also allows you to group results by many criteria, including the context of the hits found (e.g. the word to the le of the hit). Some important aspects of its design are:

### **20.2.3 AutoSearch**

For researchers who are not computational linguists or programmers, but would like to be able to quickly search their annotated texts, we have developed BlackLab AutoSearch. This application allows end users to simply upload text data in a supported format (today, FoLiA or TEI). It is then indexed on our servers, aer which it may be searched using Corpus Query Language.

If the user does not have FoLiA or TEI data yet, but rather text data in another format (e.g. Word, PDF, HTML or ePub), we have also developed OpenConvert3 , which allows users to convert their plain text data into FoLiA or TEI, and run it through a (simple) tagger/lemmatiser for Dutch. In the future, we would like to incorporate this functionality into AutoSearch, to streamline the process as much as possible.

### **20.2.4 Performance**

An elaborate comparison to other corpus retrieval systems is outside the scope of this chapter. Benchmarking would be easier if standard query and datasets were available for this purpose. Nevertheless, to obtain an indication of the performance level, we tagged and lemmatized the DUTCH PARLIAMENTARY PROCEEDINGS 1814-2012 dataset, consisting of about 700 million tokens,

3 Both AutoSearch and OpenConvert can be found in our CLARIN portal at https://portal.clarin.inl.nl/. OpenConvert is also available on GitHub: https://github.com/INL/OpenConvert. AutoSearch should soon be available under https://github.com/INL/ as well; send us a message if you are interested.

**Table 20.1:** Query times on Parliamentary Proceedings Corpus.

available from the Dutch Data Archiving and Networked Services (DANS)4 , and indexed the data with BlackLab and with CWB5 .

**UP 033 odijk odijk\_printer 2017/12/15 15:57 Page 248 #268**

The performance on some example queries is illustrated in Table 20.1. We found the systems to be roughly comparable in performance, with some queries running faster in BlackLack (command line query tool) and others in CWB (command line tool cqp).

# **20.3 Using BlackLab and BlackLab Server to build your own research environment**

This hands-on section explains how to use BlackLab and BlackLab server to build simple applications.

# **20.3.1 Indexing Data with BlackLab**

BlackLab can index any textual data, but we have focused on using it with XML. Several XML formats (including popular corpus formats FoLiA and TEI6 ) are supported out-of-the-box, and it is easy to create custom versions of indexers or add support for a new XML format.

XML corpus formats generally have an XML tag for each word, with tags or attributes for the dierent properties of the word (such as lemma and part of speech). BlackLab indexes each property in its own Lucene eld, and automatically combines these elds while searching, so you can construct complex queries that specify constraints on dierent properties as needed.

# **20.3.2 Using BlackLab Directly from Java**

Before introducing the web service, we start with the BlackLab Java API, so that we can compare the two. Here is some example code that uses the BlackLab API directly to search for a Corpus Query Language query:

```
// Convert word array to string
String words ( List < String > words ) {
    return StringUtil . join ( words , " " ) ;
}
final static File PATH_TO_MY_CORPUS = new File ("/ tmp / bla /");
// Search and show hits
public void search ( String cqlQuery ) {
```
4 https://easy.dans.knaw.nl/ui/datasets/id/easy-dataset:51640

5 BlackLab 1.3.1, cwb 3.0.0, 12-core Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz, 128G ram.

6 http://www.tei-c.org/

```
try ( Searcher searcher = Searcher . open ( PATH_TO_MY_CORPUS )) {
         TextPattern tp = CorpusQueryLanguageParser . parse ( cqlQuery ) ;
         Hits hits = searcher . find ( tp );
         for ( Hit hit : hits ) {
              Kwic kwic = hits . getKwic ( hit );
              Document document = searcher . document ( hit . doc );
              String title = document . get (" title " ) ;
              System . out . println ( words ( kwic . getLeft ( " word ")) + " ["
                       + words ( kwic . getMatch ( " word ")) + "] "
                       + words ( kwic . getRight ( " word ")) + " (" + title + ")" );
         }
    } catch ( Exception e ) {
         throw new RuntimeException ( e ) ;
    }
}
```
**UP 033 odijk odijk\_printer 2017/12/15 15:57 Page 249 #269**

As we shall see below, using BlackLab Server results in very similar code.7

### **20.3.3 BlackLab Server**

As stated, BlackLab Server allows you to use BlackLab from any programming language, and we will give two examples of this here.

### <IP_ADDRESS> A Simple Example

Here is a simple Python example of searching a corpus for a CQL pattern ([pos="a.\*"] "fox"), i.e. the word 'fox' preceded by an adjective and displaying a simple textual KWIC view with document titles.

```
import urllib
import json
def words ( context ):
    " " " Convert word array to string . " " "
    return " ". join ( context [' word '])
def search ( cqlQuery ):
    " " " Search and show hits . "" "
    url = " http :// example . com / blacklab / mycorpus / hits ? patt = "
              + urllib . quote_plus ( cqlQuery )
    with ( urllib . open ( url ) ) as f:
         response = json . loads (f . read () )
    hits = response [ ' hits ']
    docs = response [ ' doc - infos ']
    for hit in hits :
         # Show the document title and hit information
         doc = docs [ hit [ ' doc - pid ']]
         print words ( hit [ ' left ']) + " [" + words ( hit [' match ']) + " ] " +
                   words ( hit [ ' right ']) + " ( " + doc [' title '] + ")"
# " Main program "
search ( '[ pos ="a .* "] " fox " ')
```
We have translated this basic example into other languages as well (including JavaScript, R, PHP, Perl, C# and Ruby). These may be found online8 .

7 The complete Java BlackLab API documentation can be found at http://inl.github.io/BlackLab/apidocs/.

8 http://github.com/INL/BlackLab-server/wiki/Using-BlackLab-Server-from-different-languages

### <IP_ADDRESS> Slightly More Complex BlackLab Server Example

**UP 033 odijk odijk\_printer 2017/12/15 15:57 Page 250 #270**

This example draws bar charts of the collocations of certain words in some author's works.

To start, here is a simple HTML page: just a search form and a div to render our chart to. It includes jQuery (for convenience), Google Charts (for drawing the chart) and our own JavaScript le, blacklab-server.js.

```
< html >
  < head >
     < script type =" text / javascript " src = " jquery . js " ></ script >
     < script type =" text / javascript " src = " https :// www . google . com / jsapi " ></ script >
     < script type =" text / javascript " src = " blacklab - server . js " > </ script >
  </ head >
  < body >
     <h1 > Zen and the Art of Collocations </ h1 >
     < form onsubmit =" search () ; return false ;" >
       Show words that occur within 5 words of < input id =' word ' type =' text ' />
       < input type =' submit ' value =' Update ' />
     </ form >
     < div id = " chart " style =" width : 900 px ; height : 500 px ;" > </ div >
  </ body >
</ html >
```
The next example, blacklab-server.js, sends a request to the server, counts context words in the response, and draws the chart. To be precise, when the form is submitted, the search function is called, which builds a CQL query, glues it to a URL, and retrieves that URL. When the server responds, the handleResults function iterates over the hits in the response object, counting words in the le and right contexts, and renders the resulting word frequency data using Google Charts.

```
google . load (" visualization " , " 1 " , { packages :[ " corechart " ]}) ;
var whichContext = " word " ; // which collocations ? e.g. word / lemma / pos /..
function search () {
    var cqlQuery = '" ' + jQuery ( "# word "). val () + '" ';
    var url = " http :// example . com / blacklab / mycorpus / hits ? filter = author : pirsig &"
              + " context =5& patt =" + encodeURIComponent ( cqlQuery ) ;
    jQuery . get ( url , handleResponse ) ; // AJAX call to BlackLab Server
}
function handleResponse ( response ) {
    // Count context words for each hit
    var wordFreq = {};
    jQuery . each ( response [' hits '] , function ( index , hit ) {
         countWords ( hit [' left '] , wordFreq ); // left context
         countWords ( hit [' right '] , wordFreq ); // right context
    }) ;
    // Draw Google Chart
    var data = new google . visualization . DataTable () ;
    data . addColumn (' string ' , 'Word ') ;
    data . addColumn (' number ' , ' Frequency ') ;
    jQuery . each ( wordFreq , function ( word , freq ) {
         data . addRows ([[ word , freq ]]) ;
    }) ;
    data . sort ([{ column : 1, desc : true }]) ;
    var chart = new google . visualization . BarChart ( jQuery ( '# chart ') . get (0) );
    chart . draw ( data );
}
function countWords ( context , wordFreq ) {
    jQuery . each ( context [ whichContext ], function ( index , word ) {
```

```
if (! wordFreq [ word ]) wordFreq [ word ] = 0;
          wordFreq [ word ]++;
     }) ;
}
```
# **20.4 Using BlackLab and BlackLab Server for Linguistic Research**

**UP 033 odijk odijk\_printer 2017/12/15 15:57 Page 251 #271**

We summarize how BlackLab has been used for research, and analyze the requirements that can be deduced from these experiences. Finally, a use case based on the Letters as Loot corpus illustrates how a (small) research environment created with BlackLab server can support historical linguistic research.

### **20.4.1 Projects Using BlackLab**

This corpus, to which metadata from the research programme's database were added, was lemmatised, PoS-tagged and provided with elaborate search facilities by the Institute for Dutch Lexicology13 .

**Early Modern English corpora at Northwestern University** Phil Burns of Northwestern University has created an experimental corpus search site14 that is powered by BlackLab. At

9 http://www.impact-project.eu/, http://www.digitisation.eu

10 http://gysseling.corpus.taalbanknederlands.inl.nl/

11 http://corpushedendaagsnederlands.inl.nl/

12 http://opensonar.clarin.inl.nl

13 http://brievenalsbuit.inl.nl

14 http://devadorner.northwestern.edu/corpussearch/

present the corpus of Shakespeare's plays, the TCP ECCO corpus (Eighteenth Century Collections Online), the TCP Evans corpus (Evans Early American Imprints), and the Shakespeare His Contemporaries corpus (Early Modern English Drama) are publicly searchable. Martin Mueller (Professor of English & Classics) has written about his experiences with the application (Mueller, 2013).

**UP 033 odijk odijk\_printer 2017/12/15 15:57 Page 252 #272**

# **20.4.2 Research and education based on BlackLab corpora**

The following research uses the BlackLab query engine:

We also mention some research that makes use of the corpora mentioned in another way, sometimes simply because the research was performed before the corpus was online. We list this type of research because of the requirements it poses:

# <IP_ADDRESS> Requirements emerging from these experiences

**Teaching sessions** Elaborate corpus retrieval sessions with the OpenSoNaR user interface at Utrecht University in courses given by Jan Odijk yielded, among others, the following requirements:

15 https://prezi.com/ofiy5m-a6vbe/na-and-naar/?utm campaign=share&utm medium=copy

16 http://nederl.blogspot.nl/2015/11/21-22-january-2016-effects-of.html?m=1

17 This refers to a CQL feature not yet implemented by BlackLab.

18 This can be solved easily by a different indexing scheme.

**Export of query results** 1. Tab-separated export format: separate all elds by tabs; options for simple and extended part-of-speech export; options to export metadata; and 2. Export of CMDI metadata describing the result export: including query, lters, grouping criteria, number of documents/hits/groups in results. This should be uploadable to the application to reproduce the result.

**UP 033 odijk odijk\_printer 2017/12/15 15:57 Page 253 #273**

**Research experiences** Out of the above-mentioned research, the following requirements can be deduced:

In most of these cases, we can argue that the use of BlackLab Server could make it very easy to implement the requested features. For some features, extensions to BlackLab server are necessary, but mostly of a rather simple nature, e.g. an option to return all relevant subcorpus sizes corresponding to metadata grouping criteria.

### **20.4.3 Use case: signs and sounds in the Letters as Loot corpus**

For this case study, we have developed a small research environment to start exploring how we can support the kind of research that has been conducted in (Rutten and Van der Wal, 2014), most of it before the corpus appeared online. To this end, we compare some results from chapter 2 ('Sounds and signs - From local to supralocal usage') of (Rutten and Van der Wal, 2014) to results obtained from querying the corpus and analyzing the query results.

It is obvious that automatic retrieval from corpora cannot replace careful manual analysis in many cases. For instance, the analysis of the orthographical representation of etymologically distinct long e's20 requires information which is simply not present in the annotated corpus.

19 https://github.com/INL/BlackLab-server/wiki/Using-BlackLab-Server-from-different-languages#r

20 Cf. section 2.4.5 in (Rutten and Van der Wal, 2014): 'Many Dutch dialects, the southern ones in particular, maintain the phonological difference between lengthened *e¯* out of originally short vowels in open syllables, and *ˆe* out of the West Germanic diphtong *\*ai*'.

<IP_ADDRESS> H-Dropping in the 17th Century: First Case Study

**UP 033 odijk odijk\_printer 2017/12/15 15:57 Page 254 #274**

Many dialects from the south and the south-west of the Dutch language area are characterised by the absence of the phoneme h, as in, and instead of hand. In the texts, this may result in deletion or prothesis of h.

As we have seen, contrasting two result sets is a desideratum emerging from corpus research. As a test case, we have used the BlackLab Server API and Google Charts (in a similar vein to the simple concordance example (section <IP_ADDRESS>) to implement this functionality in a simple way (cf. Figure 20.1). Our environment will consist of a search and grouping form, a bar chart, and a simple concordance view.

In the example, we contrast the number of hits of the corpus query

[ lemma != "h .\* "] [ lemma ="h .\* " & word = "[ aeo ].\* " ]

(indicating h-dropping21) to the query specifying orthographic expression of h:

**Figure 20.1:** H-dropping 1: absolute frequencies.

21 The restriction on the previous word is there to avoid situations like 'hier om', where both word parts have the lemma 'hierom'.

**UP 033 odijk odijk\_printer 2017/12/15 15:57 Page 255 #275**

**Figure 20.2:** H-dropping 2: relative frequencies.

The result (cf. also Figure 20.2) indicates clearly that h-dropping is a southern (Zeeland and Western Flanders) phenomenon, and is more predominant in the 17th than in the 18th century. Comparing to the manual results (133 cases of h-dropping in the 17th-century Zeeland corpus), we should note that counts are not identical because we are using a larger corpus and the selection criteria are dierent, but the observed tendency is in agreement with the Rutten-Van der Wal results.

Summarizing, we are able to reproduce this type of analysis comparatively easily. The fact that our query results are, with respect to the phenomenon we are looking for, neither complete nor quite clean, does not impair their usefulness as a quick way to analyze a tendency. For more thorough analysis, one would need the result categorization feature discussed above.

### <IP_ADDRESS> Loss of Final -e

One of the most salient changes in the history of Dutch is apocope of nal schwa, a linguistic phenomenon that also occurred in English and to a lesser extent in German. By the 17th century, many dialects and particularly Holland dialects had a high proportion of schwa-less forms.

The change shows prominently in rst person singular forms of verbs. In nouns, forms with nal -e are hard to distinguish from plurals with loss of nal n.

Hence:

```
[ lemma =" ik " & word = " .*[ ck ].* " ]
[ pos = " VRB " & word =" .* e" & word != " .*[ td ] e "
              & lemma != " doen | gaan | staan | slaan | zien | zijn " ]
[ pos != " VRB "]
```
is a way to nd word forms with nal e, and

```
[ lemma =" ik " & word = " .*[ ck ].* " ]
[ pos = " VRB " & word != " .* e "
               & lemma != " doen | gaan | staan | slaan | zien | zijn " ]
[ pos != " VRB "]
```
nds their e-less counterparts, cf. Figure 20.3.

### **20.5 Conclusions and Future Plans**

We are still improving BlackLab and its related projects: scaling BlackLab up to ever larger corpora, making sure even complex searches remain fast, and adding useful features. We are interested in looking at distributed search and multi-corpus search, both for speeding things up and keeping larger datasets manageable. We are considering to integrate BlackLab with Solr or Elasticsearch to enable this functionality. Another feature on our wishlist is the ability to search tree- and graph-like

**UP 033 odijk odijk\_printer 2017/12/15 15:57 Page 256 #276**

**Figure 20.3:** Dropping of nal e in rst person singular.

structures (e.g. treebanks). We will look at both of these desirable features as part of CLARIAH. Other CLARIAH objectives that t in very well with the requirements that emerge from the research discussed in this chapter are so-called 'Chaining Search' (serial combination of searches in heterogeneous datasets, e.g. a corpus and a lexicon) and adding comprehensive support for dealing with subcorpora, included those dened by document metadata uploaded by researchers.

In the near future, we would like to create a library for talking to BlackLab Server from one or more popular programming languages, which could abstract away the last few technical details, making things even easier. Support for statistical explorations and visualizations should be enhanced, cf. for instance (Speelman, 2014). As has been discussed before, the aim is not to develop a monolithic application that satises all requirements, but rather the development of a platform that supports quick development of the analysis scripts and user interface elements that are necessary for a research use case.

### **References**

Nygaard, L., Priestley, J., Nøklestad, A. and Johannessen, J. B. (2008). Glossa: a multilingual, multimodal, congurable user interface., LREC, European Language Resources Association.

**UP 033 odijk odijk\_printer 2017/12/15 15:57 Page 257 #277**

**UP 033 odijk odijk\_printer 2017/12/15 15:57 Page 258 #278**

# CHAPTER 21

**UP 033 odijk odijk\_printer 2017/12/15 15:57 Page 259 #279**

# **Beyond Counting Syntactic Hits**

Erwin R. Komen

Radboud University Nijmegen, SIL-International<EMAIL_ADDRESS>
### **ABSTRACT**

Linguists who would like to make use of the increasing number of syntactically annotated text corpora in their research can use existing tools to nd and count instances of the syntactic constructions they are interested in. Soware supporting linguists in their work should also make it possible to build databases of search results where each hit is accompanied by a number of calculated (or manually addable) features. The stand-alone CorpusStudio program is able to provide this help, since it allows queries and feature calculations to be dened in the XQuery language. The web application of CorpusStudio, which is still under development, aims to have comparable functionality but with an easier accessibility. The main aim of this chapter is to demonstrate why soware should go beyond counting syntactic hits.

**Keywords**: syntax, corpus research, XQuery

### **21.1 Introduction**

A linguist who is interested in studying a particular syntactic construction in a language can do so by manually or programmatically looking through a number of texts in that language. It is the availability of syntactically annotated texts that makes this latter programmatic approach possible.

There are quite a number of programs and even web applications linguists can use to nd instances of the syntactic construction they are interested in.1 Studies conducted by linguists, however, involve more than locating constructions that satisfy particular conditions. Two other important aspects of a study are: (a) keeping a number of related searches together in a search

#### **How to cite this book chapter:**

1 Some of these programs are mentioned later on in this article.

Komen, E. R. 2017. Beyond Counting Syntactic Hits. In: Odijk, J and van Hessen, A. (eds.) CLARIN in the Low Countries, Pp. 259–268. London: Ubiquity Press. DOI: https://doi.org/10.5334/bbi.21. License: CC-BY 4.0

'project' that can be stored and retrieved to improve replicability, and (b) annotating search results automatically or semi-automatically with information that can be gleaned from the search hits. While the latter activity is an integral part of corpus linguists' everyday research, little support in terms of soware is available.

**UP 033 odijk odijk\_printer 2017/12/15 15:57 Page 260 #280**

This chapter discusses and exemplies the kind of facilities beyond those for counting syntactic hits that linguists would greatly appreciate in syntactic corpus research programs. The observations discussed are based on experience with the CorpusStudio and Cesax programs, which have so far been used in historical linguistics, second language acquisition and information structure research for Indo-European (Dutch, English, Welsh) as well as Caucasian (Chechen, Lak, Lezgi) languages (Komen 2014; Komen et al. 2014; Los and Dreschler 2012; van Vuuren 2013). The CorpusStudio application allows researchers to formulate and execute syntactic searches, store them in a 'Corpus Research Project', and annotate the search results with features that are determined programmatically.

# **21.2 The Linguist**

I would like to underscore the idea that linguists want to do more than nding syntactic constructions by considering what kinds of questions linguists ask when studying the syntax of a particular language. Linguistics is a broad research area, but I would like to focus on the research on syntax and information structure where annotated corpora are used. The important questions that researchers in this area ask are summarised in (1):

	- b. How does the occurrence of construction 'x' depend on genre, dialect or author, and how did the construction develop over time?

Finding instances of construction 'x' and counting them in a particular corpus is a good rst step towards answering these questions, but more should and could be done. Let me illustrate this with a real-life research question. Consider the examples from the 'standard' conditional construction in Dutch in (2a) and the alternative conditional inversion in (2b).

	- want dan moet er een projectvoorstel geschreven worden. because then must there a project.proposal written become

2 I use the term 'construction' here to denote a constellation of syntactic units. Any construction in this sense can be defined by making use of hierarchy and linear order of units that are identified by syntactic labels, possibly together with limitations on the content of these units.

'It is only when the partners have been found, that the matter actually starts. That's when the project proposal needs to be written.' **[fn000056:0147]**

**UP 033 odijk odijk\_printer 2017/12/15 15:57 Page 261 #281**

Suppose that linguists want to investigate the occurrence of the conditional inversion as opposed to the standard if-then conditional: they would at least want to know the numbers, so that they can gure out whether one of the two constructions is more or less exceptional. The numbers can be found by searching through syntactically annotated texts. Tools that facilitate syntactic searches are, for instance, the web applications PaQu3 (Parse and Query; see chapter 23) and GrETEL4 (see chapter 22) as well as the Windows version of CorpusStudio.5 All three search engines handle the corpus of Dutch texts in which the examples above occur: the Corpus of spoken Dutch (Oostdijk et al., 2002).6

It should be obvious from the examples in (2) that the two conditional syntactic constructions dier.7 If linguists want to nd all relevant results, they would probably need to write two dierent queries. Even if the researchers' focus is not on the 'standard' conditional, they would want to have the number of their occurrences for the sake of comparison. The two dierent queries do, however, belong to the same 'research project', which is why it would be of great help for linguist-users of the search soware to have these queries stored together – and they could do with some metadata too, identifying what the goal of the queries is, for instance. One possibility to reach this goal would be to keep searches and documentation in dierent les, but store them in a single project directory. This is a good approach, but keeping all relevant information together in one structured le (e.g. in XML format) makes it even more transparent, prevents potential errors and promotes clarity.

In line with the general research question in (1), linguists would like to know under which circumstances the conditional inversion occurs. They want to know whether its occurrence depends on linguistic factors, as in (1a), extra-linguistic factors, as in (1b), or a combination of the two. Table 21.1 identies a number of linguistic and extra-linguistic features that linguists would probably want to have for each hit.

How do linguists investigating the conditional inversion enrich their list of hits with the information they need? They could look through all the hits identied by a syntactic search program, and then nd all the relevant information for each of the hits manually, by checking the texts. But such an approach is error-prone, and, if there are no other reasons to check the texts manually, should be avoided. As for a programmatic approach, a few authors have suggested that XQuery could be used to extract the information that is required (Bouma, 2008; Bouma and Kloosterman, 2007; Yao and Bouma, 2010). The XQuery language is well suited to this task, since it allows the user to work with variables and functions (Boag et al., 2010). The language would, however, form an obstacle for linguists who are less familiar with computer languages. Applications that support XQuery, then, should consider supporting easier query denition methods for some, while allowing the use of XQuery's fuller capabilities for others.

Suppose, now, that the features mentioned in Table 21.1 have been determined for each of the hits. This gives the linguists basic data to do their research. They would be much helped if it were possible to divide the results into groups, the categories of which depend on the features that are

3 http://portal.clarin.nl/node/4182

4 http://nederbooms.ccl.kuleuven.be/eng/

5 http://erwinkomen.ruhosting.nl/software/CorpusStudio

6 Other systems that facilitate syntactic search queries are outside the scope of this chapter. Among them is the CLARIN-developed PML Tree Query web application, which aims for dependency treebanks and makes use of the PML query language (M´ırovsky et al., 2010) ´

7 No numerical results of the searches discussed here are given, since the focus of this chapter is not on this particular syntactic phenomenon, but on the question of how best to help a linguist wanting to research this and similar phenomena.

**UP 033 odijk odijk\_printer 2017/12/15 15:57 Page 262 #282**

**Table 21.1:** Features that could be relevant for choosing a conditional inversion.

calculated. That would give them a fast way to check the hypotheses that underlie the determination of the features in the rst place.

It would also be nice if they could divide their search into two parts. In a rst step, they could rst look for instances of the conditional inversion and the standard conditional, enrich them with the features listed in Table 21.1, and store them in some kind of database. They could manually check and adapt features such as 'ParaPosition' and 'FirstStatus', since these may not be determinable automatically with enough accuracy. They would need to have access to the hits in their context at this point.

The next step would be to formulate and test hypotheses that determine the choice between a standard conditional and a conditional inversion. This step would require to take the data in the result database from the previous step as input. It would be quite natural to implement this step by using the same machinery as in the previous step.

Once all of this has been done, the linguists have quite likely reached a point where they want to make use of programs such as R or SPSS to test statistical models of their hypotheses. The corpus research soware should allow the data to be exported in such a way that it can be used by statistics programs.

The facilities that corpus research soware should provide to help linguists address the kind of research questions in (1) are summed up in (3).

	- b. Provide gures that allow for the calculation of relative frequencies: the number of words, clauses, and texts that have been searched.8
	- c. Store the search results separately, so that features can be added to them.
	- d. Calculate required features automatically as much as possible.

8 Calculation of relative frequencies is left to the linguist, since the point of reference by which absolute frequencies should be divided may be taken differently depending on the purpose.

**UP 033 odijk odijk\_printer 2017/12/15 15:57 Page 263 #283**

Facility (3a) looks for and nds instances of the construction, and (3b) adds information to allow for a good quantitative study. The facilities in (3c–e) allow researchers to equip each 'hit' with as many features as are needed to help answer the research question. Facilities (3f–g) help provide more insight into how the results are divided in terms of aspects of the data itself or the metadata. Facility (3i) promotes the exchange of research projects and contributes to replicability. Facility (3c) allows for the process in (3a–h) to be divided into two parts: one where a database with hit-feature combinations is created, and one where the results in this database are divided into adjustable groups. Facility (3k) provides the connection with a possible next step: a statistical analysis.

### **21.3 Current Soware**

Soware that addresses points (3a,g) partly or completely has been made or continues to be made. The programs produced or enhanced for CLARIN-NL and CLARIN Flanders are no exception. A consortium of organisations and universities developed the Corpus Hedendaags Nederlands tool and later the OpenSONAR tool (Oostdijk et al., 2002; Reynaert et al., 2014).9 The Nederlab web application provides access to a huge (and growing) amount of Dutch texts (Brugman et al., 2016).10 All interfaces address (3a,g), some address (3f) partly, but none of these currently feature syntactic searches.

### **21.3.1 Web-based CLARIN Tools for Syntactic Research**

Two tools that have been supported by CLARIN that do allow for some kind of syntactic search are PaQu and GrETEL. PaQu has been developed by the University of Groningen.11 It not only incorporates online access to the Alpino parser of Dutch, but also provides search interfaces that allow the user to dene queries in XPath. Satisfying facility (3a), search results can be downloaded and are accompanied by some metadata.

The GrETEL tool allows searches in a number of dierent Dutch corpora as well as in Afrikaans corpora.12 Its user interface vastly diers from that of PaQu: searches are formed on the basis of a real-life example provided by the linguist. This means that researchers do not need to have indepth knowledge of what goes on inside the search engine. Augustinus et al. (2012) explain that their search engine uses XPath for the actual searches. The XPath code produced by GrETEL can, in fact, be used without changes in the PaQu web interface. The GrETEL application addresses point (3a), it allows the downloading of all the hits, and it has an option that provides a table with the counts divided per treebank; this table partly addresses facilities (3b,g).

9 opensonar.clarin.inl.nl and opensonar-cgn.science.ru.nl

10 http://www.nederlab.nl

11 http://www.let.rug.nl/alfa/paqu

12 http://nederbooms.ccl.kuleuven.be/eng/gretel

### **21.3.2 Windows-based CorpusStudio and Cesax**

**UP 033 odijk odijk\_printer 2017/12/15 15:57 Page 264 #284**

Two Windows-based programs combine into a set of tools that address most of the ambitious goals dened in (3): CorpusStudio and Cesax (Komen et al., 2013). Figure 21.1 shows how the programs cooperate.

The CorpusStudio program works with Corpus Research Projects (CRPs), XML denitions of queries, and metadata that together describe a research project. It allows the dening of searches in XQuery, which means that users can dene variables and functions and use these in their queries. CorpusStudio works on XML text corpora that are located on the user's computer, addressing (3a) fully. The search results it provides contain the total number of words and sentences of the texts being searched, allowing for (3b), the calculation of relative frequencies. Dividing the results in a data-dependent way – (3f) – is possible through a CorpusStudio-specic built-in XQuery function. Division of the results on the basis of metadata is only possible to a limited extent, so point (3g) is addressed only partly. The results can be turned into a separate XML database, and each 'hit' can be accompanied by user-denable features, addressing (3c). The extensive capabilities of the XQuery language, and the fact that it allows for user-dened functions in particular, facilitate calculation of such hit-dependent features in a comprehensive but relatively user-friendly way, as per (3d). The Cesax program allows for working with the kinds of result databases produced by CorpusStudio, so that the features can be adapted manually as per (3e). Points (3h) and (3j) are also taken care of by CorpusStudio and Cesax respectively. And where GrETEL oers an example-based denition of queries, Cesax and CorpusStudio contain a 'query wizard' that allows users to base a query on key elements of an example sentence in the corpus. Keeping queries, feature calculations, and metadata together in one research project, as indicated by (3i), is addressed fully by CorpusStudio (this was actually one of the main reasons to write the program in the rst place).

The stand-alone version of CorpusStudio does, unfortunately, come with a number of shortcomings. It is platform-dependent, since it only works on Windows. Its speed depends very much on the characteristics of the computer on which it is running, but it is not very fast. And while CorpusStudio could be adapted to work with XML texts in the FoLiA format, this is not facilitated

**Figure 21.1:** Cooperation between CorpusStudio and Cesax.

directly.13 A disadvantage related to its nature as a stand-alone program is the fact that a copy of the corpus to be researched needs to be held on each user's own machine.14 Where text corpora are being adapted, one may quickly lose track of where the most up-to-date version is located. Most of these disadvantages are alleviated in the web-version of CorpusStudio.

**UP 033 odijk odijk\_printer 2017/12/15 15:57 Page 265 #285**

# **21.4 The Web Application**

The stand-alone CorpusStudio Windows program has partly been re-written as a web application.15 The key components of the web application are shown in Figure 21.2.16

The core of the application is the 'Query Executor', a Java application that accepts a Corpus Research Project and executes the XQuery code from that project on a corpus of XML texts (in the FoLiA or the TEI-Psdx format).17 The CrpxProcessor divides the query execution workload over the available processors; the more processors, the faster the query execution. The CrpxProcessor can be run as a stand-alone application, but it is used as part of a web service within the CorpusStudio web application: the /crpp search service.

**Figure 21.2:** Principal components of the CorpusStudio web application.

13 Texts in the FoLiA format can be converted to the TEI-Psdx format in Cesax and then processed.

14 This is a particular shortcoming of CorpusStudio, not of stand-alone programmes as such. A reviewer pointed out that the Dact programme, for instance, is a stand-alone cross-platform application that supports working on remote corpora with remote parsing servers (van Noord et al., 2013). See http://rug-compling.github.io/dact/

15 http://www.clarin.nl/node/2095

16 The source code of the application is available at https://github.com/ErwinKomen.

17 A number of other formats can be converted into FoLiA or Psdx through the Cesax programme.

**UP 033 odijk odijk\_printer 2017/12/15 15:57 Page 266 #286**

**Figure 21.3:** Query input wizard.

The user of the CorpusStudio web application works with the /crpstudio service; this provides the interface between the user on the one hand, and the server on the other hand. The server contains the syntactically annotated XML text corpora that can be searched, the user's Corpus Research Projects (.crpx les), the user's search results and possibly the user's result databases. The /crpstudio service allows for the denition of the information stored in the Corpus Research Projects: the metadata of the project (Metadata Editor); the XQuery variables, denitions and queries (Denition and Query Editor); the hierarchy between the queries (Constructor Editor); and the features that need to be calculated if the output is a database (Database feature editor).

The 'corpora' part of the /crpstudio service lists the corpora that are available in the web application (the Corpus Viewer) and allows dening metadata-dependent result groupings. The 'dbases' part of the service makes interaction with the result databases possible. Once a research project has been executed, its results are available in the result viewer, which also allows downloading them.

The version of the CorpusStudio web application that has been delivered at the end of 2015 still suers a number of limitations compared to its stand-alone Windows counterpart; there are, for instance, limitations on metadata-dependent grouping of results and on working with result databases The implementation of a query wizard has started in 2016 and consists of two phases: (1) a query input wizard that allows easy input of queries, which are subsequently translated into XQuery, and (2) a system level that forms a shell around XQuery, allowing users to dene and adapt queries without the need for them to know any XQuery (queries are translated into XQuery only just before execution).

The query input wizard is currently being implemented, and Figure 21.3 gives an idea of its intermediate state. The main idea is that the user can: (1) name and identify constituents and their relations towards one another, (2) stipulate additional relations between the named constituents, and (3) formulate feature denitions on top of the standard ones (the latter of which are the labels of each of the identied constituents, and the text of these constituents). More information on the current status of the program, including the second phase (of easy access to XQuery), will be made available online.18

18 See the 'About' section of the web application: http://www.clarin.nl/node/2095.

Most importantly, the program has extended the CLARIN infrastructure with a syntactic research tool that allows interested linguists to make use of points (3a–i) in their research19

**UP 033 odijk odijk\_printer 2017/12/15 15:57 Page 267 #287**

### **21.5 Discussion and Conclusions**

Current tools available to linguists who are interested in doing syntactic research on annotated corpora allow nding and counting syntactic constructions. This chapter takes the conditional inversion as an example, and shows that more soware help can be given to address the kinds of questions a linguist asks. This chapter argues that a researcher would want to annotate all the instances of constructions like the standard conditionals and the conditional inversion with features, taking the research beyond counting syntactic hits.

Users of the stand-alone CorpusStudio have already shown that the availability of this kind of sofware inuences the research process itself: instead of focusing on nding one particular syntactic construction, the creation of feature databases that can again serve as the input to the search process leads to initially broader searches that make use of quite specic feature calculation functions.

Soware that facilitates the intended process could make use of the query language XQuery, since it not only allows searching through syntactically annotated corpora, but also allows calculating the values of the features a linguist may be interested in. The existing CorpusStudio stand-alone Windows program makes use of this query language but has the drawbacks of most stand-alone applications. It is platform-dependent and does not easily help other linguists to work with the same corpus. This chapter mentions the rst version of the CorpusStudio web application, a webbased version of the Windows program. While it does not yet oer all the facilities a researcher would like to make use of, it brings the kind of corpus-based syntactic research advocated in this chapter a step closer to users of the CLARIN infrastructure.

# **Acknowledgements**

I am grateful to CLARIN-NL, who shared my vision and nanced the development of the web application. My Radboud colleagues Meta Links and Sanne van Vuuren have provided useful comments on CorpusStudio and shared their ideas about the web application with me. I owe a lot of thanks for fruitful discussions on technical aspects to my Meertens Institute colleagues Matthijs Brouwer, Erik Tjong Kim San, Hennie Brugman, and Jan Pieter Kunst.

# **References**

19 I leave a detailed discussion of the CorpusStudio web application to another platform, since the main goal of this chapter is to show that syntactic research would be served by software that goes beyond counting hits. Much of the information one would be interested in is available in the CorpusStudio manual. The application is not a typical database application: it searches through physical texts. The search speed of the web application is proportional to the number of processor cores the server it runs on makes available, since the search of each text takes place in a separate thread. Eight cores give a speed improvement of 7.8 compared to one core. The speed of the benchmark project ( 'V2 test versie11') running on the Old English YCOE corpus (1.5 million words) is 18 min, 49 sec on the Windows stand-alone application (2 GHz processor), while it takes 22.64 sec on the web application that makes use of 20 cores.

Bouma, Gosse. 2008. XML information extraction with Xquery: processing wikipedia and Alpino trees. Groningen: Information science, university of Groningen.

**UP 033 odijk odijk\_printer 2017/12/15 15:57 Page 268 #288**

# CHAPTER 22

**UP 033 odijk odijk\_printer 2017/12/15 15:57 Page 269 #289**

# **GrETEL A Tool for Example-Based Treebank Mining**

Liesbeth Augustinus, Vincent Vandeghinste, Ineke Schuurman and Frank Van Eynde

Centre for Computational Linguistics, KU Leuven

### **ABSTRACT**

This chapter describes the use of GrETEL for linguistic research. GrETEL is a linguistic search tool that enables users to look up constructions in syntactically annotated corpora or treebanks. It provides online access to the data, allowing users to query a treebank using either an example sentence or an XPath expression in order to look for similar constructions. A major asset of GrETEL is that it enables non-technical users to consult treebanks in a user-friendly way, which is also in line with the main CLARIN goal of applying the results of speech and language technology to research in the humanities and the social sciences. Besides a description of the querying procedure in GrETEL, this chapter presents a selection of research in Dutch syntax and semantics that has been carried out using GrETEL. Furthermore, an overview is given of further developments.

### **22.1 Introduction**

The construction of syntactically annotated corpora ortreebanks has created exciting opportunities for the empirical investigation of syntax.1 For Dutch, several treebanks are available, e.g. the CGN treebank (van der Wouden et al., 2002) for spoken Dutch, and LASSY (van Noord et al., 2013) and SoNaR (Oostdijk et al., 2013) for written Dutch. While treebanks have the potential to be an added value for descriptive and theoretical linguistics, the exploitation of such treebanks usually requires that the user have in-depth knowledge of the annotation guidelines and master a formal

#### **How to cite this book chapter:**

1 We use the term treebank to refer to both manually constructed syntactically annotated corpora and automatically parsed corpora.

Augustinus, L, Vandeghinste, V, Schuurman, I and Van Eynde, F. 2017. GrETEL: A Tool for Example-Based Treebank Mining. In: Odijk, J and van Hessen, A. (eds.) CLARIN in the Low Countries, Pp. 269–280. London: Ubiquity Press. DOI: https://doi.org/10.5334/bbi.22. License: CC-BY 4.0

query language. Some users are not deterred by this, but many are, so that the potential of the treebanks will not be realised. To make the treebanks useful for the computationally less inclined we have developed GrETEL, a user-friendly search engine for treebanks (Augustinus et al., 2012; Augustinus et al., 2013). It oers the possibility to provide the system with an example sentence in order to collect relevant corpus data. Therefore, the development of GrETEL paves the way for combining treebank mining with descriptive and theoretical linguistics.

**UP 033 odijk odijk\_printer 2017/12/15 15:57 Page 270 #290**

# **22.2 What is GrETEL?**

GrETEL stands for Greedy Extraction of Trees for Empirical Linguistics. It is a linguistic search engine that enables users to extract information from treebanks in a user-friendly way. Instead of a formal search instruction, it takes a natural language example as input. This provides a convenient way for novice and non-technical users to use treebanks with a limited knowledge of the underlying syntax and formal query languages.

Since linguists tend to start their research from example sentences, example-based querying allows them to use those examples as a starting point for treebank search. Work related to our approach is the Linguist's Search Engine (Resnik and Elkiss, 2005), a tool that also made use of example-based querying, but is no longer available, and the TIGER Corpus Navigator (Hellmann et al., 2010), which is a Semantic Web system used to classify and retrieve sentences from the TIGER corpus on the basis of abstract linguistic concepts.

The system we present here is an online system,2 which shares the advantages of tools like TuNDRA (Martens, 2013) and INESS-Search (Meurer, 2012): they are platform-independent and ¨ no local installation of the treebanks is needed. This is especially attractive for (very) large parsed corpora which require a lot of disk space. Another related tool is the more recently constructed PaQu application (Odijk, 2015, see chapter 23). In addition to an online search interface, PaQu also oers the possibility to upload and parse a locally installed corpus.

For a presentation of the way in which GrETEL works we rst focus on the basic search mode of example-based querying (section 22.2.1) and then we turn to more advanced modes of querying (section 22.2.2).

# **22.2.1 Example-Based Querying**

The example-based querying procedure consists of six steps.

**1. Example** The user provides an example sentence, containing the syntactic construction (s)he is looking for. For instance, in colloquial Dutch the complementizer van 'of ' is sometimes used in constructions reecting direct speech (Coppen, 2010; Hoekstra, 2010). An example is given in (1).

(1) Hij he dacht thought van of ik I zal will dat that morgen tomorrow wel rather doen. do 'He thought: I will do that tomorrow'.

**2. Parse** GrETEL automatically parses the input construction using the Alpino parser (van Noord, 2006), and returns it as a syntax tree (see Figure 22.1). The user can verify the parse tree. If Alpino returns an erroneous parse, the user is advised to choose another input example.

**3. Selection matrix** In the selection matrix, shown in Figure 22.2, the user indicates which parts of the entered example are relevant for the construction under investigation, as well as their level of abstraction. We have indicated lemma for van 'of ', and word class of the verbs dacht 'thought' and

2 http://gretel.ccl.kuleuven.be

**UP 033 odijk odijk\_printer 2017/12/15 15:57 Page 271 #291**

**Figure 22.1:** Parse tree of the input construction.

**Figure 22.2:** Selection matrix.

zal 'will', as we want to abstract over verb forms.3 The other words in the example are not relevant for the construction under investigation, so those words are indicated as 'optional in search'.

The dependency relation and the word class (pos tag) of all selected items are automatically included in the search instruction. For instance, it will be taken into account that the word van is a preposition (tagged as vz) functioning as a complementizer (cmp).

**4. Treebank selection** In the next step the user can choose which treebank(s) to query. Currently one can choose between the CGN treebank for spoken Dutch, and LASSY Small and the SoNaR-500 treebank for written Dutch.4 It is possible to query the CGN and LASSY Small treebanks as a whole, or one can select one or more treebank components, for instance to compare data from dierent genres. Because of its size (500 million words, ca 41 million sentences), it is

3 The embedded verb is indicated in order to avoid constructions without an embedded sentence, such as *Hij dacht van wel* 'He thought so'.

4 The SoNaR-500 treebank is a subset of the LASSY Large treebank.

only possible to query SoNaR per component. For this example we have chosen the part of SoNaR containing discussion lists (WR-P-E-A, 50 million words, ca 4.5 million sentences).

**UP 033 odijk odijk\_printer 2017/12/15 15:57 Page 272 #292**

**5. Query** Based on the information provided in the selection matrix, GrETEL extracts a query tree from the parse tree (Figure 22.3). Besides the lexical information indicated in the selection matrix, the dependency relation (rel) and the phrasal category (cat) of the relevant nodes are included in the query tree, see (5). GrETEL automatically converts the query tree into an XPath expression,5 which is used to search the treebank.

**6. Results** The results of the query are presented to the user as a list of sentences, with the matching part emphasised. The user can click on any of these sentences in order to visualise the results as syntax trees. For the query in Figure 22.3 GrETEL nds 175 results in the WR-P-E-A component of SoNaR. Some are presented in (2-4).

The results show the greedy nature of GrETEL: it not only returns constructions in which the parts of the construction indicated in the matrix are adjacent, but also returns examples in which those elements are discontinuous.6 For instance, the nite verb zegt 'says' in (2) is not adjacent to van 'of '. Because of this discontinuity, looking for similar constructions in a at (raw or pos-tagged) corpus would be much harder.

If we run the same query on less informal data, such as the component of SoNaR containing periodicals and magazines (WR-P-P-H), we only nd 42 hits even though the corpus is larger

5 http://www.w3.org/TR/xpath

6 In addition, the XPath expressions that are used (by default) in GrETEL ignore word order. This also gives rise to more general queries compared to queries used in string-based methods.

in size (ca 5.5 million sentences) than WR-P-E-A (ca 4.5 million sentences). This conrms the colloquial nature of the construction.

**UP 033 odijk odijk\_printer 2017/12/15 15:57 Page 273 #293**

Example-based querying has the advantage that the user does not need to be familiar with XPath, nor with the exact syntactic structure of the XML in which the trees are represented, nor with the exact grammar implementation that is used by the parser or the annotators.

# **22.2.2 XPath Search**

In the advanced mode of example-based search, users can inspect not only the query tree (Figure 22.3), but also the corresponding XPath expression, spelled out in (5).

(5) //node[@cat="smain" and node[@rel="hd" and @pt="ww"] and node[@rel="vc" and @cat="svan" and node[@rel="cmp" and @pt="vz" and @lemma="van"] and node[@rel="body" and @cat="smain" and node[@rel="hd" and @pt="ww"]]]]

Moreover, they can make modications to this query. For instance, one can use an or-statement to construct more general queries; e.g. node[@cat="smain" or @cat="ssub"] looks for constructions in both main and subordinate clauses. This approach allows more exibility in the type of patterns that are searched.

For users who are thoroughly familiar with XPath and with the details of the annotation there is also the possibility to directly formulate an XPath query describing the syntactic pattern the user is looking for. This query is then processed in the same way as the automatically generated query in the rst approach.

### **22.3 Using GrETEL for Research and Education**

GrETEL has been used for linguistic research on various topics within Dutch syntax and semantics (section 22.3.1). In addition, it has been used for teaching, and it has been presented at several conferences and guest lectures (section 22.3.2).

### **22.3.1 Research on Dutch Syntax and Semantics**

While GrETEL has been used to investigate several linguistic topics, two strands of research received considerable attention, i.e. the investigation of verb clusters and of copular constructions.

### <IP_ADDRESS> Verb Clusters

Augustinus (2015) provides both a theoretical and a treebank-based account of Dutch verb clusters, i.e. constructions in which multiple verbs group together. She shows how such constructions can be extracted from the treebanks using GrETEL, and how the treebank observations serve as an empirical basis to verify the claims made by the theory. She conducted several case studies, such as word order variation in verb clusters, the occurrence of Innitivus pro Participio (a.k.a. the IPP eect), and interruption of the cluster by nonverbal elements.

Dutch verb clusters are characterised by an unusual type of word order variation, i.e. one that does not entail a change of meaning, as shown by the examples in (6).

	- '... in that case I also think that that man has done it.' (CGN, fna000458 166)
	- b. ah ah dan then hoop hope ik I dat that Ivo Ivo dat that gedaan done hee. has 'ah in that case I hope that Ivo has done that.' (CGN, fva400092 136)

Augustinus (2015) investigates which types of word order variation occur in non-dialectal varieties of Dutch, i.e. in the CGN and LASSY Small treebanks included in GrETEL. Barbiers and Schuurman (2015) compare the word order variation in three-verb clusters encountered in those treebanks to data obtained from MIMORE, a tool for investigating morphosyntactic variation in Dutch dialects.7

**UP 033 odijk odijk\_printer 2017/12/15 15:57 Page 274 #294**

Innitivus pro Participio or IPP refers to constructions in which an innitive occurs instead of a past participle, as in (7).

	- b. \* Ik I heb have 't it twee two keer times gezien seen gebeuren. happen

IPP appears in a subset of the Germanic languages, such as Dutch, German and Afrikaans. These languages dier, however, with respect to the set of verbs that can appear as IPP verbs, and with respect to whether the phenomenon occurs obligatorily or optionally. For some verbs, the literature is not conclusive on whether they can occur in IPP constructions or not. Augustinus and Van Eynde (2012) and Augustinus (2015) describe how a treebank-supported investigation of Dutch IPP verbs using GrETEL results in a more exhaustive and empirically valid typology of Dutch IPP verbs than the lists available in the literature.

Augustinus and Van Eynde (2017) compare the set of Dutch IPP verbs to the German IPP verbs. In order to add this cross-linguistic perspective, they queried two German treebanks using the TuNDRA treebank search tool (Martens, 2013). The case study not only illustrates how the results ¨ obtained by GrETEL can be complemented by using additional resources, but also shows how the treebank data can be employed to evaluate theoretical accounts of IPP.

A third case study on IPP using GrETEL investigates the choice of the auxiliary of the perfect in Dutch IPP constructions, i.e. the choice between hebben 'have' and zijn 'be'. Canonically the choice for the auxiliary in IPP constructions is determined by the IPP verb, as in (8a). However, one also encounters constructions in which the auxiliary is determined by the main verb, as in (8b).

	- b. hee has er there niemand no one dat that komen come.ipp zeggen say tegen to jullie? you 'Did nobody come to tell you that?' (CGN, fva400386 18)

While this variation has been reported in the literature, no large-scale corpus study was available pointing out the frequency and the distribution of the phenomenon. Van Eynde et al. (2016a) investigate the choice between hebben 'have' and zijn 'be' in IPP constructions by means of GrETEL and OpenSoNaR.8 The corpus study provides insight in the set of verbs that allow this alternation. For the verbs moeten 'must' and kunnen 'can' the distribution of the canonical and the alternative construction is investigated in more detail.

Besides word order variation and the IPP eect, Augustinus and Van Eynde (2014) and Augustinus (2015) investigate the occurrence of cluster interruption. Canonical verb clusters

7 http://www.meertens.knaw.nl/mimore/

8 OpenSoNaR provides string search of the flat SoNaR-500 corpus. (http://opensonar.clarin.inl.nl)

cannot be interrupted by nonverbal elements (9). There are some exceptions though, such ascluster creeping by separable verb particles, predicative adjectives, and stranded adpositions (10).

**UP 033 odijk odijk\_printer 2017/12/15 15:57 Page 275 #295**

	- '... the proposal that the NS has made today ...' (CGN, fnk001631 2)
	- b. \* ... ... de the voorstellen proposals die that de the NS NS hee has vandaag today gedaan done ... ...

Bavarian can of dream

'The duty that calls him now can cost him the most beautiful job a Bavarian can dream of.' (LASSY, WR-P-P-I-0000000033.p.21.s.4)

The treebank investigations conducted in Augustinus and Van Eynde (2014) and Augustinus (2015) show that the set of cluster creepers is larger than the literature suggests. This illustrates once more how a treebank-based investigation can provide additional insights into syntactic phenomena.

### <IP_ADDRESS> Copular Constructions

In addition to the research on verb clusters, GrETEL is used for research on copular constructions. Van Eynde et al. (2014) illustrate that the set of copular verbs discussed in traditional grammars is incomplete. Typically those grammars mention a set of 10 to 15 verbs, adding, as an aerthought, that the list is not complete. By means of GrETEL treebank data were collected in order to get a more complete and empirically motivated typology of Dutch copular constructions, which consists of at least 40 verbs. As the typology is based on linguistically motivated criteria, it can be used to complete the list of verbs by investigating a larger dataset.

Van Eynde et al. (2016b) deal with number agreement in copular constructions. Canonically, there is number agreement between the subject and the predicate nominal in Dutch copular constructions, as in (11). Mismatches are not excluded, however, as shown in (12).

This research demonstrates how the data obtained from the treebanks not only provide information with respect to the frequency and the distribution of number agreement in copular constructions, but also serve as an empirical basis for a theoretical analysis. In addition, the treebank data were employed to dene under which circumstances mismatches between the subject and the predicate nominal are allowed.

### **22.3.2 Dissemination**

GrETEL is currently used in courses on descriptive linguistics, syntax and semantics, corpus analysis, and computational linguistics in order to teach students how to look up syntactic constructions and their frequencies in a treebank without requiring them to familiarise themselves with the specics of XPath or the specic syntax of the treebank. It teaches students about syntactic parses and treebanks by providing them easy online access to large amounts of data.

**UP 033 odijk odijk\_printer 2017/12/15 15:57 Page 276 #296**

As GrETEL has a focus on user-friendliness and is freely available online, it is an example of how Digital Humanities applications disclose datasets and computational tools, without requiring the user to have a technical background.

GrETEL was presented to a technical audience at several conferences within the eld of computational linguistics and to an audience of potential users at general linguistic conferences and doctoral schools in Flanders and the Netherlands. Those lectures typically include a tutorial demonstrating the functionality and use of GrETEL, followed by a hands-on session. In addition, some case studies are discussed, showing how the results obtained from the treebanks in GrETEL can serve as an empirical basis for research in linguistics. One of the case studies includes the combined use case of GrETEL and MIMORE (Barbiers and Schuurman, 2015). It illustrates how GrETEL and MIMORE can be used as complementary tools for studies on Dutch syntax.

# **22.4 Further Developments**

GrETEL has been designed in such a way that it can also be used for treebanks in other languages, even if they have dierent annotation schemes compared to the Dutch treebanks. In the Afri-Booms project (Augustinus et al., 2016a), a treebank for Afrikaans has been developed, which is also included in GrETEL (section 22.4.1). In the context of the SCATE project (Vandeghinste et al., 2016), GrETEL was adapted to query parallel treebanks (section 22.4.2). The tool is also included in Taalportaal (Landsbergen et al., 2014), an online descriptive grammar of Dutch (section 22.4.3).

# **22.4.1 GrETEL for Afrikaans**

In comparison to Dutch, Afrikaans is a low-resource language, so until recently no treebanks for Afrikaans were available. In the AfriBooms project a (small) treebank containing ca 50K words has been developed, based on the corpus of the South African National Centre for Human Language Technologies (NCHLT). The annotations of the treebank are manually corrected, which makes it a reliable resource for linguistic research. In addition, a rst parser for Afrikaans was developed. Both the treebank and the parser are included in a version of GrETEL for Afrikaans (Augustinus et al., 2016a).9

### **22.4.2 Querying Parallel Treebanks with Poly-GrETEL**

In the context of the SCATE project, large-scale parallel treebanks are constructed which are used for syntax-based machine translation. Since parallel treebanks are a valuable resource for translators and linguists as well, Poly-GrETEL was developed, i.e. an extension of GrETEL for querying parallel treebanks (Augustinus et al., 2016b).10 Currently it contains the (automatically annotated) Europarl parallel treebank for Dutch and English.

**The Europarl parallel treebank** We have made an update of the treebank described in Kotze´ et al. (2016): we used the data from Europarl version 7 (Koehn, 2005) and extracted the Dutch and English sentence-aligned data from www.statmt.org. The Dutch side was parsed with the

9 http://gretel.ccl.kuleuven.be/afribooms

10 http://gretel.ccl.kuleuven.be/poly-gretel

Alpino parser and the English side with the Stanford parser (Klein and Manning, 2003) with added dependencies (de Marnee et al., 2006). The phrase structure output of the Stanford parser is converted into an XML-tree,11 analogous to the XML-output of Alpino, as shown in Figure 22.4. Besides the syntactic annotations the parallel treebank contains node alignments.12

**UP 033 odijk odijk\_printer 2017/12/15 15:57 Page 277 #297**

**Poly-GrETEL** In combination with the example-based query functionality, Poly-GrETEL avoids the need for users to be familiar with the query language and the structure of the trees in the source and target language, thus facilitating the use of parallel corpora for comparative linguistics and translation studies.

The user can query the treebanks in a similar way as in the monolingual GrETEL environment, i.e. example-based or by means of an XPath query. The main dierence is that the user can choose between a bilingual and a monolingual input. In the bilingual search option the user provides two input constructions: one in English and one in Dutch. Poly-GrETEL returns two parses, and the user can indicate the relevant parts of both the English and the Dutch input examples. Poly-GrETEL automatically extracts a search instruction in a similar fashion as the monolingual GrETEL, but provides the option to return only the constructions in which the English and the Dutch query trees are aligned. It is a syntactic concordancer for parallel treebanks, as it shows how a Dutch syntactic construction is translated in English (or vice versa). One could, for instance, investigate how the Dutch van-construction presented in section 22.2.1 is translated in English. This makes the tool interesting not only for research in (comparative) linguistics and translation studies, but also to serve as a tool for computer-aided translation for translators and language learners.

Adding the parallel English-Dutch treebank furthermore implies that GrETEL also includes English data. Since it is possible to query the English side of the parallel treebank in a monolingual way, one can use these data for a monolingual treebank investigation of syntactic phenomena in English.

**Figure 22.4:** Stanford parse with added dependencies, converted into Alpino-XML.

11 The bracketed tree and the XML tree are isomorphous.

12 In future versions alignments resulting from several different alignment algorithms will be made available. In the current version, only alignment according to Zhechev (2009) is available.

### **22.4.3 Link with Taalportaal**

**UP 033 odijk odijk\_printer 2017/12/15 15:57 Page 278 #298**

Recently, GrETEL was linked to Taalportaal, a website that contains online descriptive grammars for Dutch, Frisian and Afrikaans (Landsbergen et al., 2014, see chapter 24).13 By means of intelligent links, users can look up linguistic phenomena described in Taalportaal in a variety of online corpora, amongst others the treebanks included in GrETEL (van der Wouden et al., 2015).

The link with Taalportaal enhances the visibility of GrETEL, and encourages its use, alone or in combination with other corpus tools. Bouma et al. (2015) mention how they have used the example-based input method of GrETEL to facilitate query formulation. It turns out to be particularly useful if one does not know exactly how certain phenomena are annotated in the treebanks. In addition, the authors mention how they have used GrETEL's example-based querying functionality to become aware of dierences between the treebank annotations and the analyses of the descriptive grammar included in Taalportaal (Bouma et al., 2015: 18).

# **22.5 Conclusion and Future Work**

We have described GrETEL, a user-friendly search tool for treebanks. It originated in the context of a CLARIN-Flanders project, which aimed at the creation of tools for the exploitation of Dutch treebanks. In follow-up research, GrETEL was extended to other languages (Afrikaans and English), and other types of treebanks, i.e. parallel ones. The extensions make the tool also useful for a larger (CLARIN) audience, i.e. researchers who are not (only) working on Dutch.

Future work includes adding more languages to GrETEL, such as German and French, as for those languages we also have high-quality parsers and treebanks available.

In the framework of the Dutch CLARIAH infrastructure project and the Anncor project (University of Utrecht), there are plans to further extend the functionality of GrETEL. An upload function will be added, enabling researchers to upload their own corpus and metadata, supporting multiple formats. Another extension concerns adding options for data analysis, and creating possibilities to sort, group, and lter search results and metadata.

### **Acknowledgements**

The work on GrETEL was carried out in the framework of the following projects:

13 http://taalportaal.org

## **References**

**UP 033 odijk odijk\_printer 2017/12/15 15:57 Page 279 #299**

Gideon Kotze, Vincent Vandeghinste, Scott Martens, and J ´ org Tiedemann. 2016. Large Aligned ¨ Treebanks for Syntax-based Machine Translation. Language Resources and Evaluation. Vol. 51: 249–282.

**UP 033 odijk odijk\_printer 2017/12/15 15:57 Page 280 #300**

# CHAPTER 23

**UP 033 odijk odijk\_printer 2017/12/15 15:57 Page 281 #301**

# **The Parse and Query (PaQu) Application**

Jan Odijk, Gertjan van Noord, Peter Kleiweg and Erik Tjong Kim Sang Utrecht University – RU Groningen – Meertens Institute<EMAIL_ADDRESS>
### **ABSTRACT**

In this chapter we describe the web application PaQu (Parse and Query), and carry out a small case study to illustrate its use. PaQu is an application for searching in Dutch treebanks and for analysing the search results. One can search in the LASSY and CGN treebanks, or upload one's own Dutch corpus, which is then parsed and made available for search and analysis. PaQu oers, next to an interface to formulate Xpath queries, a dedicated interface for searching for dependency triples. This makes it easy to search in treebanks for grammatical dependencies, which would otherwise require very complex queries. It oers extensive functionality for analysing the search results. The dedicated search interface makes PaQu a prime example of the kind of applications that CLARIN promotes. The case study provides an analysis of the syntactic selectional dierences between two near-synonymous verbs.

### **23.1 Introduction**

In this chapter we describe the web application PaQu1 (Parse and Query), and carry out a small case study to illustrate its use. PaQu is an application for searching in treebanks (i.e. text corpora in which each sentence has been assigned a syntactic structure) and for analysing the search results. PaQu was developed by the University of Groningen in the CLARIN-NL project and is an extension of the LASSY Dependency Relations Search application originally developed by Tjong Kim Sang et al. (2010). Here is a direct link to the application and its documentation.

#### **How to cite this book chapter:**

1 This chapter contains many hyperlinks. In electronic versions they are easily visible by special marking (blue colour). In the printed black and white version the hyperlinks may be visible as grey text. The relevant URLs are explicitly listed in the appendix.

Odijk, J, van Noord, G, Kleiweg, P and Tjong Kim Sang, E. 2017. The Parse and Query (PaQu) Application. In: Odijk, J and van Hessen, A. (eds.) CLARIN in the Low Countries, Pp. 281–297. London: Ubiquity Press. DOI: https://doi.org/10.5334/bbi.23. License: CC-BY 4.0

PaQu has been used for carrying out research, inter alia for an analysis of the Dutch intensiers heel, erg, zeer (each meaning 'very') by Odijk (2015a) and Odijk (2016), and for an analysis of normative and non-normative variants of Dutch constructions by Odijk (2015b) and Van Noord and Odijk (2016). It is also heavily used in the Taalportaal, a digital grammar for Dutch (Van der Wouden et al., 2016), where many of the links with queries for corpus examples use the PaQu interface (see Bouma et al. (2015), Van der Wouden et al. (2015) and chapter 24).

**UP 033 odijk odijk\_printer 2017/12/15 15:57 Page 282 #302**

PaQu oers three types of functionality: corpus management (described in section 23.2), search (described in section 23.3), and analysis (described in section 23.4). Section 23.5 contains an analysis of the predicative complements of the near-synonymous copular verbs worden 'become' and raken 'get' as a small case study. Section 23.6 summarises our conclusions and sketches future work. The appendix contains a list of all hyperlinks.

## **23.2 Corpus Management**

PaQu enables one to select a treebank to search in from the publicly available Dutch treebanks in PaQu. These include LASSY-Small, the Wikipedia part of LASSY-Large (Van Noord et al., 2013), and the treebank of the Spoken Dutch Corpus (Oostdijk et al., 2002). One can also upload one's own Dutch text corpus.2 One's own corpora can be kept private or shared with others. Login is required for managing one's corpora, since the user wants to see his/her own corpora but not someone else's (unless they are shared).

PaQu accepts as input plain text (in multiple varieties), which it then parses using Alpino (Van der Beek et al., 2002), or a text corpus already parsed by Alpino in the LASSY XML format. Aer this, the treebank is available for search and analysis. New input formats are being added (see section 23.6).

PaQu oers full parses of sentences in one's own corpus, but these parses are generated in a fully automatic manner, so they inevitably will contain errors. It is therefore required to evaluate the quality of the automatically generated parses.3 Odijk (2015a) describes the results of such an evaluation for the words heel, erg and zeer in the CHILDES Van Kampen subcorpus, and he concludes that PaQu is excellent for carrying research on these phenomena, since the accuracy of Alpino for these data is in part very high (accuracy well over 90%), and the part where the accuracy is low is easily identiable with PaQu. Though these results cannot be automatically generalised to other cases, they show that PaQu can be very useful even in cases where Alpino's accuracy is low, provided that the problematic cases can be automatically identied with PaQu.

Finally, PaQu enables the user to store the results of a query in a new corpus.

# **23.3 Search**

PaQu enables search for utterances in the treebank by selecting for properties in the syntactic structures of these utterances. Before discussing this in more detail in section 23.3.2, we will say some words on the nature of the syntactic structures in the treebanks that PaQu deals with (section 23.3.1). We conclude with a description of the search results in section 23.3.3.

2 This raises the question of how long these user corpora are retained, as one of the anonymous reviewers noted. On 18 May 2016, PaQu hosted 47 user corpora for 34 different users, and so far there are no problems in hosting these. Ultimately, the retention policy will be determined by the CLARIN Centre that will host PaQu.

3 See Bloem (2016) for general strategies to evaluate automatically generated parses in the absence of a gold standard.

### **23.3.1 Syntactic Structures in Dutch Treebanks**

**UP 033 odijk odijk\_printer 2017/12/15 15:57 Page 283 #303**

The character of the syntactic structures in the treebanks in PaQu is in accordance with the de facto standard for syntactic structures in treebanks for the Dutch language (Hoekstra et al., 2003; Van Noord et al., 2011), dened in projects for the construction of Dutch treebanks such as CGN (for spoken Dutch; Oostdijk et al., 2002) and LASSY (for written Dutch; Van Noord et al., 2013). They are tree structures of a particular kind.4 They dier from pure constituent structures (as used e.g. in Chomskyan generative grammar (Chomsky, 1965) and LFG's c-structures (Bresnan, 1982)) in two respects: they explicitly label grammatical relations, and order in the tree is not signicant and need not correspond to the surface order.5 They dier from dependency structures (on the analytical layer) as used in the Prague Dependency Treebank (Hajic and Haji ˇ cov ˇ a, 1997) in explicitly allowing ´ grouping of nodes under non-word nodes. The following simple (English) example illustrates the dierences between a constituent structure (1a), a dependency structure (1b) and a CGN/LASSYstructure (2) for the sentence The man saw a boy: 6

4 There is no generally accepted name for these tree structures. Van Noord et al. (2011) call them 'dependentiestructuren' (dependency structures), but that is not a very fortunate choice.

5 The surface order is specified in attributes on nodes for words.

6 The nonsignificance of order in the CGN/LASSY tree is not visible in these trees.

The syntactic structure for the Dutch sentence De man zal dat boek niet willen lezen 'The man will not want to read that book' in (3) clearly shows that order in the tree does not necessarily correspond to surface order:7

**UP 033 odijk odijk\_printer 2017/12/15 15:57 Page 284 #304**

Another characteristic of the syntactic structures that is in accordance with the Dutch de facto standard for syntactic structures in treebanks is that a structure cannot contain nodes that dominate a single node. So, e.g., if the subject of a sentence consists of a single pronoun (e.g. a pronoun such as 'he'), then the structure is not like in (4), since that structure contains a node (NP) that dominates a single node (he):

$$\begin{array}{c} \text{s} \\ \text{s} \\ \text{l} \begin{array}{c} \text{l} \\ \text{l} \\ \text{NP} \end{array} \text{VP} \\ \text{(4)} \\ \text{(4)} \\ \text{ } \begin{array}{c} \text{l} \\ \text{h} \\ \text{h} \end{array} \end{array} \text{VP} $$

but is instead as in (5):

7 Thus enabling the representation of discontinuous dependencies or what are called *non-projective structures* in Dependency Grammar.

$$\begin{array}{c} \text{s} \\ \text{(5)} \\ \text{(5)} \\ \text{ } \quad \text{u} \\ \text{ } \quad \text{h} \\ \quad \text{ } \quad \text{VP} \\ \quad \text{ } \quad \text{ } \quad \text{} \end{array}$$

Syntactic structures can also contain nodes that have no content except for an index attribute ('empty categories'). These are always co-indexed with some node with content. This mechanism is used to express constructions in which a single word or phrase plays multiple syntactic roles. We will see several examples of this in section 23.3.2.

**UP 033 odijk odijk\_printer 2017/12/15 15:57 Page 285 #305**

### **23.3.2 Queries in PaQu**

PaQu oers two interfaces for formulating queries to search in the treebanks. The rst one enables one to formulate queries in the Xpath query language. This requires expert knowledge of the query language Xpath. Learning Xpath, or at least the most common elements required in PaQu, is easy, so this actually poses no problems. A more serious problem is that Xpath queries tend to become quite complex very quickly. Here PaQu aids the user by providing a syntax check: is the Xpath query well-formed (white background colour), or not (red background colour), and are valid elements and attributes used in the query or not (yellow background)?. It also provides suggestions for completing the Xpath expression that is being constructed. In addition, extensions of Xpath originally developed for the desktop Alpino corpus tool DACT (Van Noord et al., 2013) can be used: macros, which can be used for abbreviating oen occurring complex Xpath expressions, and query pipelines, which are useful for improving performance.

Most problematic for constructing Xpath queries is that the user must know, in every ne detail, what the syntactic structures for a variety of constructions actually look like in the treebank. The GrETEL application ((Augustinus et al., 2012) and chapter 22) avoids this problem through a dedicated user interface that makes example-based querying possible. This imposes restrictions on the types of queries that can be made, but many oen used query types can be constructed in this way on the basis of an example without actually having to formulate a query in a formal query language or to know in great detail what the syntactic structures look like.

The second query interface that PaQu oers is, like GrETEL's interface, a highly user-friendly dedicated interface, but it is user-friendly in a completely dierent way than GrETEL's interface. PaQu enables searching for dependency triples (see Figure 23.1). Such triples are of the form '(dependent word, relation, head word)'. PaQu's dedicated interface enables one to formulate queries in terms of these three elements. One can query for the head word and the dependent word by specifying some of their properties, in particular, their lemma, their word form, and their part of speech. The relation element is an attribute that takes atomic labels for grammatical relations such as subject (su), direct object (obj1), modier (mod), etc. The query can basically be formulated in terms of the seven attributeslemma, word, postag (for the dependent word), hlemma, hword, hpostag (for the headword) and rel for the relation. One has to specify minimally one of these elements in the query. If one does not specify anything at all for any of these elements, it eectively acts as a variable and matches with anything. Some examples may illustrate this:8

(6) a. Yield sentences from CGN that contain the lemma heel as a modier: (lemma='heel', rel='mod').

8 Clicking on the queries executes them.

**UP 033 odijk odijk\_printer 2017/12/15 15:57 Page 286 #306**

**Figure 23.1:** PaQu web interface with a query for occurrences of the lemma heel as modier.

Searching for dependency triples is very oen needed in linguistic research, so having a simple and very user-friendly way of formulating such queries is one of PaQu's greatest benets. Obviously, the simple and user-friendly way in which this can be done also restricts the kind of queries one can pose, but it does so in a completely dierent way than GrETEL does. In this way PaQu is complementary to GrETEL.

The relevant queries for dependencies triples are not easily expressed in Xpath. Such a query has to take into account not only headed structures but also coordinated structures and co-indexed nodes in the syntactic structure. In addition, the dependent word can be contained in a phrase that is a dependent of the head word. We will elaborate a little bit on this.

A word w that modies some other word h can be modied itself. In that case w is the head of a constituent, and h is actually modied by the constituent dominating w. For example, the result of the query (word='erg', rel='mod') in LASSY-Small contains such examples, in particular the resulting sentences 1, 18, 71, 77.9 The relevant part of the structure of 18 (heel erg zwaar, lit. very awfully heavy, 'really very heavy') is given in (7):

9 The order of the query output is not always guaranteed to be the same, so one might find these examples under other numbers in a different instantiation of the query. The links should work, however.

In (7) erg 'awfully' is the head of the AP (heel erg) 'very awfully', which modies zwaar 'heavy': in such cases we want the dependency triple (erg, mod, zwaar) in the results, which is what PaQu's dedicated interface achieves.

**UP 033 odijk odijk\_printer 2017/12/15 15:57 Page 287 #307**

Suppose that one is interested in subjects of the verb lopen. This word, as almost any word in natural language, is highly ambiguous: the online Van Dale lists nine senses (and De Boer (2007) even lists eleven), and possible translations into English include 'walk', 'run', 'stream', 'last', 'move', 'work', and others). The ambiguity is oen reduced and occasionally even resolved by combining it with its arguments. For investigating combinations of the verb lopen with subjects, one can use the query (rel=su, hword='lopen') in LASSY-Small. But a subject of a verb can be a conjunction of multiple words, e.g. droom en werkelijkheid (dream and reality) in the sentence Droom en werkelijkheid lopen door elkaar10 (example 10), as well as in many other examples, among them 21, 65 and 90 of the result set. PaQu counts each conjunct and the conjunction itself as a subject of the verb in such cases, which is a reasonable and very useful strategy.

Another pecularity of the syntactic trees is that they can contain nodes that are co-indexed with other nodes (as described above). If a node a is co-indexed with node b, and b is a dependent of some head h, then we want to consider a to be a dependent of h as well. Co-indexed nodes are used for a range of linguistic phenomena. Ellipsis is one: in the query (word='heel', rel='mod') examples 2 and 4 have been analysed as containing ellipsis; in an informal notation, for example 2, the part heel afwisselende en rijke is analysed as heel afwisselende en heel rijke, in which the striked-out heel is an 'empty category' co-indexed with the rst occurrence of heel.

Coindexation is also used for control, for subject and object raising, and for long distance movements (e.g., preposed relative and interrogative pronouns). In the query (rel=su, hword='lopen') we nd subject raising in examples 15 (subject of was and subject of gelopen), 16 (subject of blijkt and lopen) and 32 (subject of leken and lopen); ellipsis and subject control in example 17 (subject of wil and lopen); object raising in 21 (object of laat and lopen), subject raising twice in 22 (subject of hebben, kunnen, and lopen); and object raising in 28 (direct object of zagen and subject of lopen); etc. etc. An example of wh-movement can be found in example 15 (preposing of the relative pronoun die) .

Making queries that take into account all these dierent aspects is very dicult, and these queries are not easily expressed in Xpath (see the DACT Cookbook, sectionAntecedents of co-indexed nodes for an implementation of including indexed nodes in Xpath). PaQu's dedicated search interface, however, does it all automatically without the user having to worry about these cases: when searching for any relation, PaQu not only searches for nodes with that relation, but also searches for nodes that are the head of a phrase with that relation, or that are a conjunct in coordinated phrases; and when it nds a node searched for, it also searches for nodes co-indexed with the found node.

Each executed query in PaQu yields a URL that can be used to replicate the query. The URL encodes the server that PaQu runs on and parameters of the query to be performed. Any web

10 'Dream and reality intermingle'.

browser can interpret these URLs (as HTTP GET requests), so that it is easy to replicate the query exactly. We have used many of these URLs in this document in hyperlinks to queries. They have also been used extensively in the Taalportaal (Van der Wouden et al., 2016) to provide the links from descriptions of constructions to examples in text corpora (see Bouma et al. (2015),Van der Wouden et al. (2015) and chapter 24).

**UP 033 odijk odijk\_printer 2017/12/15 15:57 Page 288 #308**

### **23.3.3 Search Results**

The output of a PaQu query is a list of utterances that match the query. They are preceded by a specication of the query (for example 'word' = "heel" AND 'rel' = "mod" for the query (word='heel', rel='mod')).

PaQu shows the results on dierent web pages, with by default 10 utterances per page, though one can have up to 500 utterances per page. To see more results, one has to browse to the next results page. Representing only a limited number of utterances per page, and initially only one results page, enables users to experiment with a query, to quickly inspect the initial results, and, if needed, to adapt the query, while at the same time saving computing time and avoiding unnecessary waiting time on the user side. If one is interested in all results, one can compute and analyse them in the analysis part.

Each utterance in the results is assigned a number, and the syntactic structure of the utterance can be viewed by clicking on the clover button to the right of the utterance. In the dependency relations search interface the head and the dependent words are marked by colour: green for the head word(s), yellow for the dependent word(s). Each utterance is followed by the dependency triple(s) found in the utterance, and each of these triples is a query itself, automatically generated by PaQu. For example, in the query (word='heel', rel='mod'), result 3 is followed by the triple hele:adj – mod – land:n, which itself is a link containing a query (word='hele', postag='adj', rel='mod', hword='land', hpostag='n') that searches for utterances containing the adjective hele as a modier of the noun land. Clicking on it executes this query.

### **23.4 Analysis**

The results page of a query shows not only the resulting utterances, but also several options for analysing the search results. The button Tellingen algemeen (General Counts) provides counts of the occurrences of each of the seven attributes that dene a dependency relation query: word, lemma, postag, rel, hword, hlemma, hpostag. The resulting statistics can be downloaded as a tab-separated text le.

One can also compute counts of combinations of these attributes, using the button Tellingen van Combinaties (Combination Counts; see Figure 23.2). The result is a table with counts of the examples that meet the criteria, with instantiation of the unspecied attributes by the values occurring in the search results (i.e. they are acting as variables). The counts in the table are hyperlinks with automatically generated queries for exploring specic subcases in more detail. Clicking on the column header sorts the table by the values in this column (the header is then represented in italics).

PaQu already enables analysis of search results in combination with metadata, at least for some corpora, e.g. the Spoken Dutch Corpus. This result was achieved in the CLARIAH project, the successor of CLARIN-NL. This aspect has been heavily used in Van Noord and Odijk (2016), where it is shown that certain non-normative variants of Dutch construction only occur in a specic region, e.g. hun 'them' as a subject only occurs in the Netherlands part of the corpus, while 'm 'him' as a subject only occurs in the Flanders part of the corpus.

**UP 033 odijk odijk\_printer 2017/12/15 15:57 Page 289 #309**

**Figure 23.2:** PaQu analysis: count of occurrences of the lemma heel as modier by part of speech of the modiee.

# **23.5 Case Study**

Odijk (2011) sketched a research problem concerning the acquisition of syntactic selectional properties of words. Odijk (2015a) and Odijk (2016) investigate this problem for the words heel, zeer and erg (all meaning 'very'): heel (under this reading) only selects adjectival predicates while erg and zeer select adjectival, verbal and adpositional predicates.

Here we illustrate an initial step in an investigation of this problem for a similar but dierent case: on the basis of initial data, we assume that the verb worden 'become' selects only adjectival and nominal predicates (8), while the almost synonymous11 verb raken 'get' selects only adjectival and adpositional12 predicates (9):13

(8) a. Zij she werd became zwanger pregnant 'She became pregnant'

11 The only semantic difference, as far as we can see, is that *raken* implies accidentality of the state or property change, while *worden* is neutral in this respect.

12 This includes words that are traditionally called locational and directional 'adverbs'.

13 Of course, these verbs, like almost all natural language words, have multiple uses: *worden* is also used to form passive constructions, and *raken* is also a transitive verb meaning 'hit'. We will ignore these uses here, which is easy since PaQu enables us to separate them from the copular use of these verbs.

	- b. Zij she raakte got in in verwachting expectation 'She got pregnant'
	- c. \* Zij she raakte got dokter doctor

We search, using PaQu, in the LASSY-Small corpus for the predicative complements of raken using the query (rel='predc', hlemma='raken') and analyse the results (see Table 23.1). We do the same for worden using the query (rel='predc', hlemma='worden'), and the analysis of the results yields Table 23.2. The counts in the Tables 23.1 and 23.2 are links to queries that search for utterances containing predicative complements with the relevant postag, just as in the analysis table of PaQu. These queries, which are automatically created by PaQu, are characterised in the third column of Tables 23.1 and 23.2.14

**UP 033 odijk odijk\_printer 2017/12/15 15:57 Page 290 #310**

**Table 23.2:** Counts of predicative complements to worden grouped by postag.

14 The relevant part of speech codes used in the LASSY-Small treebank are:adj (adjective), ww (verb), vz (adposition), vg (conjunction), mwu (multiword unit), n (noun), tw (numeral), vnw (pronoun), bw (adverb), and spec (special).

For raken, we nd, as expected, adjectival predicative complements (adj) and adpositional predicative complements (vz). PaQu also lists verbal predicative complements (ww), but many of these are adjectives that happen to be identical in form to passive participles of verbs: such adjectives are always analysed in the Dutch treebanks as verbs, but in reality they are adjectives, as can be shown with standard tests to distinguish verbal participles from homophonous adjectives. An example is besmet 'contaminated' in this example, also represented in (10):

**UP 033 odijk odijk\_printer 2017/12/15 15:57 Page 291 #311**

(10) In in China China zijn are varkens pigs besmet contaminated geraakt got met with de the vogelpest fowl pest 'In China, pigs got contaminated with the fowl pest'

It is not fully clear to me that all examples can be analysed as adjectives, though many researchers claim this (e.g. Broekhuis and Corver (2015:section 6.2.3, I, p. 989)). This requires further research. If not, we must allow verbal participial complements to the verb raken as well. Examples with the part of speech tag vg concern conjoined structures (in all cases with adjectival conjuncts). The code mwu stands for multiword unit: the treebank identies certain word combinations as multiword units without assigning them any part of speech tag other than mwu. The two examples are the expressions in de war and door de war, both meaning 'mixed up' or 'confused'. They are clearly adpositional phrases, even though the treebank does not explicitly state this (probably because the expressions have an idiosyncratic meaning and the word war only occurs in these expressions).

Finally, PaQu lists one example with a nominal predicate (n), which is not expected. Closer inspection shows that the syntactic structure for this example contains an error: the example contains the transitive verb raken and the nominal complement should have been assigned the grammatical relation obj1. Even in treebanks that are claimed to have been manually veried some errors will occur. The fact that this error occurs is actually good, because it shows that the grammar behind the treebank does not exclude nominal predicative complements to the verb raken. If it did, there would be a danger of circularity: we would then not nd any nominal complements because the grammar behind the treebank cannot create such structures. But this example shows that there is no circularity here.

Of course, the presence of such errors raises the question of how reliable the data underlying the analysis given here are. Van Noord et al. (2013) mention an independent validation which reports a sentence accuracy, i.e. sentences without any error in the syntactic analysis divided by all sentences, of 97.8% for LASSY-Small. To assess the accuracy for dependency triples with raken or worden as hlemma, we manually annotated all 1,039 dependency triples with hlemma=raken (yielding an accuracy of over 96.1%) and a sample of 866 out of the 27,697 dependency triples with hlemma=worden (yielding an accuracy of over 98.8%). These high accuracy gures clearly indicate that the data that the analysis given here is based on are reliable.

It makes sense to also look at examples where raken has a complement marked with the grammatical relation ld. The label ld is intended for marking locative and directional complements, but it is not obvious that a syntactic distinction with predc must or can be made. The distinction between ld and predc might just be semantic in nature. This distinction is especially dicult to make in many cases because many predicates are expressed by locative phrases in some metaphorical abstract form of space, and many expressions that are literally locative have an idiomatic or metaphorical reading as a state (e.g. aan lager wal raken, lit. 'end up at the lee shore', g. 'come down in the world', as in this example). All ld-complements to raken are adpositional (vz), except for one (this example), in which the complement has postag bw (adverb) and is a locative adverb that falls under adpositions.15 Furthermore, there is one example with an ld complement in

15 The relevant adverb is *waarin*, lit. where-in', for which an analysis as two separate syntactic formatives, *waar* and *in*, that form an adpositional phrase and that happen to be a single word phonologically and orthographically is very plausible.

which the verb raken is actually the (passivised) transitive verb (with meaning 'hit'). It is not obvious that the phrase marked as a complement here is indeed a complement.16

**UP 033 odijk odijk\_printer 2017/12/15 15:57 Page 292 #312**

Finally, we should consider elements with the grammatical relation svp, which is for separated verbal particles. Such particles can be adpositional, nominal or adjectival, and single word predicative complements can oen behave as such particles. A concrete example is opraken 'run out', which is composed of the (intransitive) adposition op 'exhausted' and the verb raken 'get'. But oen the combination of a particle and a verb leads to an unpredictable meaning and unpredictable syntactic selectional properties. There are 19 occurrences of an svp-complement to raken in LASSY-Small, and in 12 cases it indeed involves svp-complements, for the idiosyncratic verbs aanraken 'touch', transitive kwijtraken 'lose' (which are irrelevant for our analysis), and for opraken. In 7 cases, the analysis as an svp-complement is dubious. The examples all involve idiosyncratic expressions such asin de vergetelheid raken 'become forgotten' and in zwang raken 'get fashionable', slaags raken 'start ghting' and (ergens) verzeild raken 'get (somewhere)'. Even when they are analysed as predicative complements (which is a more plausible analysis and is also adopted in traditional grammars), they conform to our assumptions on the syntactic selectional properties of the copular verb raken.

To summarise, these data fully conrm our assumptions about the syntactic selectional properties of the copular verb raken, though perhaps participial complements should be added as an option.

Forworden, we nd many nominal (n) and adjectival (adj) predicative complements, as expected. The examples labelled with vg involve nominal and adjectival conjoined structures. The examples labelled with ww concern true verbal participles incorrectly labelled as predcinstead of as vc(verbal complement), or adjectives that are identical to participles, e.g. opgewonden 'excited' as an adjective, 'wound up' as a participle, in this example, the relevant part of which is in (11):

(11) Je one overschat overestimates jezelf oneself en and wordt becomes opgewonden excited

'One overrates oneself and becomes excited'