We present a neural framework for learning associations between interrelated groups of words such as the ones found in Subject-Verb-Object (SVO) structures. Our model induces a joint function-specific word vector space, where vectors of e.g. plausible SVO compositions lie close together. The model retains information about word group membership even in the joint space, and can thereby effectively be applied to a number of tasks reasoning over the SVO structure. We show the robustness and versatility of the proposed framework by reporting state-of-the-art results on the tasks of estimating selectional preference (i.e., thematic fit) and event similarity. The results indicate that the combinations of representations learned with our task-independent model outperform task-specific architectures from prior work, while reducing the number of parameters by up to 95%. The proposed framework is versatile and holds promise to support learning function-specific representations beyond the SVO structures. Word representations are in ubiquitous usage across all areas of natural language processing (NLP) (Collobert et al., 2011; Chen & Manning, 2014; Melamud et al., 2016) . Standard approaches rely on the distributional hypothesis (Harris, 1954; Schütze, 1993) and learn a single word vector space based on word co-occurrences in large text corpora (Mikolov et al., 2013b; Pennington et al., 2014; Bojanowski et al., 2017) . This purely context-based training produces general word representations that capture the broad notion of semantic relatedness and conflate a variety of possible semantic relations into a single space (Hill et al., 2015; Schwartz et al., 2015) . However, this mono-faceted view of meaning is a well-known deficiency in NLP applications (Faruqui, 2016; Mrkšić et al., 2017) as it fails to distinguish between fine-grained word associations. In this work we present a novel approach to word representation learning that moves beyond the restrictive single-space assumption. We propose to learn a joint function-specific word vector space grouped by the different roles/functions a word can take in text. The space is trained specifically for one structure (such as SVO), 1 and the space topology is governed by the associations between the groups. In other words, vectors for plausible combinations from two or more groups will lie close, as illustrated by Figure 1 . For example, the verb vector study will be close to plausible subject vectors researcher or scientist and object vectors subject or art. For words that can occur as either subject or object such as chicken, this effectively means we may obtain several vectors, one per group, e.g., one for chicken as subject and another one for chicken as object. We achieve this through a novel multidirectional neural representation learning approach, which takes a list of N groups of words (G 1 , . . . , G N ), factorises it into all possible "group-to-group" sub-models, and trains them jointly with an objective similar to skip-gram negative sampling used in WORD2VEC (Mikolov et al., 2013a; b, §3) . In other words, we learn the joint function-specific word vector space by relying on sub-networks which consume one group G i on the input side and predict words from a second group G j on the output side, i, j = 1, . . . , N ; i = j. At the same time, all sub-network losses are tied into a single joint loss, and all groups G 1 , . . . , G n are shared between all sub-networks. 2 1 We choose the SVO structure as there is a number of well defined tasks reasoning over it. Future work could look at different phenomena and how to combine vectors from several function-specific spaces. 2 This can be seen as a form of multi-task learning on shared parameters (Ruder, 2017 Figure 1 : Left: Nearest neighbours in a function-specific space trained for the SVO structure. In the Joint SVO space (bottom) we show nearest neighbors for verbs (V) from the two other subspaces (O and S). Right: Illustration of three neighbourhoods in a function-specific space trained for the SVO structure. The space is structured by group (i.e. S, V, and O) and optimised such that vectors for plausible SVO compositions will be close. Note that one word can have several vectors, for example chicken can occur as subject or object (e.g., it can eat something or someone/something can eat it). To validate the effectiveness of our multidirectional model in language applications, we focus on modeling a prominent linguistic phenomenon: a general model of who does what to whom (Gell-Mann & Ruhlen, 2011) . In language, this event understanding information is typically uttered by the SVO structures and, according to the cognitive science literature, is well aligned with how humans process sentences (McRae et al., 1997; 1998; Grefenstette & Sadrzadeh, 2011a; ; it reflects the likely distinct storage and processing of objects (typically nouns) and actions (typically verbs) in the brain (Caramazza & Hillis, 1991; Damasio & Tranel, 1993) . When focusing on the SVO structures, the model will produce one joint space for the three groups (S, V and O) by tying 6 sub-networks (S→V ; V →S; S→O, . . .) with shared parameters and a joint loss (i.e. there are no duplicate parameters, the model has one set of parameters for each group). The vectors from the induced function-specific space can then be composed by standard composition functions (Milajevs et al., 2014) to yield the so-called event representations (Weber et al., 2018) , that is, representations for the full SVO structure. The quantitative results are reported on two established test sets for the compositional event similarity task (Grefenstette & Sadrzadeh, 2011a; which requires reasoning over SVO structures: it quantifies the plausibility of the SVO combinations by scoring them against human judgments. We report consistent gains over standard single vector spaces as well as over two recent tensor-based architectures (Tilk et al., 2016; Weber et al., 2018) which were tailored to solve the event similarity task in particular. Furthermore, we show that our method is general and not tied to the 3-group condition. We conduct additional experiments in a 4-group setting where indirect objects are also modeled, along with a selectional preference 3 evaluation of 2-group SV and VO relationships (Chambers & Jurafsky, 2010; Van de Cruys, 2014) , yielding the highest scores on several established benchmarks. We presented a novel multidirectional neural framework for learning function-specific word representations, which can be easily composed into multi-word representations to reason over event similarity and thematic fit. We induce a joint vector space in which several groups of words (e.g., S, V, and O words forming the SVO structures) are represented while taking into account the mutual associations between the groups. We found that resulting function-specific vectors yield state-of-the-art results on established benchmarks for the tasks of estimating event similarity and evaluating thematic fit, previously held by task-specific methods. In future work we will investigate more sophisticated neural (sub-)networks within the proposed framework. We will also apply the idea of function-specific training to other interrelated linguistic phenomena and other languages, probe the usefulness of function-specific vectors in other language tasks, and explore how to integrate the methodology with sequential models. The pre-trained word vectors used in this work are available online at: [URL] . 14 In the asynchronous setup we update the shared parameters per sub-network directly based on their own loss, instead of relying on the joint synchronous loss as in §3. 15 With separate parameters we merge vectors from "duplicate" vector spaces by non-weighted averaging.
Task-independent neural model for learning associations between interrelated groups of words. The paper proposed a method for training function-specific word vectors, in which each word is represented with three vectors each in a different category (Subject-Verb-Object). This paper proposes a neural network to learn function-specific work representations and demonstrates the advantage over alternatives.