TASK DEFINITION: In this task, you are given a paper review. Based on the review, your job is to identify language and generate "en" if the review is in English or generate "es" if the review is in Spanish. Note that URLs in the text have been replaced with [Link].
PROBLEM: - Strengths: A well written paper, examining the use of context in lexical entailment task is a great idea, a well defined approach and experimental set-up and good analysis of the results   - Weaknesses: Some information is missing or insufficient, e.g., the table captions should be more descriptive, a clear description for each of the word type features should be given.  General Discussion:   The paper presents a proposal of consideration of context in lexical entailment task. The results from the experiments demonstrate that context-informed models do better than context-agnostic models on the entailment task.   I liked the idea of creating negative examples to get negative annotations automatically in the two ways described in the paper based on WordNet positive examples. (new dataset; an interesting method to develop dataset)  I also liked the idea of transforming already-used context-agnostic representations into contextualized representations, experimenting with different ways to get contextualized representations (i.e., mask vs contetx2vec), and testing the model on 3 different datasets (generalizability not just across different datasets but also cross-linguistically).  Motivations for various decisions in the experimental design were good to see, e.g., why authors used the split they used for CONTEXT-PPDB (it showed that they thought out clearly what exactly they were doing and why).  Lines 431-434: authors might want to state briefly how the class weights were determined and added to account for the unbalanced data in the CONTEXT-WN experiments. Would it affect direct comparisons with previous work, in what ways?   Change in Line 589: directionality 4 --> directionality, as in Table 4  Suggested change in Line 696-697: is-a hierarchy of WordNet --> "is-a" hierarchy of WordNet   For the sake of completeness, represent "mask" also in Figure 1.  I have read the author response.

SOLUTION: en

PROBLEM: This paper introduces UCCA as a target representation for semantic parsing and also describes a quite successful transition-based parser for inference into that representation. I liked this paper a lot. I believe there is a lot of value simply in the introduction of UCCA (not new, but I believe relatively new to this community), which has the potential to spark new thinking about semantic representations of text. I also think the model was well thought out. While the model itself was fairly derivative of existing transition-based schemes, the extensions the authors introduced to make the model applicable in this domain were reasonable and well-explained, at what I believe to be an appropriate level of detail.  The empirical evaluation was pretty convincing -- the results were good, as compared to several credible baselines, and the authors demonstrated this performance in multiple domains. My biggest complaint about this paper is the lack of multilingual evaluation, especially given that the formalism being experimented with is exactly one that is supposed to be fairly universal. I'm reasonably sure multilingual UCCA corpora exist (in fact, I think the "20k leagues" corpus used in this paper is one such), so it would be good to see results in a language other than English.  One minor point: in section 6, the authors refer to their model as "grammarless", which strikes me as not quite correct. It's true that the UCCA representation isn't derived from linguistic notions of syntax, but it still defines a way to construct a compositional abstract symbolic representation of text, which to me, is precisely a grammar. (This is clearly a quibble, and I don't know why it irked me enough that I feel compelled to address it, but it did.)  Edited to add: Thanks to the authors for their response.

SOLUTION: en

PROBLEM: The paper describes an MT training data selection approach that scores and ranks general-domain sentences using a CNN classifier. Comparison to prior work using continuous or n-gram based language models is well done, even though  it is not clear of the paper also compared against bilingual data selection (e.g. sum of difference of cross-entropies). The motivation to use a CNN instead of an RNN/LSTM was first unclear to me, but it is a strength of the paper to argue that certain sections of a text/sentence are more important than others and this is achieved by a CNN. However, the paper does not experimentally show whether a BOW or SEQ (or the combination of both( representation is more important and why. The textual description of the CNN (one-hot or semi-supervised using pre-trained embeddings)  is clear, detailed, and points out the important aspects. However, a picture of the layers showing how inputs are combined would be worth a thousand words.  The paper is overall well written, but some parentheses for citations are not necessary (\citet vs. \citep) (e.g line 385).  Experiments and evaluation support the claims of the paper, but I am a little bit concerned about the method of determining the number of selected in-domain sentences (line 443) based on a separate validation set: - What validation data is used here? It is also not clear on what data hyperparameters of the CNN models are chosen. How sensitive are the models to this? - Table 2 should really compare scores of different approaches with the same number of sentences selected. As Figure 1 shows, the approach of the paper still seems to outperform the baselines in this case.   Other comments: - I would be interested in an experiment that compares the technique of the paper against baselines when more in-domain data is available, not just the development set. - The results or discussion section could feature some example sentences selected by the different methods to support the claims made in section 5.4. - In regards to the argument of abstracting away from surface forms in 5.4: Another baseline to compare against could have been the work of Axelrod, 2015, who replace some words with POS tags to reduce LM data sparsity to see whether the word2vec embeddings provide an additional advantage over this. - Using the sum of source and target classification scores is very similar to source & target Lewis-Moore LM data selection: sum of difference of cross-entropies. A reference to this work around line 435 would be reasonable.  Finally, I wonder if you could learn weights for the sum of both source & target classification scores by extending the CNN model to the bilingual/parallel setting.

SOLUTION:
en

